Spaces:

Anuj2209
/

openenv-customer-support

Sleeping

App Files Files Community

openenv-customer-support / README.md

Anuj2209

README: point submission links to Hugging Face Space repo URLs.

f5820b4 3 months ago

preview code

Raw

History Blame Contribute Delete

25.4 kB

metadata

title: Enterprise Incident Command Center
emoji: 🚨
colorFrom: purple
colorTo: red
sdk: docker
tags:
  - openenv
  - rl
  - incident-response
app_port: 7860
pinned: false
license: mit
short_description: Enterprise incident response world model for OpenEnv

OpenEnv: Enterprise Incident Command Center (EICC)

Tagline: When everything is on fire, can your agent keep its cool?

EICC is a deterministic OpenEnv environment that trains LLM agents to behave like a senior SRE / incident commander: diagnose cascading microservice failures, cross-verify potentially outdated knowledge bases, coordinate tools and stakeholders, and execute multi-step remediation - all under SLA pressure and partial observability.

Hugging Face Space (repo): Anuj2209/openenv-customer-support
OpenEnv API: async reset() / step() / state() / close() + HTTP server
Three modes: Ticket mode · Incident mode (Mock env) · Incident mode (VM env)

Theme Alignment

Theme #3 - World Modeling Sub-theme #3.1 - Professional Tasks

Environments that require real interaction with tools, APIs, or dynamic systems where the model is expected to do real hard work instead of exploiting short-cuts. Learning from these environments enables agents to maintain consistent internal state, update beliefs based on outcomes, and orchestrate multi-step workflows. Goal: strengthen causal reasoning and persistent world models. Expected outcome: an environment capturing nuances of a defined partially observable world and improve LLM interaction with it.

How EICC maps to the brief:

Theme requirement	How EICC delivers it
Real interaction with tools / APIs / dynamic systems	8 enterprise tool subsystems (Monitoring, CRM, Billing, KB, Policy Engine, Incident History, Runbooks, Stakeholder Manager). Mode 3 also routes the same action API to a live 5-service Docker cluster.
No shortcut exploits	11 explicit anti-shortcut mechanisms: phase-gated actions, investigation-before-action gate, KB cross-verification gate, blast-radius penalty on wrong fixes, CAB approval for risky changes, resource budgets, tone matching, etc.
Maintain consistent internal state	Long-horizon consistency metric scores whether post-fix actions stay in the consistent set; `known_facts` accumulator forces the agent to reconcile evidence across steps.
Update beliefs based on outcomes	Partial observability - root cause is hidden, KB can be outdated, policy drifts. The agent must call tools, interpret returns, and revise its plan.
Multi-step workflow orchestration	21 typed actions across 4 phases (TRIAGE → INVESTIGATION → RESPONSE → RESOLUTION) with up to 80 steps per nightmare-tier episode.
Causal reasoning	5-service causal mesh with cascading failures; reward function explicitly rewards root-cause-aligned `apply_fix` and penalizes symptom-only fixes.
Persistent world model	Dynamic ticket arrivals, decaying stakeholder patience, evolving service health, policy drift across episode - none of it is reset between steps.
Partial observability	True root cause hidden, full dependency graph hidden on hard+, KB accuracy hidden, policy values hidden, internal customer risk hidden.

This is not a quiz benchmark. It is a partially observable enterprise world where naive prompting fails and only investigation-driven, causally-grounded agents accumulate positive reward.

Public Submission Links

Hugging Face Space (Env): https://huggingface.co/spaces/Anuj2209/openenv-customer-support/tree/main
Training Notebook: https://huggingface.co/spaces/Anuj2209/openenv-customer-support/blob/main/train_notebook.ipynb
Blog Post: https://huggingface.co/spaces/Anuj2209/openenv-customer-support/blob/main/Blog.md

The 3 Modes (what this repo ships)

#	Mode	Purpose	Backend	When to use
1	Ticket mode	Backward-compatible support-ticket triage	Pure simulation	Sanity / regression
2	Incident mode - Mock env	Full incident world (services, tools, customers, stakeholders) for training	Deterministic in-process simulation	All RL training, fast eval, reproducibility
3	Incident mode - VM env	Same incident world, but actions hit a real container cluster	5 Dockerized microservices + chaos controller	Demos, sim→VM transfer scoring, live drill recovery

Modes 2 and 3 share the same OpenEnv API (reset / step / state / close) and the same 21-action schema. You train once on Mode 2 and evaluate the same checkpoint on Mode 3 to get a real-world transfer score.

Walkthrough

This submission is an RL environment for OpenEnv, plus a complete GRPO training pipeline that demonstrably improves an LLM agent's behavior on it.

The RL environment (core deliverable): 21 typed actions across 4 incident phases, 5-service causal mesh, 18 hand-crafted scenarios, deterministic per-step reward (no LLM judge), partial observability (root cause hidden), and reproducible seeds. Tools, policies, and a customer/stakeholder layer are part of the world model so the agent has to investigate before acting rather than guess.
Why it is hard for current LLMs: real enterprise incidents need partial observability, causal reasoning, long-horizon execution, and tool orchestration - not isolated single-turn skills.
Training (RL pipeline): GRPO on Qwen2.5-3B-Instruct via Unsloth + HF TRL in Colab. Reward shaping enforces strict JSON-shape, phase-availability, investigation-before-action, KB cross-verification, policy awareness, and trajectory-grounded action values.
Evidence: evaluate.py produces per-difficulty stage reward curves (reward_curve_easy.png, reward_curve_medium.png, reward_curve_hard.png), root-cause accuracy, 8 tracked behavioral skills, and a machine-readable trained_report.json that records policy_used (trained_checkpoint vs trained_heuristic fallback) so judges can see exactly which policy produced each run.
Mode 3 differentiator: the trained checkpoint is also evaluated on a live Docker cluster, with a transfer_report.json that quantifies how much of the simulated improvement carries over to real infrastructure.

Quick Reproduction (Colab / HF Space)

Open train_notebook.ipynb and run in order. The notebook has two lanes, you can run either or both:

Lane A - Mock environment (recommended first run)

Step	Purpose	Typical time
1	Clone repo	< 10 s
2	Install deps (Unsloth + TRL + peft + bitsandbytes)	~3 min
3	Dry-run sanity check	< 30 s
4	Baseline evaluation → `artifacts/eval_simple/baseline_report.json`	~1 min
5	Quick GRPO training (`--iterations 6 --episodes 8 --k 2`)	~45–75 min on A10/T4
6A	Compare baseline vs trained on Mock env → `artifacts/eval_simple/`	~5–15 min
7	Inspect reports + `policy_used` provenance	instant
8	Display 3 mentor curves (easy/medium/hard)	instant

Lane B - VM environment (live cluster on the same Colab/HF Space)

Run after Lane A Step 5 has produced a trained checkpoint. The VM lane needs a second terminal.

Step	Where	Command
B0	Same machine, new terminal	`pip install -U fastapi uvicorn` (one-time)
B1	New terminal, leave it running	`python -m sandbox.launch_no_docker`
B2	Notebook (after B1 prints "all services listening")	run the Step 6B cell to evaluate the same trained checkpoint against the live cluster → `artifacts/eval_sandbox/`

Step B1 starts 5 FastAPI microservices (auth, database, payments, analytics, notifications) + chaos controller on 127.0.0.1, all in-process - no Docker needed. The notebook's Step 6B cell sets OPENENV_SANDBOX_CLUSTER_URL=http://127.0.0.1 and OPENENV_SANDBOX_CHAOS_URL=http://127.0.0.1:6660 so the same evaluate.py invocation routes through the live backend.

Why two terminals? The cluster is a long-lived process. The notebook runs the eval as a one-shot. Keeping them in separate terminals matches how you would run this on a real on-call laptop.

Want to test the VM environment on your own laptop instead of Colab/HF Space? Follow the full Windows + PowerShell walkthrough in sandbox/Local_Testing_Guide.md. It covers Docker Desktop setup, building the cluster, starting the OpenEnv API in sandbox mode, manual Invoke-RestMethod smoke checks, and the full evaluation commands end-to-end.

How the Pipeline Works (Train → Evaluate)

This is a two-phase workflow. Both phases are run from train_notebook.ipynb.

Phase 1 - Training (weights are actually updated)

train.py rolls out episodes against the EICC RL environment, scores each action with the deterministic reward function in graders/, and uses GRPO (Unsloth + HF TRL) to update a LoRA adapter on top of Qwen2.5-3B-Instruct. The updated weights are saved to artifacts/train/trained_adapter/. This is real RL: the gradient signal comes from our environment's reward, not from a separate teacher.

Phase 2 - Evaluation (frozen weights, before vs after)

evaluate.py then plays a fresh set of deterministic episodes twice, once per policy, against the same environment. Weights are frozen during evaluation - we just measure what training learned.

Side	What it actually is	Why we run it
`baseline`	The untrained `Qwen2.5-3B-Instruct` model. The base LLM walks into an incident cold, with no exposure to our reward function, and tries to solve it using only its pretraining.	The "before" picture. It tells us what an off-the-shelf LLM does when it sees this world for the first time.
`trained`	The same model, but now with the LoRA adapter from Phase 1 applied (`trained_checkpoint`). Same prompts, same env, same seeds - only the weights changed.	The "after" picture. Any improvement here is attributable to RL training in our environment.

Both runs are logged to baseline_report.json and trained_report.json so the gap is auditable per-difficulty, per-skill. The 3 reward curves (reward_curve_{easy,medium,hard}.png) plot the two side by side.

Each trained_report.json also contains a policy_used field (trained_checkpoint if the LoRA adapter loaded successfully, trained_heuristic as a guarded fallback) so judges can verify the trained-side numbers are not from a fallback.

Two lanes, one trained checkpoint

Lane A (Mock env, Mode 2) runs Phase 1 + Phase 2 on the deterministic simulation. Used for training and reproducible scoring.
Lane B (VM env, Mode 3) runs only Phase 2 with the same trained adapter, but against the live container cluster. Used to measure how much of the simulated improvement transfers to real infrastructure.

Where Results Live

We use two top-level folders with very different jobs.

`artifacts/` - scratch space (gitignored, regenerated on every run)

This is where the notebook actually writes during a run. Wiped and overwritten freely. Never referenced by the README, the blog, or the HF Space - judges should never look here.

Folder	Created by	Contents
`artifacts/train/`	Step 5 (`train.py`)	`trained_adapter/` (the LoRA weights), `reward_history.json` (per-iteration training reward), `trajectories.json` (raw rollouts), `checkpoint_eval/` (internal mid-training checks)
`artifacts/eval_simple/`	Step 6A (`evaluate.py --policy compare`, Mock env)	`baseline_report.json`, `trained_report.json`, `reward_curve_easy.png`, `reward_curve_medium.png`, `reward_curve_hard.png`
`artifacts/eval_sandbox/`	Step 6B (`evaluate.py --policy compare --sandbox`, VM env)	Same 5 files as above, but produced against the live cluster

`results/` - curated submission snapshots (committed to git)

Final files referenced by this README, Blog.md, and the HF Space. This is what judges see. Each subfolder is a different lane of the same trained checkpoint.

results/
├── simple/        Mode 2 (Mock env) - baseline-vs-trained on deterministic sim
│   ├── baseline_report.json           ← untrained Qwen2.5-3B-Instruct numbers
│   ├── trained_report.json            ← same model after RL training (with policy_used)
│   ├── reward_curve_easy.png          ← reward curve, easy difficulty
│   ├── reward_curve_medium.png        ← reward curve, medium difficulty
│   └── reward_curve_hard.png          ← reward curve, hard difficulty
├── sandbox/       Mode 3 (VM env) - same checkpoint, but on the live cluster
│   ├── baseline_report.json           ← untrained model, live cluster
│   ├── trained_report.json            ← trained model, live cluster
│   ├── reward_curve_easy.png          ← live-cluster easy curve
│   ├── reward_curve_medium.png        ← live-cluster medium curve
│   └── reward_curve_hard.png          ← live-cluster hard curve
└── training/      Phase-1 training-time signal (one number per iteration)
    └── reward_history.json            ← list of avg rewards across training iterations

We deliberately do not commit trained_adapter/ (LoRA weights) into results/ - it is large and reproducible from the notebook in ~1 hour.

Promotion: scratch → submission

After a clean notebook run, copy these specific files from artifacts/ into results/:

From (`artifacts/...`)	To (`results/...`)
`eval_simple/baseline_report.json`	`simple/baseline_report.json`
`eval_simple/trained_report.json`	`simple/trained_report.json`
`eval_simple/reward_curve_easy.png`	`simple/reward_curve_easy.png`
`eval_simple/reward_curve_medium.png`	`simple/reward_curve_medium.png`
`eval_simple/reward_curve_hard.png`	`simple/reward_curve_hard.png`
`eval_sandbox/baseline_report.json`	`sandbox/baseline_report.json`
`eval_sandbox/trained_report.json`	`sandbox/trained_report.json`
`eval_sandbox/reward_curve_easy.png`	`sandbox/reward_curve_easy.png`
`eval_sandbox/reward_curve_medium.png`	`sandbox/reward_curve_medium.png`
`eval_sandbox/reward_curve_hard.png`	`sandbox/reward_curve_hard.png`
`train/reward_history.json`	`training/reward_history.json`

Want to see our past run before re-training? Open the JSON reports and PNG curves under results/ directly in GitHub or the HF Space file browser. Those are the exact numbers and plots the submission references.

Environment Design

Architecture

                         ENTERPRISE INCIDENT COMMAND CENTER

  Services (causal mesh):
    AUTH ────────► PAYMENTS ────────► NOTIFICATIONS
       \                |
        \               v
         ─────────► DATABASE ◄──────── ANALYTICS

  Enterprise systems (the agent's tools):
    Monitoring · CRM · Billing · Knowledge Base · Policy Engine ·
    Incident History · Runbooks · Stakeholder Manager

  Agent loop:
    Observation → Action → Env transition → Reward → Next observation
    (every report records `policy_used` for provenance)

  Dual-backend evaluation (Modes 2 & 3 share the same API):
    Mock backend  → official deterministic score (training + reproducibility)
    VM backend    → live container cluster (transfer + drill recovery)

Action space (21 actions)

Ticket mode (legacy 6): classify, route, respond, escalate, resolve, request_info.

Incident mode (15 new):

Investigation: check_monitoring, probe_service, fetch_logs, fetch_user_data, check_billing, query_kb, check_policy, query_incident_history, follow_runbook_step
Response: apply_fix, rollback_fix, notify_stakeholders
Resolution: verify_fix, write_postmortem, update_kb

Actions are phase-gated: the agent sees only valid actions for the current phase via available_actions in the observation.

Incident scenario tiers

Tier	Count	Steps	Customers	Characteristics
easy	3	40	2–3	single failure, clear root cause
medium	5	50	4–5	cascading issues, red herrings, one policy drift
hard	7	70	8	deeper cascades, outdated KB, multiple drifts
nightmare	3	80	10	compound faults, high noise, maximum complexity

Partial observability

Hidden from agent (must be discovered): true root cause, full dependency graph on hard+, KB accuracy, current policy values, internal customer risk, red-herring symptoms.

Visible: alert text, tool results, accumulated known_facts, stakeholder patience, SLA/step counter, phase-restricted available_actions.

Reward Design

Per-step deterministic rewards shaped by:

structural validity + phase availability of the action
investigation-before-action, KB cross-verification, policy-awareness
root-cause correctness of apply_fix (with CAB approval gate + blast-radius penalty on wrong fixes)
customer tone matching against sentiment
stakeholder proactivity before patience decays
resource budgets (max fix attempts, escalations, notifications)

Training-time reward adds strict output-shape penalties (single JSON, no extra prose, cap-hit penalty) so the agent is pushed toward compact, parseable actions rather than noisy prose.

No LLM judge is used. All reward logic is in the repo (graders/ and env/).

Composable Rubric API (OpenEnv-native)

The reward signal is also exposed as composable openenv.core.rubrics.Rubric subclasses in graders/openenv_rubrics.py. The top-level IncidentRewardRubric nests Sequential, Gate, and WeightedSum containers so external tooling can introspect every reward dimension by dot-path:

from graders.openenv_rubrics import IncidentRewardRubric

rubric = IncidentRewardRubric()
for path, dim in rubric.named_rubrics():
    print(path)
# -> investigation_before_action, kb_cross_verification, policy_awareness,
#    root_cause_accuracy, blast_radius_safe, resource_budget_respected,
#    weighted_sum, shape_gate, phase_gate, gated_reward

The rubric tree is read-only with respect to the env: it inspects info.reward_breakdown produced by the canonical reward path without mutating env state. Tests in tests/test_openenv_rubrics.py verify both the API contract and that no env reward drift is introduced.

Training Pipeline

train.py - collects trajectories from the live environment across a curriculum (easy → nightmare), runs GRPO (Unsloth + TRL) on Qwen2.5-3B with LoRA, saves adapter to artifacts/train/trained_adapter/, then runs baseline vs trained evaluation with per-difficulty stage reward curves.
evaluate.py - deterministic incident episodes per difficulty tier; reports normalized / raw reward, SLA compliance, root-cause accuracy, long-horizon consistency, 8 behavioral skills, and policy_used provenance.
train_notebook.ipynb - Colab/HF-Space-first flow with both Mock and VM lanes, dry-run, baseline eval, training (quick or full), compare eval, plotting, and optional two-seed reproducibility.

Key CLI examples

# Dry-run sanity (local, no GPU needed)
python train.py --iterations 1 --episodes 1 --k 2 --dry-run

# Quick training (Colab/HF A10 or T4, ~1 hr)
python train.py --iterations 6 --episodes 8 --k 2 --max-completion-length 96 --output-dir artifacts/train

# Full training (T4, ~6-8 hrs)
python train.py --iterations 20 --episodes 30 --k 4 --max-completion-length 128 --output-dir artifacts/train

# Mock env compare (Mode 2)
python evaluate.py --policy compare \
  --compare-trained-policy trained_checkpoint \
  --checkpoint-dir artifacts/train/trained_adapter \
  --checkpoint-base-model Qwen/Qwen2.5-3B-Instruct \
  --episodes-per-difficulty 7 --plot --output-dir artifacts/eval_simple

# VM env compare (Mode 3) - start `python -m sandbox.launch_no_docker` first in another terminal
OPENENV_SANDBOX_CLUSTER_URL=http://127.0.0.1 \
OPENENV_SANDBOX_CHAOS_URL=http://127.0.0.1:6660 \
python evaluate.py --policy compare \
  --compare-trained-policy trained_checkpoint \
  --checkpoint-dir artifacts/train/trained_adapter \
  --checkpoint-base-model Qwen/Qwen2.5-3B-Instruct \
  --episodes-per-difficulty 7 --plot --sandbox --output-dir artifacts/eval_sandbox

Evaluation Metrics

Per-episode:

Normalized reward (clamped to [0, 1])
Raw cumulative reward (can be negative, exposed explicitly)
SLA compliance (pending tickets == 0 at end)
Root-cause accuracy (at least one correct apply_fix landed)
Long-horizon consistency (post-fix actions stay in the consistent set)
8 tracked skills: investigation-before-action, KB cross-verification, policy checking, stakeholder proactivity, root-cause accuracy, tone matching, resource efficiency, red-herring dismissal

Aggregates: avg_normalized_reward, avg_raw_reward, per-difficulty averages, policy_used, episodes_per_difficulty.

In Mode 3 (VM), each step response additionally carries:

info.sandbox.live_action - the live cluster call/return
observation.tool_results.sandbox_live - the same payload surfaced as a tool result

In Mode 3 with --sandbox-drill-mode, the response info also reports per-step drill telemetry (info.sandbox.drill.events, injected_steps, resolved_steps, drill_score).

Results

Final committed snapshots are in results/:

results/simple/ - Mock env baseline + trained reports + 3 reward curves
results/sandbox/ - VM env baseline + trained reports + 3 reward curves
results/training/reward_history.json - training reward history

To view our past results without re-running anything: click into results/ and open the .json reports and .png curves directly. Each trained_report.json includes a policy_used field so you can see whether the numbers came from the real Qwen LoRA checkpoint (trained_checkpoint) or the deterministic heuristic fallback (trained_heuristic).

policy_used field: both baseline_report.json and trained_report.json include policy_used so you can confirm whether trained-side numbers came from the real Qwen LoRA checkpoint (trained_checkpoint) or the deterministic heuristic fallback (trained_heuristic). We keep this explicit in logs/reports for transparency.

HTTP API (OpenEnv-compatible)

Endpoints:

POST /reset, POST /step, GET /state, POST /close
GET /health, POST /inference

Optional production headers:

X-Session-ID: isolate concurrent episodes per client/session
X-API-Key: required only when the server is configured with OPENENV_API_KEY

Example:

curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"mode":"incident","difficulty":"easy","seed":0}'

# Mode 3 with drill (extra optional fields)
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"mode":"incident","difficulty":"hard","seed":0,"drill_mode":true,"drill_seed":7}'

Setup (local)

pip install -e ".[dev]"
python -m pytest tests/ -q

Reproducible runtime dependencies are pinned in requirements.lock.

Deployment (Hugging Face Spaces)

docker build -t eicc .
docker run -p 7860:7860 eicc
curl http://localhost:7860/reset -X POST \
  -H "Content-Type: application/json" \
  -d '{"mode":"incident","difficulty":"easy"}'

The container defaults to 1 worker with a healthcheck on /health. Episode state is in-process, so horizontal scaling requires an external session store.

VM Mode (Live Cluster) - two ways to run it

Mode 3 (VM env) runs the same OpenEnv API but routes incident actions to a real container cluster. There are two supported runners:

Option 1 - Docker Compose (local desktop, full cluster)

docker compose -f sandbox/docker-compose.yml up --build -d
set OPENENV_SANDBOX=true
set OPENENV_SANDBOX_CLUSTER_URL=http://localhost
set OPENENV_SANDBOX_CHAOS_URL=http://localhost:6660
python -m server.app

Detailed walkthrough: sandbox/Local_Testing_Guide.md.

Option 2 - No Docker (Colab / HF Space)

pip install -U fastapi uvicorn       # one-time
python -m sandbox.launch_no_docker   # leave running in a separate terminal

This starts all 5 services + chaos controller as in-process uvicorn threads on 127.0.0.1. Then run the same evaluate.py --sandbox command, pointing it at http://127.0.0.1.

The notebook's Step 6B uses Option 2 by default since Colab and HF Spaces don't run Docker-in-Docker.

Determinism Contract

Seeded scenario selection and service-mesh behavior
No wall-clock / time-based randomness
Same (seed, difficulty, mode) triple deterministically yields the same scenario
Greedy decode (do_sample=False) for checkpoint evaluation

License

MIT