README.md

metadata

title: Triage Agent Env Environment Server
emoji: 🎫
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv

Links

🖥️ Live Demo	yahid/triage_agent_env on HF Spaces
🧠 Trained Model	yahid/triage-agent-qwen3b
📓 Training Notebook (Colab)	Open in Colab & Available in this repo
📝 Full Blog Post	When my RL agent started writing about Star Wars...
🪵 Full Logs	WandB
📈 All Graphs	WandB training graphs

---

Enterprise IT Ticket Triaging — OpenEnv RL Environment

Theme #3.1 — World Modeling: Professional Tasks Teaching a 3B model to behave like a careful NOC (Network Operations Center) engineer: cite only what you've read, express confidence only when you're right, and escalate when you don't know.

---

The Problem

I work on a production RAG system at Verizon — the kind that's supposed to surface the right runbook when a NOC (Network Operations Center) engineer types "BGP session won't come up after the Tuesday maintenance window" at 3 AM. After months of working on it, I know where these systems actually break: not at retrieval, at the LLM layer.

The model retrieves the right article and still hallucinates the answer. It cites KB IDs that don't exist. It sounds confident when it should escalate. It pads responses because longer feels more authoritative.

None of these are bugs you fix with better embeddings. They're behavioral — and they're trainable with RL.

This environment gives a small LLM the right feedback signal to fix exactly these four failure modes on enterprise IT tickets.

---

What the Agent Does

The agent receives an IT support ticket and a set of retrieved KB articles (gold + distractors). It must emit a single submit_resolution action with:

• A resolution string
• Cited artifact IDs (only from provided context)
• A confidence score (0–1)
• An escalation flag (True if it can't resolve)

No tool-calling loop — the environment tests reading and reasoning, the capability that fails in production RAG.

---

Reward Function

Five independently scored components — no single scalar to game:

Component	Weight	What it teaches
primary	1.0	Resolution F1 > 0.7 AND citation F1 > 0.5
grounding	0.3	Cite only IDs present in the provided context
efficiency	0.2	Don't over-search relative to ticket difficulty
calibration	0.15	Brier-style: be confident when right, uncertain when wrong
format	0.05	Well-formed JSON submission

Unanswerable tickets: escalate=True earns full primary reward.

---

Training Results — Qwen2.5-3B-Instruct, 200 GRPO steps

Reward	Steps 1–10	Steps 150–200	Δ
Calibration	0.528	0.977	+85% ↑
Parsimony	0.246	0.940	+282% ↑
Citation Grounding	0.756	0.862	+14% ↑
Resolution Quality	0.162	0.181	+12%
Format Adherence	1.000	0.992	— (already maxed)

What moved: The behavioral properties — calibration, conciseness, citation grounding. These are things RL is good at: shaping how the model speaks over context it already has.

What didn't move much: Resolution quality. Writing the correct BGP fix requires knowing it. RL doesn't add knowledge — it shapes behavior. This is the correct ceiling for GRPO on a 3B base without domain pretraining.

Key result: +85% calibration means the difference between an agent that confidently misroutes a P1 incident at 3 AM and one that escalates honestly when it doesn't know. That's a safety property, not a benchmark number.

Reward Curves (200 GRPO steps)

Six reward signals over 200 steps. Calibration and parsimony show classic S-curve convergence (steps 1–30). Citation grounding trends upward with honest noise. Format/repetition flat at ceiling — the base model already knew the schema.

Loss Curve

GRPO loss oscillates around zero throughout — expected behavior for the dr_grpo objective. Watch the reward curves, not the loss.

---

Quick Start

Against the live Space:

from openenv import OpenEnvClient
client = OpenEnvClient("https://yahid-triage-agent-env.hf.space")
obs = await client.reset()
obs = await client.step({"tool_name": "search_kb", "query": "VPN tunnel down"})

Run locally:

git clone https://huggingface.co/spaces/yahid/triage_agent_env
cd triage_agent_env
uv sync
uvicorn server.app:app --reload --port 8000

---

Environment Spec

OpenEnv theme	#3.1 World Modeling — Professional Tasks
Action space	7 tools: search_kb, search_tickets, search_incidents, get_article, get_ticket, get_incident, submit_resolution
Dataset	50 training tickets + 20 eval tickets across networking, security, cloud, infra
Episode length	Max 20 turns; timeout penalty −0.3
Reward range	[−0.3, ~1.65]
Concurrent sessions	✅ (SUPPORTS_CONCURRENT_SESSIONS = True)
Runtime	< 30s per episode on CPU
openenv validate	✅

---

Why This Matters

"You don't need to swap your 7B for a 70B to fix hallucinated citations and miscalibrated confidence. You need to teach the model you already have to behave correctly over retrieved context."

Working on a live NOC (Network Operations Center) triage system at Verizon made this concrete: the common LLM-layer failures in production RAG — hallucinated citations, miscalibrated confidence, padded answers, format drift — are tractable with a few hundred GRPO steps against the right verifier. This environment is that verifier.

Full write-up: Blog.md