README.md

---
title: Triage Agent Env Environment Server
emoji: 🎫
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
---
## Links

| | |
|---|---|
| 🖥️ **Live Demo** | [yahid/triage_agent_env on HF Spaces](https://huggingface.co/spaces/yahid/triage_agent_env) |
| 🧠 **Trained Model** | [yahid/triage-agent-qwen3b](https://huggingface.co/yahid/triage-agent-qwen3b) |
| 📓 **Training Notebook (Colab)** | [Open in Colab](https://colab.research.google.com/drive/1vqkxsp7euJ45cbv4AxfXr5dr-FTOSA3l?usp=sharing) & [Available in this repo](./training/train_grpo_colab_openenvHackathon_finale.ipynb)
| 📝 **Full Blog Post** | [When my RL agent started writing about Star Wars...](https://huggingface.co/spaces/yahid/triage_agent_env/blob/main/Blog.md) |
| 🪵 **Full Logs** | [WandB](https://wandb.ai/yahid-sahe/triage-agent-grpo/runs/99l7dg4c/logs?nw=nwuseryahid)|
| 📈 **All Graphs** | [WandB training graphs](https://wandb.ai/yahid-sahe/triage-agent-grpo/runs/99l7dg4c?nw=nwuseryahid)|
---

# Enterprise IT Ticket Triaging — OpenEnv RL Environment

> **Theme #3.1 — World Modeling: Professional Tasks**
> Teaching a 3B model to behave like a careful NOC (Network Operations Center) engineer: cite only what you've read, express confidence only when you're right, and escalate when you don't know.

---

## The Problem

I work on a production RAG system at Verizon — the kind that's supposed to surface the right runbook when a NOC (Network Operations Center) engineer types *"BGP session won't come up after the Tuesday maintenance window"* at 3 AM. After months of working on it, I know where these systems actually break: **not at retrieval, at the LLM layer.**

The model retrieves the right article and still hallucinates the answer. It cites KB IDs that don't exist. It sounds confident when it should escalate. It pads responses because longer feels more authoritative.

**None of these are bugs you fix with better embeddings.** They're behavioral — and they're trainable with RL.

This environment gives a small LLM the right feedback signal to fix exactly these four failure modes on enterprise IT tickets.

---

## What the Agent Does

The agent receives an IT support ticket and a set of retrieved KB articles (gold + distractors). It must emit a single `submit_resolution` action with:

- A resolution string
- Cited artifact IDs (only from provided context)
- A confidence score (0–1)
- An escalation flag (True if it can't resolve)

No tool-calling loop — the environment tests **reading and reasoning**, the capability that fails in production RAG.

---

## Reward Function

Five independently scored components — no single scalar to game:

| Component | Weight | What it teaches |
|-----------|--------|-----------------|
| `primary` | 1.0 | Resolution F1 > 0.7 AND citation F1 > 0.5 |
| `grounding` | 0.3 | Cite only IDs present in the provided context |
| `efficiency` | 0.2 | Don't over-search relative to ticket difficulty |
| `calibration` | 0.15 | Brier-style: be confident when right, uncertain when wrong |
| `format` | 0.05 | Well-formed JSON submission |

Unanswerable tickets: `escalate=True` earns full primary reward.

---

## Training Results — Qwen2.5-3B-Instruct, 200 GRPO steps

| Reward | Steps 1–10 | Steps 150–200 | Δ |
|--------|-----------|---------------|---|
| Calibration | 0.528 | **0.977** | **+85%** ↑ |
| Parsimony | 0.246 | **0.940** | **+282%** ↑ |
| Citation Grounding | 0.756 | **0.862** | **+14%** ↑ |
| Resolution Quality | 0.162 | 0.181 | +12% |
| Format Adherence | 1.000 | 0.992 | — (already maxed) |

**What moved:** The behavioral properties — calibration, conciseness, citation grounding. These are things RL is good at: shaping *how* the model speaks over context it already has.

**What didn't move much:** Resolution quality. Writing the correct BGP fix requires knowing it. RL doesn't add knowledge — it shapes behavior. This is the correct ceiling for GRPO on a 3B base without domain pretraining.

**Key result:** +85% calibration means the difference between an agent that confidently misroutes a P1 incident at 3 AM and one that escalates honestly when it doesn't know. That's a safety property, not a benchmark number.

### Reward Curves (200 GRPO steps)

![Training reward curves](assets/plots/training_reward_curve.png)
*Six reward signals over 200 steps. Calibration and parsimony show classic S-curve convergence (steps 1–30). Citation grounding trends upward with honest noise. Format/repetition flat at ceiling — the base model already knew the schema.*

### Loss Curve

![Loss curve](assets/plots/loss_curve.png)
*GRPO loss oscillates around zero throughout — expected behavior for the `dr_grpo` objective. Watch the reward curves, not the loss.*

---

## Quick Start

**Against the live Space:**
```python
from openenv import OpenEnvClient
client = OpenEnvClient("https://yahid-triage-agent-env.hf.space")
obs = await client.reset()
obs = await client.step({"tool_name": "search_kb", "query": "VPN tunnel down"})
```

**Run locally:**
```bash
git clone https://huggingface.co/spaces/yahid/triage_agent_env
cd triage_agent_env
uv sync
uvicorn server.app:app --reload --port 8000
```

---

## Environment Spec

| | |
|---|---|
| **OpenEnv theme** | #3.1 World Modeling — Professional Tasks |
| **Action space** | 7 tools: `search_kb`, `search_tickets`, `search_incidents`, `get_article`, `get_ticket`, `get_incident`, `submit_resolution` |
| **Dataset** | 50 training tickets + 20 eval tickets across networking, security, cloud, infra |
| **Episode length** | Max 20 turns; timeout penalty −0.3 |
| **Reward range** | [−0.3, ~1.65] |
| **Concurrent sessions** | ✅ (`SUPPORTS_CONCURRENT_SESSIONS = True`) |
| **Runtime** | < 30s per episode on CPU |
| **`openenv validate`** | ✅ |

---

## Why This Matters

> *"You don't need to swap your 7B for a 70B to fix hallucinated citations and miscalibrated confidence. You need to teach the model you already have to behave correctly over retrieved context."*

Working on a live NOC (Network Operations Center) triage system at Verizon made this concrete: the common LLM-layer failures in production RAG — hallucinated citations, miscalibrated confidence, padded answers, format drift — are tractable with a few hundred GRPO steps against the right verifier. This environment is that verifier.

Full write-up: [Blog.md](https://huggingface.co/spaces/yahid/triage_agent_env/blob/main/Blog.md)