docs/pitch.md

TRACE — Teaching AI to Fix Production Incidents

Triage · Response · Action · Cause · Evaluation

---

🔥 The Problem

Production incidents cost companies millions per hour.

When systems break at 3 AM, engineers scramble through dashboards, comb through logs, and guess at root causes. The average MTTR (Mean Time to Resolution) for critical incidents is 4+ hours — and it's getting worse as systems grow more complex.

Current AI coding agents can write code, but none of them can operate production infrastructure under pressure.

Why? Because there's no training ground for it.

---

💡 The Solution: TRACE

TRACE is the first RL environment designed to teach AI agents how to respond to production incidents.

Think of it as a flight simulator for Site Reliability Engineers — but for AI.

Agent observes metrics → inspects systems → diagnoses root cause → executes fix → validates recovery

One environment. Three difficulty levels. Infinite training runs.

---

🎯 How It Works

The Agent Loop

┌─────────────────────────────────────────────┐
│                  LLM Agent                  │
│        "CPU is at 85%... let me check       │
│         the logs for api_workers"           │
└──────────────────┬──────────────────────────┘
                   │ action: inspect_logs("api_workers")
                   ▼
┌─────────────────────────────────────────────┐
│              TRACE Environment              │
│                                             │
│  📊 Metrics    🖥️ Services    🚨 Alerts      │
│  CPU: 85%     api: degraded   cpu_high      │
│  Latency: 500ms                             │
│                                             │
│  🔍 "Traffic spike detected. Scale workers" │
└──────────────────┬──────────────────────────┘
                   │ reward: +1.0 (useful inspection)
                   ▼
          Agent learns to diagnose → fix → validate

Partial Observability — Like Real Life

The agent sees symptoms, not causes:

• ✅ CPU usage, memory, latency, error rates
• ✅ Service statuses (healthy / degraded / down)
• ✅ Alert names

But root cause is hidden behind inspection actions — just like a real engineer who has to grep the logs.

Three Scenarios, Increasing Complexity

Scenario	What Breaks	Root Cause	How to Fix
🟢 Easy	Traffic spike	Worker overload	Scale horizontally
🟡 Medium	Cascading failures	Queue memory leak	Restart service
🔴 Hard	Multi-service outage	DB pool + bad release	Restart DB + rollback

Smart Grading

score = 0.6 × did_you_fix_it + 0.4 × how_fast_were_you

No hand-wavy metrics. Binary success + speed. That's it.

---

🏗️ Technical Design

OpenEnv Compliant

TRACE is built to the OpenEnv specification — the emerging standard for RL environment benchmarks:

• pyproject.toml + openenv.yaml — auto-validated
• 4 REST endpoints: /reset, /step, /state, /health
• Docker-ready, deploys to HF Spaces in one push

Deterministic & Reproducible

Same seed → same metrics → same trajectory. Every time.

This isn't a toy random environment. The scenarios are hand-crafted to test real incident response patterns — triage, diagnosis, remediation, validation.

Action Space

10 structured actions using (type, target, value) triples:

inspect_logs("database")           → reveals root cause
scale_workers("api_workers", 5)    → horizontal scaling
restart_database()                 → resets connection pool
declare_healthy()                  → "I fixed it" (terminal)

---

📊 Results

Running our benchmark with heuristic agents:

Scenario	Steps	Grade	Status
Easy CPU Spike	3/5	0.76	✅ Solved
Medium Cascade	3/7	0.83	✅ Solved
Hard Mixed	4/8	0.80	✅ Solved

Average: 0.80 — and that's with hand-coded heuristics. The ceiling for LLM agents is much higher.

---

🦾 Why This Matters

For AI Research

• First standardized benchmark for operational AI (not just coding)
• Partial observability forces genuine reasoning under uncertainty
• Structured action space enables reward shaping without reward hacking

For the Industry

• Train agents to reduce MTTR from hours to minutes
• Build autonomous incident response systems
• Bridge the gap between "AI writes code" and "AI runs production"

For the Hackathon

• ✅ OpenEnv validator passes
• ✅ 28 tests, all green
• ✅ Full inference pipeline with LLM agent
• ✅ Interactive Gradio demo
• ✅ Docker builds and serves
• ✅ Push-to-deploy HF Spaces ready

---

🚀 Demo

Live Interactive UI

python ui.py
→ Opens Gradio dashboard at localhost:7861
→ Select scenario, take actions, watch metrics change in real-time

LLM Agent Run

export HF_TOKEN=your-token
python inference.py

[START] task=easy_cpu_spike env=trace model=openai/gpt-oss-20b
[STEP] step=1 action=inspect_logs(api_workers,) reward=1.00 done=false error=null
[STEP] step=2 action=scale_workers(api_workers,5) reward=5.00 done=false error=null
[STEP] step=3 action=declare_healthy(,) reward=10.00 done=true error=null
[END] success=true steps=3 score=0.840 rewards=1.00,5.00,10.00

---

👤 Team

Rajarshi Datta — Builder, designer, engineer.

---

📦 Stack

Component	Technology
Environment	Python, Pydantic
Server	FastAPI, Uvicorn
Inference	OpenAI Client, HF Router
Model	openai/gpt-oss-20b
Deployment	Docker, HF Spaces
Demo	Gradio
Spec	OpenEnv

---

One Line

TRACE teaches AI agents to fix production incidents — the missing benchmark between "AI writes code" and "AI runs production."

---

Built for the Meta × PyTorch × Hugging Face OpenEnv Hackathon.