---
title: TRACE v1
emoji: 🔧
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
---

# Trace

So here's the deal: your AI agent wakes up at 3 AM to a production incident. It sees some dashboards blinking red, but doesn't know what's actually wrong.

![alt text](docs/image.png)

TRACE teaches the agent to:
- **Look at the dashboards** (CPU, memory, error rates)
- **Dig into the logs** (inspect_logs = ask "what happened?")
- **Figure out the root cause** (by trying fixes and seeing if they work)
- **Fix it** (restart services, scale up workers, etc.)

## The Twist

This isn't an easy game. We made it realistic:

✅ **Agent can't just see everything** — logs and deep metrics are hidden. You gotta ask for them.
✅ **Every scenario is the same if you replay it** — no randomness to hide behind  
✅ **Actions are structured** — every fix needs a target (e.g., `api_workers`)  
✅ **Rewards build up** — one bad decision doesn't break everything immediately  
✅ **We only grade on results** — did you fix it in time? That's what matters.  

## Get It Running

**Setup** (one time):
```bash
python -m venv venv
./venv/Scripts/activate  # Windows
pip install -e .
```

**Then**:
```bash
# Terminal 1: Start the server
uvicorn server.app:app --host 0.0.0.0 --port 7860

# Terminal 2: Run tests to make sure it works
pytest tests/ -v

# Terminal 3: Try the demo agent
python inference.py
```

## What's Inside 📦

```
TRACE/
├── pyproject.toml         # Project config (OpenEnv wants this)
├── openenv.yaml           # Tells OpenEnv how to run us
├── Dockerfile             # For containerization
├── README.md              # This file
│
├── server/                # The API server
│   └── app.py             # Actually runs /reset, /step, /state, /health
│
├── trace/                 # The environment logic
│   ├── models.py          # Data structures
│   ├── scenarios.py       # The 3 incidents
│   ├── simulator.py       # Runs the scenario step by step
│   ├── rewards.py         # Calculates points
│   └── graders.py         # Final score
│
├── tests/                 # Everything's tested
│   ├── test_scenarios.py  # Do scenarios work?
│   ├── test_rewards.py    # Do points work?
│   └── test_env.py        # Does the whole thing work?
│
└── inference.py           # Demo: how an agent would play
```

## Three Incidents to Solve

### 1. Easy: The Traffic Spike (5 steps max)

Your API is getting crushed. CPU is maxed out. Something's overloaded.

**What you see:** CPU at 85%, latency jumping, errors starting  
**What you don't see:** It's just too much traffic  
**What to do:** Add more workers (`scale_workers`)

---

### 2. Medium: The Cascade (7 steps max)

Your queue service has a memory leak. As memory fills up, it starts dropping requests. Other services timeout waiting for it. Everything falls apart together.

**What you see:** Queue backing up, workers getting slower, more errors  
**What you don't see:** There's a memory leak you need to restart to fix  
**What to do:** Restart the queue service

---

### 3. Hard: The Two-Problem Incident (8 steps max)

Someone deployed new code that queries the database inefficiently. Now the DB connection pool is exhausted. Plus there's a high CPU spike that's... actually a symptom, not the problem.

**What you see:** Tons of errors, crazy high latency, CPU spike, DB is slow  
**What you don't see:** The deploy broke the queries, and the pool is full  
**What to do:** Restart the database (and maybe rollback the release)  

## API (How to Talk to TRACE)

### Start an incident
```bash
POST /reset
{"task_id": "easy_cpu_spike", "seed": 42}
→ You get the first observation (dashboards showing the problem)
```

### Take an action
```bash
POST /step
{"action": {"action_type": "scale_workers", "target": "api_workers", "value": 5}}
→ You get the new state, points earned this step, and whether it's fixed
```

### Check status anytime
```bash
GET /state
→ Current dashboards, points so far, step count
```

### Is the server alive?
```bash
GET /health
→ {"status": "healthy"}
```

## How Scoring Works 

**You get points for smart moves**, deductions for dumb ones. But points don't count until the episode ends (no cliff-falling mid-incident).

**During the incident:**
- +1 for asking smart questions (logging, metrics checks)
- +5 for actually fixing something
- -0.5 for doing the same thing twice
- -2 for making things worse
- +10 for successfully resolving it
- -5 if you claim it's fixed but it's not

**Final score:**
```
Did you fix it?           → 60% of score
How fast did you fix it?  → 40% of score
```

So speed matters, but not as much as actually *fixing* things.

## Deploy It

**Local** (for testing):
```bash
docker build -t trace:latest .
docker run -p 7860:7860 trace:latest
```

**To the cloud** (Hugging Face Spaces):
Push to the `meta-trace` repo and enable auto-deploy. Done.

---

## Check If It Works

```bash
openenv validate  # Does the API work?
./validate-submission.sh  # Full checks
```

Should see:
- ✅ Server comes up  
- ✅ Scenarios work and are reproducible  
- ✅ Rewards actually accumulate  
- ✅ Agent can fix incidents  
- ✅ Tests pass    

---

**Want the deep dive?** See [agent.md](agent.md).

**Ready to build?** `pip install -e .` and go!