README.md

metadata

title: Debug Env - AI Debugging Benchmark
emoji: 🐛
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: true
app_port: 7860
tags:
  - openenv
  - debugging
  - benchmark
  - ai-agents

debug-env — AI Code Debugging Benchmark

An OpenEnv benchmark where LLM agents fix broken Python code. Agents use four tools to diagnose and repair bugs across 9 tasks of increasing complexity. Rewards are shaped on test pass rate (0.0–1.0) with efficiency bonuses.

Live Space: vanshgoel1-debug-env.hf.space
GitHub: VanshGoel-1/debug-env

---

Environment Description

Real-world code debugging is a core developer skill and a meaningful benchmark for LLM agents. debug-env places agents inside an isolated working directory containing buggy Python code and a pytest test suite. The agent must use tools to understand the code, identify the bug, and write a corrected file — all within a fixed step budget.

The environment models genuine debugging scenarios: syntax errors, logic bugs, multi-file issues, type errors, code quality problems, and collaborative refactoring tasks. These are tasks developers do every day.

---

Action Space

class DebugAction(Action):
    tool: str   # one of: "list_files", "run_tests", "read_file", "edit_file"
    args: dict  # tool-specific arguments (see below)

Tool	Args	Description
list_files	{}	List all editable source files in the task workdir
run_tests	{}	Run pytest and return pass rate + full output
read_file	{"path": "broken_code.py"}	Read a source file
edit_file	{"path": "broken_code.py", "content": "..."}	Overwrite file, then run tests

---

Observation Space

class DebugObservation(Observation):
    pass_rate: float  # fraction of tests passing (0.0–1.0)
    logs: str         # test output, file content, or error message
    reward: float     # shaped reward for this step
    done: bool        # True when all tests pass (pass_rate == 1.0)

---

Reward Function

Defined in debug_env/server/grader.py:

reward = pass_rate
       − min(max(0, (steps − 3) × 0.01), 0.3)   # step penalty after step 3
       + 0.1  if pass_rate == 1.0                  # completion bonus
       + 0.2 × max(0, 1 − steps/max_steps)         # efficiency bonus on full solve

• Partial credit for partial test passage (not sparse)
• Penalises thrashing (many steps without progress)
• Rewards solving quickly

---

Tasks

ID	Title	Difficulty	Type	Files
task1	Fix Syntax Error	Easy	Bug fix	broken_code.py
task2	Fix Logic Error	Medium	Bug fix	broken_code.py
task3	Fix Multi-file Bug	Hard	Bug fix	broken_code.py, helper.py
task4	Fix Type Errors	Medium	Type check	typed_code.py
task5	Remove Dead Code	Medium	Code quality	messy_code.py
task6	Architecture Refactor	Hard	Refactoring	4 files
task7	Code Review Workflow	Medium	Collaborative	auth.py
task8	Cross-team Bug Investigation	Hard	Collaborative	3 files
task9	Collaborative Refactoring	Hard	Collaborative	4 files

Tasks 1–3 cover the mandatory competition range (easy → medium → hard). Tasks 4–9 provide additional challenge for frontier models.

---

Setup

Prerequisites

• Python 3.10+
• uv package manager

curl -LsSf https://astral.sh/uv/install.sh | sh

Install

git clone https://github.com/VanshGoel-1/debug-env
cd debug-env

uv sync

Configure

cp .env.example .env
# Edit .env and set:
#   HF_TOKEN=hf_...          (required)
#   API_BASE_URL=https://router.huggingface.co/v1
#   MODEL_NAME=Qwen/Qwen2.5-72B-Instruct

Verify

python verify_setup.py
# Expected: 18 passed, 0 failed

---

Usage

Start the server

# Terminal 1
uv run server
# Server runs at http://localhost:8000

Run the benchmark

# Terminal 2
python inference.py

# Run a specific task
TASK=task2 python inference.py

# Multiple runs for Pass@k
TASK=task1 NUMBER_OF_RUNS=3 python inference.py

Verify the server manually

curl http://localhost:8000/health
# {"status":"healthy","service":"debug-env"}

curl -X POST http://localhost:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"task": "task1"}'

curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action": {"tool": "list_files", "args": {}}}'

---

Inference Script

inference.py uses the OpenAI client against any OpenAI-compatible endpoint.

Required environment variables:

Variable	Description
HF_TOKEN	HuggingFace / API key
API_BASE_URL	LLM API endpoint (default: https://router.huggingface.co/v1)
MODEL_NAME	Model identifier (default: Qwen/Qwen2.5-72B-Instruct)

Competition stdout format:

[START] task=task1 env=debug-env model=Qwen/Qwen2.5-72B-Instruct
[STEP]  step=1 action=list_files() reward=0.00 done=false error=null
[STEP]  step=2 action=run_tests() reward=0.00 done=false error=null
[STEP]  step=3 action=read_file('broken_code.py') reward=0.00 done=false error=null
[STEP]  step=4 action=edit_file('broken_code.py') reward=1.10 done=true error=null
[END]   success=true steps=4 score=1.10 rewards=0.00,0.00,0.00,1.10

Results are saved to results_{task}_{timestamp}.json with pass@k, success_rate, avg_steps, and avg_final_reward.

---

Docker

# Build
docker build -t debug-env .

# Run
docker run -p 8000:8000 debug-env

# With Docker Compose
docker-compose up --build

The Dockerfile uses a two-stage build: dependencies are installed in the builder stage (cache-friendly), source code is copied into the runtime stage. The CMD runs uvicorn directly against debug_env.server.app:app.

---

Deployment to Hugging Face Spaces

# Authenticate
huggingface-cli login

# Validate before pushing
openenv validate

# Push
openenv push --repo-id your-username/debug-env

The openenv.yaml at repo root defines the Space configuration. After push, the Space auto-builds from Dockerfile and exposes:

• GET /health — health check
• POST /reset — start episode
• POST /step — execute tool
• GET /tasks — list all tasks
• POST /mcp — MCP JSON-RPC interface
• /web — interactive web UI
• /docs — OpenAPI docs

---

API Reference

Core endpoints

GET  /health                              → {"status":"healthy","service":"debug-env"}
POST /reset   {"task":"task1"}            → initial observation
POST /step    {"action":{"tool":...}}     → observation, reward, done
GET  /state                               → current episode state
WS   /ws                                  → persistent session (low latency)

Task endpoints

GET  /tasks                               → paginated task list
GET  /tasks/{taskId}                      → task metadata
GET  /tasks/{taskId}/files                → editable file list
POST /tasks/{taskId}/episodes             → create episode
GET  /tasks/{taskId}/episodes/{id}        → get episode
PATCH /tasks/{taskId}/episodes/{id}       → update episode status

MCP endpoint

POST /mcp   {"jsonrpc":"2.0","method":"initialize","params":{"meta":{"task":"task1"}},"id":1}
POST /mcp   {"jsonrpc":"2.0","method":"tools/list","id":2}
POST /mcp   {"jsonrpc":"2.0","method":"tools/call","params":{"name":"run_tests","arguments":{}},"id":3}

---

Project Structure

debug-env/                    ← repo root, all commands run here
├── pyproject.toml            ← single package definition
├── uv.lock                   ← committed lock file
├── Dockerfile                ← single Dockerfile
├── docker-compose.yml
├── openenv.yaml              ← OpenEnv spec
├── inference.py              ← competition entry point
├── train.py                  ← RL training (GRPO)
├── verify_setup.py
└── debug_env/
    ├── models.py             ← DebugAction, DebugObservation
    ├── client.py             ← DebugEnv HTTP/WS client
    ├── server/
    │   ├── app.py            ← FastAPI entry point
    │   ├── debug_env_environment.py
    │   ├── grader.py         ← reward shaping
    │   ├── core/apis.py      ← /health
    │   ├── database/         ← SQLAlchemy + SQLite
    │   ├── handlers/         ← MCP JSON-RPC dispatcher
    │   ├── mcp/              ← MCP router + tool specs
    │   ├── schemas/          ← Pydantic schemas
    │   ├── tasks/            ← registry, loader, API routes
    │   ├── tools/            ← run_tests, read_file, edit_file, list_files
    │   │                       + 6 advanced analysis tools
    │   └── utils/            ← path validation
    ├── rl/
    │   ├── dataset.py        ← curriculum dataset for GRPO
    │   └── rollout.py        ← reward bridge for TRL
    └── tasks/
        ├── task1/ – task3/   ← static broken_code.py + test_code.py
        └── task4/ – task9/   ← seed_data.py (generated at runtime)

---

RL Training (Optional)

GRPO training against the live environment using TRL + Unsloth:

uv sync --extra training
uv run server          # Terminal 1 — environment must be running
python train.py        # Terminal 2

# Smaller model for low VRAM:
MODEL=Qwen/Qwen2.5-1.5B-Instruct python train.py

# Easy tasks only (recommended starting point):
TASK_FILTER=easy python train.py

VRAM reference (Unsloth 4-bit QLoRA):

Model	Min VRAM
Qwen2.5-1.5B	~2 GB
Qwen2.5-7B	~6 GB
Qwen2.5-14B	~10 GB

---

License

BSD 3-Clause — see LICENSE