--- title: Debug Env - AI Debugging Benchmark emoji: 🐛 colorFrom: blue colorTo: indigo sdk: docker pinned: true app_port: 7860 tags: - openenv - debugging - benchmark - ai-agents --- # debug-env — AI Code Debugging Benchmark An [OpenEnv](https://github.com/meta-pytorch/OpenEnv) benchmark where LLM agents fix broken Python code. Agents use four tools to diagnose and repair bugs across 9 tasks of increasing complexity. Rewards are shaped on test pass rate (0.0–1.0) with efficiency bonuses. **Live Space**: [vanshgoel1-debug-env.hf.space](https://vanshgoel1-debug-env.hf.space) **GitHub**: [VanshGoel-1/debug-env](https://github.com/VanshGoel-1/debug-env) --- ## Environment Description Real-world code debugging is a core developer skill and a meaningful benchmark for LLM agents. debug-env places agents inside an isolated working directory containing buggy Python code and a pytest test suite. The agent must use tools to understand the code, identify the bug, and write a corrected file — all within a fixed step budget. The environment models genuine debugging scenarios: syntax errors, logic bugs, multi-file issues, type errors, code quality problems, and collaborative refactoring tasks. These are tasks developers do every day. --- ## Action Space ```python class DebugAction(Action): tool: str # one of: "list_files", "run_tests", "read_file", "edit_file" args: dict # tool-specific arguments (see below) ``` | Tool | Args | Description | |------|------|-------------| | `list_files` | `{}` | List all editable source files in the task workdir | | `run_tests` | `{}` | Run pytest and return pass rate + full output | | `read_file` | `{"path": "broken_code.py"}` | Read a source file | | `edit_file` | `{"path": "broken_code.py", "content": "..."}` | Overwrite file, then run tests | --- ## Observation Space ```python class DebugObservation(Observation): pass_rate: float # fraction of tests passing (0.0–1.0) logs: str # test output, file content, or error message reward: float # shaped reward for this step done: bool # True when all tests pass (pass_rate == 1.0) ``` --- ## Reward Function Defined in `debug_env/server/grader.py`: ``` reward = pass_rate − min(max(0, (steps − 3) × 0.01), 0.3) # step penalty after step 3 + 0.1 if pass_rate == 1.0 # completion bonus + 0.2 × max(0, 1 − steps/max_steps) # efficiency bonus on full solve ``` - Partial credit for partial test passage (not sparse) - Penalises thrashing (many steps without progress) - Rewards solving quickly --- ## Tasks | ID | Title | Difficulty | Type | Files | |----|-------|-----------|------|-------| | task1 | Fix Syntax Error | Easy | Bug fix | `broken_code.py` | | task2 | Fix Logic Error | Medium | Bug fix | `broken_code.py` | | task3 | Fix Multi-file Bug | Hard | Bug fix | `broken_code.py`, `helper.py` | | task4 | Fix Type Errors | Medium | Type check | `typed_code.py` | | task5 | Remove Dead Code | Medium | Code quality | `messy_code.py` | | task6 | Architecture Refactor | Hard | Refactoring | 4 files | | task7 | Code Review Workflow | Medium | Collaborative | `auth.py` | | task8 | Cross-team Bug Investigation | Hard | Collaborative | 3 files | | task9 | Collaborative Refactoring | Hard | Collaborative | 4 files | Tasks 1–3 cover the mandatory competition range (easy → medium → hard). Tasks 4–9 provide additional challenge for frontier models. --- ## Setup ### Prerequisites - Python 3.10+ - [uv](https://github.com/astral-sh/uv) package manager ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` ### Install ```bash git clone https://github.com/VanshGoel-1/debug-env cd debug-env uv sync ``` ### Configure ```bash cp .env.example .env # Edit .env and set: # HF_TOKEN=hf_... (required) # API_BASE_URL=https://router.huggingface.co/v1 # MODEL_NAME=Qwen/Qwen2.5-72B-Instruct ``` ### Verify ```bash python verify_setup.py # Expected: 18 passed, 0 failed ``` --- ## Usage ### Start the server ```bash # Terminal 1 uv run server # Server runs at http://localhost:8000 ``` ### Run the benchmark ```bash # Terminal 2 python inference.py # Run a specific task TASK=task2 python inference.py # Multiple runs for Pass@k TASK=task1 NUMBER_OF_RUNS=3 python inference.py ``` ### Verify the server manually ```bash curl http://localhost:8000/health # {"status":"healthy","service":"debug-env"} curl -X POST http://localhost:8000/reset \ -H "Content-Type: application/json" \ -d '{"task": "task1"}' curl -X POST http://localhost:8000/step \ -H "Content-Type: application/json" \ -d '{"action": {"tool": "list_files", "args": {}}}' ``` --- ## Inference Script `inference.py` uses the OpenAI client against any OpenAI-compatible endpoint. **Required environment variables:** | Variable | Description | |----------|-------------| | `HF_TOKEN` | HuggingFace / API key | | `API_BASE_URL` | LLM API endpoint (default: `https://router.huggingface.co/v1`) | | `MODEL_NAME` | Model identifier (default: `Qwen/Qwen2.5-72B-Instruct`) | **Competition stdout format:** ``` [START] task=task1 env=debug-env model=Qwen/Qwen2.5-72B-Instruct [STEP] step=1 action=list_files() reward=0.00 done=false error=null [STEP] step=2 action=run_tests() reward=0.00 done=false error=null [STEP] step=3 action=read_file('broken_code.py') reward=0.00 done=false error=null [STEP] step=4 action=edit_file('broken_code.py') reward=1.10 done=true error=null [END] success=true steps=4 score=1.10 rewards=0.00,0.00,0.00,1.10 ``` Results are saved to `results_{task}_{timestamp}.json` with `pass@k`, `success_rate`, `avg_steps`, and `avg_final_reward`. --- ## Docker ```bash # Build docker build -t debug-env . # Run docker run -p 8000:8000 debug-env # With Docker Compose docker-compose up --build ``` The Dockerfile uses a two-stage build: dependencies are installed in the builder stage (cache-friendly), source code is copied into the runtime stage. The CMD runs uvicorn directly against `debug_env.server.app:app`. --- ## Deployment to Hugging Face Spaces ```bash # Authenticate huggingface-cli login # Validate before pushing openenv validate # Push openenv push --repo-id your-username/debug-env ``` The `openenv.yaml` at repo root defines the Space configuration. After push, the Space auto-builds from `Dockerfile` and exposes: - `GET /health` — health check - `POST /reset` — start episode - `POST /step` — execute tool - `GET /tasks` — list all tasks - `POST /mcp` — MCP JSON-RPC interface - `/web` — interactive web UI - `/docs` — OpenAPI docs --- ## API Reference ### Core endpoints ``` GET /health → {"status":"healthy","service":"debug-env"} POST /reset {"task":"task1"} → initial observation POST /step {"action":{"tool":...}} → observation, reward, done GET /state → current episode state WS /ws → persistent session (low latency) ``` ### Task endpoints ``` GET /tasks → paginated task list GET /tasks/{taskId} → task metadata GET /tasks/{taskId}/files → editable file list POST /tasks/{taskId}/episodes → create episode GET /tasks/{taskId}/episodes/{id} → get episode PATCH /tasks/{taskId}/episodes/{id} → update episode status ``` ### MCP endpoint ``` POST /mcp {"jsonrpc":"2.0","method":"initialize","params":{"meta":{"task":"task1"}},"id":1} POST /mcp {"jsonrpc":"2.0","method":"tools/list","id":2} POST /mcp {"jsonrpc":"2.0","method":"tools/call","params":{"name":"run_tests","arguments":{}},"id":3} ``` --- ## Project Structure ``` debug-env/ ← repo root, all commands run here ├── pyproject.toml ← single package definition ├── uv.lock ← committed lock file ├── Dockerfile ← single Dockerfile ├── docker-compose.yml ├── openenv.yaml ← OpenEnv spec ├── inference.py ← competition entry point ├── train.py ← RL training (GRPO) ├── verify_setup.py └── debug_env/ ├── models.py ← DebugAction, DebugObservation ├── client.py ← DebugEnv HTTP/WS client ├── server/ │ ├── app.py ← FastAPI entry point │ ├── debug_env_environment.py │ ├── grader.py ← reward shaping │ ├── core/apis.py ← /health │ ├── database/ ← SQLAlchemy + SQLite │ ├── handlers/ ← MCP JSON-RPC dispatcher │ ├── mcp/ ← MCP router + tool specs │ ├── schemas/ ← Pydantic schemas │ ├── tasks/ ← registry, loader, API routes │ ├── tools/ ← run_tests, read_file, edit_file, list_files │ │ + 6 advanced analysis tools │ └── utils/ ← path validation ├── rl/ │ ├── dataset.py ← curriculum dataset for GRPO │ └── rollout.py ← reward bridge for TRL └── tasks/ ├── task1/ – task3/ ← static broken_code.py + test_code.py └── task4/ – task9/ ← seed_data.py (generated at runtime) ``` --- ## RL Training (Optional) GRPO training against the live environment using TRL + Unsloth: ```bash uv sync --extra training uv run server # Terminal 1 — environment must be running python train.py # Terminal 2 # Smaller model for low VRAM: MODEL=Qwen/Qwen2.5-1.5B-Instruct python train.py # Easy tasks only (recommended starting point): TASK_FILTER=easy python train.py ``` VRAM reference (Unsloth 4-bit QLoRA): | Model | Min VRAM | |-------|---------| | Qwen2.5-1.5B | ~2 GB | | Qwen2.5-7B | ~6 GB | | Qwen2.5-14B | ~10 GB | --- ## License BSD 3-Clause — see [LICENSE](LICENSE)