Spaces:
Runtime error
Runtime error
File size: 2,814 Bytes
27d7338 351c200 27d7338 351c200 27d7338 bd428dc 40ab31f 27d7338 351c200 27d7338 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | # code-review-env
Core environment package for Code Review OpenEnv.
## Structure
```
env/
├── environment.py # Reset / step loop
├── models.py # Pydantic schemas
├── reward_engine.py # Dense reward computation
├── state_manager.py # Observation tracking
├── graders/ # Per-task deterministic graders
└── tasks/ # Task definitions (easy, medium, hard)
server.py # FastAPI endpoints
inference.py # Implementation engine (Execute from ROOT via ../inference.py)
tests/ # Pytest suite (70 tests)
```
## Endpoints
| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/health` | Health check |
| `POST` | `/reset` | Start task (body: `{"task_id": "easy"}`) |
| `POST` | `/step` | Submit action, get observation + reward |
| `GET` | `/state` | Debug current state |
## Inference Modes
| Mode | Env Var | LLM Needed | Deterministic |
|------|---------|:---:|:---:|
| Benchmark | `REVIEW_STRATEGY=benchmark` | No | Yes |
| LLM | `REVIEW_STRATEGY=llm` | Yes | No |
Features: schema normalization, line clamping, early-stop on complete findings, deterministic fallback on provider errors, telemetric confidence calibration tracking, red herring traps, adversarial injection hooks.
## Benchmark Results (Latest)
For a complete breakdown, refer to `benchmark_comparison.md` in the repository root.
**Hugging Face Native (Production Phase):**
| Model | Environment | Avg F1 | Avg Conf |
|---|---|---|---|
| `deepseek-ai/DeepSeek-V3` | ✨ HuggingFace | **0.743** | 97% |
| `Qwen/Qwen2.5-72B-Instruct` | ✨ HuggingFace | **0.358** | 95% |
| `meta-llama/Meta-Llama-3-8B-Instruct` | ✨ HuggingFace | **0.144** | 96% |
**OpenRouter (Final Validation):**
| Model | Environment | Avg F1 | Avg Conf |
|---|---|---|---|
| `deepseek-ai/DeepSeek-V3` | 🚀 OpenRouter | **0.712** | 92% |
| `openai/gpt-4o-mini` | 🚀 OpenRouter | **0.694** | 90% |
| `meta-llama/llama-3.3-70b-instruct` | 🚀 OpenRouter | **0.626** | 94% |
| `qwen/qwen-2.5-72b-instruct` | 🚀 OpenRouter | **0.619** | 97% |
### 🧠 Performance Analysis: Why Models Succeed or Fail
- **DeepSeek-V3:** Excels due to perfect **confidence calibration**. It ignores adversarial red herring traps and accurately links vulnerabilities across multiple files.
- **Qwen-2.5-72B:** Strong at localized syntax checking but weak at long-context, cross-file reasoning tracking variables between modules.
- **Llama-3.3-70B:** Severely punished by the F1 grader for "overconfidence syndrome"—guessing wildly at false-positives with `95%` certainty.
- **Small Models:** Primarily fail due to JSON parsing collapse or timeout limits while analyzing heavy source files.
## Tests
```bash
python -m pytest tests -v # 70 passed
```
|