Hugging Face's logo

Spaces:
DeepParmar
/
code-review
Runtime error

App Files Community
code-review / README.md
DeepParmar's picture
DeepParmar
Update README.md
7a94b1c unverified
preview code
|
Raw History Blame Contribute Delete
11.3 kB
metadata 
title: Code Review OpenEnv
emoji: 🔎
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false

Code Review OpenEnv

A deterministic, OpenEnv-style benchmark environment for evaluating AI code review agents. The agent receives buggy Python pull requests, leaves structured review comments, and is graded on precision, recall, and semantic understanding against ground-truth bugs.

Live Space: https://deepparmar-code-review.hf.space

Environment Description & Motivation

Traditional code review benchmarks measure LLMs on a binary metric: Did the model flag the correct line? As frontier models approach ceiling performance on these shallow evaluations, we need environments that test deeper capabilities.

This environment simulates real-world code review — a task that software engineers perform daily. An AI agent must:

  1. Read buggy Python code (pull request diffs)
  2. Identify security vulnerabilities, logic bugs, and performance issues
  3. Explain why something is broken (not just where)
  4. Avoid false positives and deliberately planted traps

Action Space

Operation Parameters Description
add_comment line_number, severity, category, message, confidence Flag a bug with explanation + Telemetry calibration tracking
approve summary Approve the PR (risky if bugs remain)
request_changes summary Request changes with summary
done (none) Finish review, trigger final grading
inspect_file filename View a specific file (hard task)
inspect_lines filename, start_line, end_line View specific lines

Severity: critical | major | minor | nit Category: bug | security | performance | style

Observation Space

Field Type Description
task_id str easy, medium, or hard
pr_title / pr_description str Pull request metadata
full_file str Complete file under review
code_diff str Unified diff
existing_comments list Agent's prior comments
step_number / max_steps int Step progress
available_files list Files available for inspection (hard task)

Tasks (3 Difficulty Tiers)

Task Domain Real Bugs Files Traps Semantic Check
easy List processing 3 1 — —
medium Web API handler 4 1 — —
hard Async crypto service 6 3 1 red herring + 2 adversarial comments ✓ explanation_tiers

Task Details

  • Easy: Off-by-one IndexError, null safety check, assignment-in-conditional syntax bug
  • Medium: SQL injection, XSS, IDOR, hardcoded API secret
  • Hard: Unsafe YAML deserialization (RCE), ECB cipher mode, async generator leak, race condition on shared dict, hardcoded secret key fallback, sync I/O blocking event loop — plus a red herring try-except: pass in a retry loop

Reward Function

Condition Reward
Correct bug match (±5 lines, severity+category) +0.15 to +0.30
Severity / category match bonus +0.05
Semantic keyword miss (hard task) −0.10
Confidence Calibration (if confidence > 80%) +0.05 (correct) or −0.10 (wrong)
Duplicate comment −0.05
False positive −0.10
Red herring flagged −0.20
done action Final weighted F1 grader score
Efficiency bonus (fast + high score) +0.10

Grader: Weighted F1 with strict 1-to-1 bug-to-comment matching. Severity weights: critical=3, major=2, minor=1, nit=0.5. False positives are weighted by their assigned severity to penalize spamming. All scores deterministic and reproducible.

Baseline Scores (Latest Results)

Includes Telemetric Confidence Scoring.

🏆 HUGGING FACE NATIVE SERVERLESS (Final Production Phase)

Native inference parsing successfully verified directly over https://router.huggingface.co/v1. DeepSeek-V3 completely dominated the native test group, surgically identifying every web vulnerability in the medium test environment to achieve a mathematically perfect 0.999 limit ceiling.

Native Model Identifier Environment Easy F1 Medium F1 Hard F1 Avg F1 Avg Conf.
deepseek-ai/DeepSeek-V3 ✨ HuggingFace 0.667 0.999 0.564 0.743 97%
Qwen/Qwen2.5-72B-Instruct ✨ HuggingFace 0.200 0.588 0.286 0.358 95%
meta-llama/Meta-Llama-3-8B-Instruct ✨ HuggingFace 0.429 0.001 0.001 0.144 96%
meta-llama/Llama-3.3-70B-Instruct ❌ Rate Limited - - - - -
mistralai/Mixtral-8x7B-Instruct-v0.1 ❌ Model Unsupported - - - - -

🌐 POST-SUBMISSION OPENROUTER BENCHMARKS

Final stress test verification leveraging OpenRouter API failover.

Native Model Identifier Environment Easy F1 Medium F1 Hard F1 Avg F1 Avg Conf.
deepseek-ai/DeepSeek-V3 🚀 OpenRouter 0.750 0.667 0.720 0.712 92%
openai/gpt-4o-mini 🚀 OpenRouter 0.833 0.667 0.581 0.694 90%
meta-llama/llama-3.3-70b-instruct 🚀 OpenRouter 0.500 0.833 0.545 0.626 94%
qwen/qwen-2.5-72b-instruct 🚀 OpenRouter 0.800 0.556 0.500 0.619 97%
mistralai/mistral-small-3.1-24b 🚀 OpenRouter 0.001 0.001 0.999 0.334 10%

🧠 Performance Analysis: Why Models Succeed or Fail

Our deterministic grading environment reveals deep behaviors not captured by standard multiple-choice benchmarks:

  • 🥇 DeepSeek-V3 (The Winner): Dominated because of superior confidence calibration and semantic reasoning. Unlike other models, DeepSeek doesn't just guess. When faced with the adversarial "Red Herring" (try...except: pass inside a backoff loop), its confidence drops, allowing it to bypass the trap entirely. It correctly uses multi-step logic to deduce why code is conceptually flawed rather than just syntactically incorrect.
  • 🥈 Qwen-2.5-72B: Highly capable at identifying localized syntax and logic errors in the Easy and Medium environments. However, it suffered in the Hard task, demonstrating limitations in long-context, cross-file reasoning. It often failed to correctly track how keys generated in config_loader.py were insecurely consumed in crypto_service.py.
  • 🥉 Llama-3.3-70B (The Overconfident Guesser): Suffered mathematically due to overconfidence syndrome. The environment heavily penalizes false positives submitted with >80% confidence. Llama consistently flagged totally secure, verified code blocks as "Critical Vulnerabilities" with 95% confidence, causing its F1 score to crash dynamically. It could not differentiate real bugs from the adversarial comment injections.
  • 📉 Smaller/Local Models (Mixtral, Meta-Llama-8B, Gemma): Generally failed either due to JSON parsing collapse (outputting conversational text or reasoning tags instead of strict operation schemas) or by reaching maximum timeout limits when scanning larger codeblocks.

See FINDINGS_PAPER.md for full analysis.

Setup & Usage

Prerequisites 

pip install -r code-review-env/requirements.txt

Run Server 

cd code-review-env
uvicorn server:app --host 0.0.0.0 --port 7860

Docker 

cd code-review-env
docker build -t code-review-env .
docker run -p 7860:7860 code-review-env

Run Inference

The inference script reads these environment variables:

  • API_BASE_URL — The API endpoint for the LLM (default: https://router.huggingface.co/v1)
  • MODEL_NAME — The model identifier (default: Qwen/Qwen2.5-72B-Instruct)
  • HF_TOKEN — Your Hugging Face / API key (required)
  • ENV_BASE_URL — Environment server URL (default: http://127.0.0.1:7860)
# LLM mode (requires API key)
API_BASE_URL=https://router.huggingface.co/v1 \
MODEL_NAME=Qwen/Qwen2.5-72B-Instruct \
HF_TOKEN=<your-token> \
python inference.py

# Benchmark mode (deterministic, no LLM needed)
REVIEW_STRATEGY=benchmark TASK_IDS=easy,medium,hard \
python inference.py

Run Tests 

python -m pytest code-review-env/tests -v

Project Structure 

code-reviewer-v2/
├── server.py                    # Root FastAPI entrypoint (delegates to impl)
├── inference.py                 # Root inference shim (delegates to impl)
├── openenv.yaml                 # OpenEnv specification manifest
├── Dockerfile                   # Container build definition
├── pyproject.toml               # Project config + pytest settings
├── requirements.txt             # Python dependencies
├── README.md                    # This file
├── FINDINGS_PAPER.md            # Academic findings paper
├── ARCHITECTURE_BLUEPRINT.md    # Technical architecture docs
├── AUDIT_RESULTS.md             # Full audit & compliance results
├── REQUIREMENTS_CHECKLIST.md    # OpenEnv requirements checklist
├── updatess.txt                 # Benchmark progression & change log
├── lgos-complate.txt            # Benchmark log (Run 1 — 5 models)
├── logs-2m.txt                  # Benchmark log (Run 2 — 6 models)
├── logs-3.txt                   # Benchmark log (Run 3 — 6 models, latest)
├── code-review-env/
│   ├── server.py                # FastAPI server with /reset, /step, /state, /health
│   ├── inference.py             # Full LLM inference engine + benchmark mode
│   ├── openenv.yaml             # OpenEnv spec (impl copy)
│   ├── Dockerfile               # Impl Dockerfile
│   ├── requirements.txt         # Impl dependencies
│   ├── env/
│   │   ├── environment.py       # Core environment engine
│   │   ├── reward_engine.py     # Shaped reward computation
│   │   ├── state_manager.py     # Episode state tracking
│   │   ├── models.py            # Pydantic data schemas
│   │   ├── graders/
│   │   │   ├── base_grader.py   # Weighted F1 with semantic gates
│   │   │   ├── grader_easy.py   # Easy task grader
│   │   │   ├── grader_medium.py # Medium task grader
│   │   │   └── grader_hard.py   # Hard task grader (multi-file)
│   │   └── tasks/
│   │       ├── task_easy.py     # 3 runtime logic bugs
│   │       ├── task_medium.py   # 4 security vulnerabilities
│   │       └── task_hard.py     # 6 bugs across 3 files + 1 red herring
│   └── tests/                   # 70+ automated tests
└── server/                      # Alternate ASGI entrypoint

Validation 

openenv validate
  • pytest → 70 passed
  • openenv validate → Passes
  • All live endpoints return HTTP 200
  • Dockerfile builds and runs cleanly
  • Inference completes in <20 minutes on vcpu=2, memory=8gb