HACKATHON.md

CodeReview-Env — Hackathon Impact Statement

(Meta × PyTorch × HuggingFace OpenEnv Hackathon 2026)

---

The Problem

Code review is one of the most time-consuming, high-stakes, and yet most poorly-automated tasks in software engineering:

• Developers spend 6–12 hours per week reviewing pull requests.
• Most LLMs today produce dangerously generic feedback: "Looks good!", "LGTM", "Maybe add a test."
• This low-quality output creates alert fatigue — developers stop trusting AI feedback and the review becomes perfunctory.
• Real production incidents (auth bypasses, duplicate payments, pagination data corruption) are missed because the model never looked beyond the first few lines of a diff.

---

Why This Environment Is Unique

1. Genuinely Multi-Step Episodic Structure

Most RL environments for LLMs are single-step (prompt-in, completion-out). CodeReview-Env is genuinely multi-step: an agent must decide in sequence whether to gather more evidence (open_artifact) or commit to a final review (submit_review). This forces the model to learn a temporal exploration strategy, not just a single inference pattern.

reset() → observe diff + ticket
  ├── open_artifact(auth_middleware)   → +0.12 reward, artifact revealed
  ├── open_artifact(security_policy)  → +0.10 reward, policy revealed
  └── submit_review(findings=[...])   → grader score + efficiency bonus, done=True

2. Deterministic, Reproducible Grading — No LLM-as-a-Judge

The core reward signal uses a multi-criterion keyword grader (server/tasks.py) with deterministic, reproducible scoring — no external LLM calls, no flakiness:

• Per-criterion scoring uses 5 weighted factors: issue keywords, recommendation keywords, severity label, file path, and evidence trail (which artifacts were opened).
• Rewards are always in the strictly open interval (0.05, 0.95) — never exactly 0 or 1 — enforced by _clamp_score() at every layer (tasks.py, reward.py, environment.py).

3. Evidence-Gated Difficulty Hierarchy

The three tasks are carefully designed so that the harder tasks require more artifact exploration to score well, creating a natural curriculum:

Task	# Criteria	Artifacts Needed to Score ≥ 0.60	Baseline Score
pagination-regression	2	1 (test_log)	~0.74
tenant-export-auth	2	2 (auth_middleware + security_policy)	~0.61
refund-idempotency	3	4 (payment_client + worker_log + db_model + regression_test)	~0.38

The hardest task (refund-idempotency) requires multi-artifact correlation to detect all three issues: the retry-without-idempotency, the concurrent status-update race, and the missing regression test. A model that skips evidence gathering will miss at least one.

4. Strict Pydantic Enforcement

Agent outputs are validated by codereview_env/models.py before entering the environment:

• ReviewFinding requires title (≥5 chars), file_path (≥3 chars), rationale (≥20 chars), recommendation (≥12 chars), and a valid severity enum.
• CodeReviewObservation enforces score and reward fields as Pydantic gt=0.0, lt=1.0.

This forces agents to output structured, actionable findings rather than free-text markdown.

5. Interactive Web Dashboard

Unlike typical headless RL environments, CodeReview-Env ships with a fully integrated real-time web dashboard (frontend/index.html) served at / by the FastAPI backend:

• Live episode tracking — watch the agent open artifacts and form its review.
• Diff and artifact viewer — renders code files and policies directly in the browser.
• Reward breakdown — shows how the grader evaluated each finding in real-time.
• Task selector — switch between all three tasks interactively.

6. Safety Layer

codereview_env/safety.py provides:

• PaginationValidator — Guards all pagination inputs against type errors and out-of-range values before they reach the underlying pagination system. This mirrors the exact bug in the pagination-regression task, making the environment self-documenting.
• SafeRewardCalculator — Wraps reward math with the same max/min clamping and late-rounding pattern used across the rest of the codebase.

---

Technical Novelty Summary

Feature	Other RL Envs	CodeReview-Env
Episode structure	Single step	Multi-step (up to 7 steps)
Reward signal	LLM judge (flaky)	Deterministic keywords + evidence trail
Reward bounds	Often 0.0 or 1.0	Strictly (0.05, 0.95) — enforced everywhere
Output validation	Free-form text	Pydantic schema with min-length constraints
Artifact exploration	None	3–6 artifacts per task, each with reward
Difficulty tiers	Flat	3 tiers with evidence-gated scoring
Training framework	Custom	TRL-compatible GRPO reward function
Deployment	Often local-only	Docker + HuggingFace Spaces ready

---

Real-World Impact

• A model trained here could autonomously triage PRs at enterprise scale — reviewing 100+ PRs per day with precise, finding-level comments.
• Replaces vague AI suggestions with targeted, line-specific security interventions that static analysis tools universally miss (auth bypasses, TOCTOU races, payment idempotency).
• Trained agents learn to justify their findings — providing rationale and recommendation in a format engineers can immediately act on.

---

Future Extensions

• Continuous task generation — Dynamically generate new CVE-inspired tasks to prevent benchmark overfitting.
• CI/CD integration — Deploy the trained agent as a GitHub Actions bot that reviews incoming PRs automatically.
• Fix-Application loop — Add a push_commit action where the agent patches the code based on its own review findings and verifies unit tests pass.
• Multi-agent review — Multiple agents reviewing the same PR and reaching consensus.

---

Compatibility Matrix

Fully tested and optimized for:

Framework	Usage
openenv-core	Environment base class (Environment[Obs, Act, State])
TRL (GRPO / PPO)	Reward function via CodeReviewEnv.get_reward_breakdown()
Unsloth	Drop-in with TRL GRPO config
SkyRL	Compatible via standard OpenAI-style API
HuggingFace Spaces	Docker SDK, port 7860, root-level Dockerfile

See examples/run_grpo_training.py for the TRL GRPO training integration boilerplate.