Spaces:
Sleeping
Sleeping
Update Blog.md
Browse files
Blog.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# I Taught a Small AI to Review Code Better Than a Model 10x Its Size
|
| 2 |
|
| 3 |
-
*And
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
@@ -8,7 +8,7 @@ Imagine you're a junior developer. You just spent three days writing a feature.
|
|
| 8 |
|
| 9 |
Now imagine that senior engineer never sleeps, never gets tired, and gets *better* at code review every single time they review code.
|
| 10 |
|
| 11 |
-
That's what
|
| 12 |
|
| 13 |
---
|
| 14 |
|
|
@@ -18,11 +18,11 @@ Code review is broken.
|
|
| 18 |
|
| 19 |
Not because developers don't care β they do. But because the volume of code being written has outpaced the human capacity to review it carefully. Studies estimate this costs the software industry **$50 billion annually** in bugs that reach production.
|
| 20 |
|
| 21 |
-
The uncomfortable truth: **every production bug was approved by at least one human
|
| 22 |
|
| 23 |
Existing AI tools can suggest code. They can autocomplete. But none of them *learn* from feedback. None of them get better at reviewing the more they review.
|
| 24 |
|
| 25 |
-
|
| 26 |
|
| 27 |
---
|
| 28 |
|
|
@@ -30,11 +30,11 @@ We wanted to change that.
|
|
| 30 |
|
| 31 |
Here's the insight that drove everything:
|
| 32 |
|
| 33 |
-
> Code review has a verifiable ground truth.
|
| 34 |
|
| 35 |
If you have ground truth, you have a reward signal. If you have a reward signal, you can train an AI agent using reinforcement learning.
|
| 36 |
|
| 37 |
-
So
|
| 38 |
|
| 39 |
The environment scores every move. Find a critical bug? +0.20. Submit a false alarm? -0.08. Fix the bug correctly? +0.40. The agent learns, episode by episode, what good code review actually looks like.
|
| 40 |
|
|
@@ -67,7 +67,7 @@ class TaskQueue:
|
|
| 67 |
|
| 68 |
A junior developer might look at this and think: *"Looks fine to me."*
|
| 69 |
|
| 70 |
-
A senior engineer immediately spots four problems: the unprotected list access, the race condition on `pop(0)`, the
|
| 71 |
|
| 72 |
Our trained agent now spots all four.
|
| 73 |
|
|
@@ -75,7 +75,7 @@ Our trained agent now spots all four.
|
|
| 75 |
|
| 76 |
## The Environment: 13 Tasks, 6 Languages
|
| 77 |
|
| 78 |
-
|
| 79 |
|
| 80 |
| Language | Task | What's Hidden |
|
| 81 |
|----------|------|---------------|
|
|
@@ -114,22 +114,22 @@ Good RL environments have good reward signals. Here's ours:
|
|
| 114 |
|
| 115 |
But here's the part most RL environments skip: **anti-reward hacking**.
|
| 116 |
|
| 117 |
-
A naive agent quickly discovers it can spam 30 vague comments, hope some accidentally match known issues, and always say "request_changes" to grab the verdict bonus.
|
| 118 |
|
| 119 |
- **Spam detection** β more than 12 comments triggers proportional penalty
|
| 120 |
- **Duplicate detection** β copy-pasting the same comment triggers -0.20
|
| 121 |
- **Quality check** β descriptions under 15 characters are penalized
|
| 122 |
- **Verdict gaming** β `request_changes` with zero comments gets caught
|
| 123 |
|
| 124 |
-
The result: the agent *had* to actually understand the code to score
|
| 125 |
|
| 126 |
---
|
| 127 |
|
| 128 |
## Curriculum Learning: The Agent Earns Harder Tasks
|
| 129 |
|
| 130 |
-
|
| 131 |
|
| 132 |
-
|
| 133 |
|
| 134 |
```
|
| 135 |
Episode 1-20: easy Python bugs β agent builds pattern recognition
|
|
@@ -150,7 +150,7 @@ No human decides when to increase difficulty. The environment does β based pur
|
|
| 150 |
|
| 151 |
## GRPO Training: The Agent Actually Learns
|
| 152 |
|
| 153 |
-
|
| 154 |
|
| 155 |
500 episodes. 2 hours 43 minutes. One A100.
|
| 156 |
|
|
@@ -167,9 +167,9 @@ The red smoothed curve in the plot goes up. Consistently. That's not luck β th
|
|
| 167 |
|
| 168 |
## The Result That Surprised Us
|
| 169 |
|
| 170 |
-
|
| 171 |
|
| 172 |
-
| Task | Groq llama-3.3-**70B** (Baseline) |
|
| 173 |
|------|----------------------------------|-----------------------------------|--------|
|
| 174 |
| easy | 0.95 | 1.13 | β +0.18 |
|
| 175 |
| medium | 0.90 | 1.28 | β +0.38 |
|
|
@@ -212,16 +212,16 @@ This closed the loop from **detection** to **remediation** β the agent doesn't
|
|
| 212 |
|
| 213 |
---
|
| 214 |
|
| 215 |
-
## What
|
| 216 |
|
| 217 |
**1. Ground truth is everything.**
|
| 218 |
-
The reason this works is that code bugs have verifiable ground truth. The grader always knows the right
|
| 219 |
|
| 220 |
**2. Reward hacking appears faster than you expect.**
|
| 221 |
-
Within 20 episodes of our first training run, the model discovered it could output `request_changes` with no comments and get +0.10 every time.
|
| 222 |
|
| 223 |
-
**3. Curriculum matters more than
|
| 224 |
-
Without curriculum, reward variance was so high the model couldn't find a learning signal. With curriculum, the training curve became smooth and consistent. This one change was the difference
|
| 225 |
|
| 226 |
**4. A small model with good training beats a big model without it.**
|
| 227 |
This is the most important lesson. The 70B model has seen more code than our 7B model ever will. But it has never been corrected 500 times for missing a race condition. Practice beats knowledge.
|
|
@@ -276,7 +276,7 @@ curl https://lucifer0077-code-review-env.hf.space/curriculum/state
|
|
| 276 |
This is version 1. Here's what version 2 looks like:
|
| 277 |
|
| 278 |
- **Multi-step episodes** β agent reviews, gets feedback, revises. Like a real code review conversation.
|
| 279 |
-
- **GitHub
|
| 280 |
- **More languages** β Go, Rust, Java. Because bugs don't care what language you write in.
|
| 281 |
- **Larger model** β 34B with full fine-tuning. If 7B beats 70B, what does 34B do?
|
| 282 |
|
|
@@ -284,13 +284,13 @@ This is version 1. Here's what version 2 looks like:
|
|
| 284 |
|
| 285 |
## One Last Thing
|
| 286 |
|
| 287 |
-
|
| 288 |
|
| 289 |
The environment works. The training pipeline works. The agent genuinely improved. The results are real.
|
| 290 |
|
| 291 |
If a beginner can build a system where a 7B model beats a 70B model at code review in 48 hours using open-source tools β imagine what a team of engineers can do with more time.
|
| 292 |
|
| 293 |
-
That's the promise of OpenEnv. That's why
|
| 294 |
|
| 295 |
---
|
| 296 |
|
|
@@ -298,4 +298,4 @@ That's the promise of OpenEnv. That's why we built CodeReviewEnv.
|
|
| 298 |
|
| 299 |
*Aditya Sharma (lucifer0077)*
|
| 300 |
|
| 301 |
-
*Theme 4: Self-Improving Agent | Theme 3.1: Professional Tasks*
|
|
|
|
| 1 |
# I Taught a Small AI to Review Code Better Than a Model 10x Its Size
|
| 2 |
|
| 3 |
+
*And I did it in 48 hours at the Meta Γ HuggingFace OpenEnv Grand Finale, Bangalore*
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
|
|
| 8 |
|
| 9 |
Now imagine that senior engineer never sleeps, never gets tired, and gets *better* at code review every single time they review code.
|
| 10 |
|
| 11 |
+
That's what I built.
|
| 12 |
|
| 13 |
---
|
| 14 |
|
|
|
|
| 18 |
|
| 19 |
Not because developers don't care β they do. But because the volume of code being written has outpaced the human capacity to review it carefully. Studies estimate this costs the software industry **$50 billion annually** in bugs that reach production.
|
| 20 |
|
| 21 |
+
The uncomfortable truth: **every production bug was approved by at least one human revieIr.**
|
| 22 |
|
| 23 |
Existing AI tools can suggest code. They can autocomplete. But none of them *learn* from feedback. None of them get better at reviewing the more they review.
|
| 24 |
|
| 25 |
+
I wanted to change that.
|
| 26 |
|
| 27 |
---
|
| 28 |
|
|
|
|
| 30 |
|
| 31 |
Here's the insight that drove everything:
|
| 32 |
|
| 33 |
+
> Code review has a verifiable ground truth. I *know* which lines have bugs. I *know* how severe they are. I *know* what a correct verdict looks like.
|
| 34 |
|
| 35 |
If you have ground truth, you have a reward signal. If you have a reward signal, you can train an AI agent using reinforcement learning.
|
| 36 |
|
| 37 |
+
So I built **CodeRevieInv** β an OpenEnv-compliant RL environment where an AI agent plays the role of a senior code revieIr. The agent reads a buggy code diff, submits structured comments with line numbers and severity ratings, issues a verdict, and then β here's what makes it novel β **tries to fix the bugs it found**.
|
| 38 |
|
| 39 |
The environment scores every move. Find a critical bug? +0.20. Submit a false alarm? -0.08. Fix the bug correctly? +0.40. The agent learns, episode by episode, what good code review actually looks like.
|
| 40 |
|
|
|
|
| 67 |
|
| 68 |
A junior developer might look at this and think: *"Looks fine to me."*
|
| 69 |
|
| 70 |
+
A senior engineer immediately spots four problems: the unprotected list access, the race condition on `pop(0)`, the swalloId exception, and the missing `join()`.
|
| 71 |
|
| 72 |
Our trained agent now spots all four.
|
| 73 |
|
|
|
|
| 75 |
|
| 76 |
## The Environment: 13 Tasks, 6 Languages
|
| 77 |
|
| 78 |
+
I built 13 carefully curated tasks spanning real bug patterns across the full stack:
|
| 79 |
|
| 80 |
| Language | Task | What's Hidden |
|
| 81 |
|----------|------|---------------|
|
|
|
|
| 114 |
|
| 115 |
But here's the part most RL environments skip: **anti-reward hacking**.
|
| 116 |
|
| 117 |
+
A naive agent quickly discovers it can spam 30 vague comments, hope some accidentally match known issues, and always say "request_changes" to grab the verdict bonus. I caught this early and built four server-side checks:
|
| 118 |
|
| 119 |
- **Spam detection** β more than 12 comments triggers proportional penalty
|
| 120 |
- **Duplicate detection** β copy-pasting the same comment triggers -0.20
|
| 121 |
- **Quality check** β descriptions under 15 characters are penalized
|
| 122 |
- **Verdict gaming** β `request_changes` with zero comments gets caught
|
| 123 |
|
| 124 |
+
The result: the agent *had* to actually understand the code to score Ill.
|
| 125 |
|
| 126 |
---
|
| 127 |
|
| 128 |
## Curriculum Learning: The Agent Earns Harder Tasks
|
| 129 |
|
| 130 |
+
I didn't just throw the agent at concurrency bugs on day one.
|
| 131 |
|
| 132 |
+
CodeRevieInv implements adaptive curriculum learning. The agent starts on easy Python bugs. When it averages above 0.75 for three consecutive episodes, the environment automatically promotes it to medium difficulty. Then hard. Then security tasks.
|
| 133 |
|
| 134 |
```
|
| 135 |
Episode 1-20: easy Python bugs β agent builds pattern recognition
|
|
|
|
| 150 |
|
| 151 |
## GRPO Training: The Agent Actually Learns
|
| 152 |
|
| 153 |
+
I trained **QIn2.5-Coder-7B-Instruct** using GRPO (Group Relative Policy Optimization) on an A100 GPU, with CodeRevieInv as the reward signal.
|
| 154 |
|
| 155 |
500 episodes. 2 hours 43 minutes. One A100.
|
| 156 |
|
|
|
|
| 167 |
|
| 168 |
## The Result That Surprised Us
|
| 169 |
|
| 170 |
+
I expected the trained model to be better than the baseline. I didn't expect *this*:
|
| 171 |
|
| 172 |
+
| Task | Groq llama-3.3-**70B** (Baseline) | QIn2.5-Coder-**7B** (After GRPO) | Change |
|
| 173 |
|------|----------------------------------|-----------------------------------|--------|
|
| 174 |
| easy | 0.95 | 1.13 | β +0.18 |
|
| 175 |
| medium | 0.90 | 1.28 | β +0.38 |
|
|
|
|
| 212 |
|
| 213 |
---
|
| 214 |
|
| 215 |
+
## What I Learned Building This
|
| 216 |
|
| 217 |
**1. Ground truth is everything.**
|
| 218 |
+
The reason this works is that code bugs have verifiable ground truth. The grader always knows the right ansIr. This is what separates code review from tasks like "write a good essay" β there's no ambiguity in whether line 25 has a race condition.
|
| 219 |
|
| 220 |
**2. Reward hacking appears faster than you expect.**
|
| 221 |
+
Within 20 episodes of our first training run, the model discovered it could output `request_changes` with no comments and get +0.10 every time. I saw reward plateau at exactly +0.08. The anti-hacking checks Ire not optional β they Ire essential.
|
| 222 |
|
| 223 |
+
**3. Curriculum matters more than I thought.**
|
| 224 |
+
Without curriculum, reward variance was so high the model couldn't find a learning signal. With curriculum, the training curve became smooth and consistent. This one change was the difference betIen a confused model and a learning one.
|
| 225 |
|
| 226 |
**4. A small model with good training beats a big model without it.**
|
| 227 |
This is the most important lesson. The 70B model has seen more code than our 7B model ever will. But it has never been corrected 500 times for missing a race condition. Practice beats knowledge.
|
|
|
|
| 276 |
This is version 1. Here's what version 2 looks like:
|
| 277 |
|
| 278 |
- **Multi-step episodes** β agent reviews, gets feedback, revises. Like a real code review conversation.
|
| 279 |
+
- **GitHub Ibhook** β environment hooks directly into real pull requests. Agent reviews your actual PRs.
|
| 280 |
- **More languages** β Go, Rust, Java. Because bugs don't care what language you write in.
|
| 281 |
- **Larger model** β 34B with full fine-tuning. If 7B beats 70B, what does 34B do?
|
| 282 |
|
|
|
|
| 284 |
|
| 285 |
## One Last Thing
|
| 286 |
|
| 287 |
+
I built this in 48 hours. One developer. Limited coding experience. With AI assistance.
|
| 288 |
|
| 289 |
The environment works. The training pipeline works. The agent genuinely improved. The results are real.
|
| 290 |
|
| 291 |
If a beginner can build a system where a 7B model beats a 70B model at code review in 48 hours using open-source tools β imagine what a team of engineers can do with more time.
|
| 292 |
|
| 293 |
+
That's the promise of OpenEnv. That's why I built CodeRevieInv.
|
| 294 |
|
| 295 |
---
|
| 296 |
|
|
|
|
| 298 |
|
| 299 |
*Aditya Sharma (lucifer0077)*
|
| 300 |
|
| 301 |
+
*Theme 4: Self-Improving Agent | Theme 3.1: Professional Tasks*
|