lucifer0077 commited on
Commit
3eb0c56
Β·
verified Β·
1 Parent(s): 088b5d8

Update Blog.md

Browse files
Files changed (1) hide show
  1. Blog.md +24 -24
Blog.md CHANGED
@@ -1,6 +1,6 @@
1
  # I Taught a Small AI to Review Code Better Than a Model 10x Its Size
2
 
3
- *And we did it in 48 hours at the Meta Γ— HuggingFace OpenEnv Grand Finale, Bangalore*
4
 
5
  ---
6
 
@@ -8,7 +8,7 @@ Imagine you're a junior developer. You just spent three days writing a feature.
8
 
9
  Now imagine that senior engineer never sleeps, never gets tired, and gets *better* at code review every single time they review code.
10
 
11
- That's what we built.
12
 
13
  ---
14
 
@@ -18,11 +18,11 @@ Code review is broken.
18
 
19
  Not because developers don't care β€” they do. But because the volume of code being written has outpaced the human capacity to review it carefully. Studies estimate this costs the software industry **$50 billion annually** in bugs that reach production.
20
 
21
- The uncomfortable truth: **every production bug was approved by at least one human reviewer.**
22
 
23
  Existing AI tools can suggest code. They can autocomplete. But none of them *learn* from feedback. None of them get better at reviewing the more they review.
24
 
25
- We wanted to change that.
26
 
27
  ---
28
 
@@ -30,11 +30,11 @@ We wanted to change that.
30
 
31
  Here's the insight that drove everything:
32
 
33
- > Code review has a verifiable ground truth. We *know* which lines have bugs. We *know* how severe they are. We *know* what a correct verdict looks like.
34
 
35
  If you have ground truth, you have a reward signal. If you have a reward signal, you can train an AI agent using reinforcement learning.
36
 
37
- So we built **CodeReviewEnv** β€” an OpenEnv-compliant RL environment where an AI agent plays the role of a senior code reviewer. The agent reads a buggy code diff, submits structured comments with line numbers and severity ratings, issues a verdict, and then β€” here's what makes it novel β€” **tries to fix the bugs it found**.
38
 
39
  The environment scores every move. Find a critical bug? +0.20. Submit a false alarm? -0.08. Fix the bug correctly? +0.40. The agent learns, episode by episode, what good code review actually looks like.
40
 
@@ -67,7 +67,7 @@ class TaskQueue:
67
 
68
  A junior developer might look at this and think: *"Looks fine to me."*
69
 
70
- A senior engineer immediately spots four problems: the unprotected list access, the race condition on `pop(0)`, the swallowed exception, and the missing `join()`.
71
 
72
  Our trained agent now spots all four.
73
 
@@ -75,7 +75,7 @@ Our trained agent now spots all four.
75
 
76
  ## The Environment: 13 Tasks, 6 Languages
77
 
78
- We built 13 carefully curated tasks spanning real bug patterns across the full stack:
79
 
80
  | Language | Task | What's Hidden |
81
  |----------|------|---------------|
@@ -114,22 +114,22 @@ Good RL environments have good reward signals. Here's ours:
114
 
115
  But here's the part most RL environments skip: **anti-reward hacking**.
116
 
117
- A naive agent quickly discovers it can spam 30 vague comments, hope some accidentally match known issues, and always say "request_changes" to grab the verdict bonus. We caught this early and built four server-side checks:
118
 
119
  - **Spam detection** β€” more than 12 comments triggers proportional penalty
120
  - **Duplicate detection** β€” copy-pasting the same comment triggers -0.20
121
  - **Quality check** β€” descriptions under 15 characters are penalized
122
  - **Verdict gaming** β€” `request_changes` with zero comments gets caught
123
 
124
- The result: the agent *had* to actually understand the code to score well.
125
 
126
  ---
127
 
128
  ## Curriculum Learning: The Agent Earns Harder Tasks
129
 
130
- We didn't just throw the agent at concurrency bugs on day one.
131
 
132
- CodeReviewEnv implements adaptive curriculum learning. The agent starts on easy Python bugs. When it averages above 0.75 for three consecutive episodes, the environment automatically promotes it to medium difficulty. Then hard. Then security tasks.
133
 
134
  ```
135
  Episode 1-20: easy Python bugs β†’ agent builds pattern recognition
@@ -150,7 +150,7 @@ No human decides when to increase difficulty. The environment does β€” based pur
150
 
151
  ## GRPO Training: The Agent Actually Learns
152
 
153
- We trained **Qwen2.5-Coder-7B-Instruct** using GRPO (Group Relative Policy Optimization) on an A100 GPU, with CodeReviewEnv as the reward signal.
154
 
155
  500 episodes. 2 hours 43 minutes. One A100.
156
 
@@ -167,9 +167,9 @@ The red smoothed curve in the plot goes up. Consistently. That's not luck β€” th
167
 
168
  ## The Result That Surprised Us
169
 
170
- We expected the trained model to be better than the baseline. We didn't expect *this*:
171
 
172
- | Task | Groq llama-3.3-**70B** (Baseline) | Qwen2.5-Coder-**7B** (After GRPO) | Change |
173
  |------|----------------------------------|-----------------------------------|--------|
174
  | easy | 0.95 | 1.13 | ↑ +0.18 |
175
  | medium | 0.90 | 1.28 | ↑ +0.38 |
@@ -212,16 +212,16 @@ This closed the loop from **detection** to **remediation** β€” the agent doesn't
212
 
213
  ---
214
 
215
- ## What We Learned Building This
216
 
217
  **1. Ground truth is everything.**
218
- The reason this works is that code bugs have verifiable ground truth. The grader always knows the right answer. This is what separates code review from tasks like "write a good essay" β€” there's no ambiguity in whether line 25 has a race condition.
219
 
220
  **2. Reward hacking appears faster than you expect.**
221
- Within 20 episodes of our first training run, the model discovered it could output `request_changes` with no comments and get +0.10 every time. We saw reward plateau at exactly +0.08. The anti-hacking checks were not optional β€” they were essential.
222
 
223
- **3. Curriculum matters more than we thought.**
224
- Without curriculum, reward variance was so high the model couldn't find a learning signal. With curriculum, the training curve became smooth and consistent. This one change was the difference between a confused model and a learning one.
225
 
226
  **4. A small model with good training beats a big model without it.**
227
  This is the most important lesson. The 70B model has seen more code than our 7B model ever will. But it has never been corrected 500 times for missing a race condition. Practice beats knowledge.
@@ -276,7 +276,7 @@ curl https://lucifer0077-code-review-env.hf.space/curriculum/state
276
  This is version 1. Here's what version 2 looks like:
277
 
278
  - **Multi-step episodes** β€” agent reviews, gets feedback, revises. Like a real code review conversation.
279
- - **GitHub webhook** β€” environment hooks directly into real pull requests. Agent reviews your actual PRs.
280
  - **More languages** β€” Go, Rust, Java. Because bugs don't care what language you write in.
281
  - **Larger model** β€” 34B with full fine-tuning. If 7B beats 70B, what does 34B do?
282
 
@@ -284,13 +284,13 @@ This is version 1. Here's what version 2 looks like:
284
 
285
  ## One Last Thing
286
 
287
- We built this in 48 hours. One developer. Limited coding experience. With AI assistance.
288
 
289
  The environment works. The training pipeline works. The agent genuinely improved. The results are real.
290
 
291
  If a beginner can build a system where a 7B model beats a 70B model at code review in 48 hours using open-source tools β€” imagine what a team of engineers can do with more time.
292
 
293
- That's the promise of OpenEnv. That's why we built CodeReviewEnv.
294
 
295
  ---
296
 
@@ -298,4 +298,4 @@ That's the promise of OpenEnv. That's why we built CodeReviewEnv.
298
 
299
  *Aditya Sharma (lucifer0077)*
300
 
301
- *Theme 4: Self-Improving Agent | Theme 3.1: Professional Tasks*
 
1
  # I Taught a Small AI to Review Code Better Than a Model 10x Its Size
2
 
3
+ *And I did it in 48 hours at the Meta Γ— HuggingFace OpenEnv Grand Finale, Bangalore*
4
 
5
  ---
6
 
 
8
 
9
  Now imagine that senior engineer never sleeps, never gets tired, and gets *better* at code review every single time they review code.
10
 
11
+ That's what I built.
12
 
13
  ---
14
 
 
18
 
19
  Not because developers don't care β€” they do. But because the volume of code being written has outpaced the human capacity to review it carefully. Studies estimate this costs the software industry **$50 billion annually** in bugs that reach production.
20
 
21
+ The uncomfortable truth: **every production bug was approved by at least one human revieIr.**
22
 
23
  Existing AI tools can suggest code. They can autocomplete. But none of them *learn* from feedback. None of them get better at reviewing the more they review.
24
 
25
+ I wanted to change that.
26
 
27
  ---
28
 
 
30
 
31
  Here's the insight that drove everything:
32
 
33
+ > Code review has a verifiable ground truth. I *know* which lines have bugs. I *know* how severe they are. I *know* what a correct verdict looks like.
34
 
35
  If you have ground truth, you have a reward signal. If you have a reward signal, you can train an AI agent using reinforcement learning.
36
 
37
+ So I built **CodeRevieInv** β€” an OpenEnv-compliant RL environment where an AI agent plays the role of a senior code revieIr. The agent reads a buggy code diff, submits structured comments with line numbers and severity ratings, issues a verdict, and then β€” here's what makes it novel β€” **tries to fix the bugs it found**.
38
 
39
  The environment scores every move. Find a critical bug? +0.20. Submit a false alarm? -0.08. Fix the bug correctly? +0.40. The agent learns, episode by episode, what good code review actually looks like.
40
 
 
67
 
68
  A junior developer might look at this and think: *"Looks fine to me."*
69
 
70
+ A senior engineer immediately spots four problems: the unprotected list access, the race condition on `pop(0)`, the swalloId exception, and the missing `join()`.
71
 
72
  Our trained agent now spots all four.
73
 
 
75
 
76
  ## The Environment: 13 Tasks, 6 Languages
77
 
78
+ I built 13 carefully curated tasks spanning real bug patterns across the full stack:
79
 
80
  | Language | Task | What's Hidden |
81
  |----------|------|---------------|
 
114
 
115
  But here's the part most RL environments skip: **anti-reward hacking**.
116
 
117
+ A naive agent quickly discovers it can spam 30 vague comments, hope some accidentally match known issues, and always say "request_changes" to grab the verdict bonus. I caught this early and built four server-side checks:
118
 
119
  - **Spam detection** β€” more than 12 comments triggers proportional penalty
120
  - **Duplicate detection** β€” copy-pasting the same comment triggers -0.20
121
  - **Quality check** β€” descriptions under 15 characters are penalized
122
  - **Verdict gaming** β€” `request_changes` with zero comments gets caught
123
 
124
+ The result: the agent *had* to actually understand the code to score Ill.
125
 
126
  ---
127
 
128
  ## Curriculum Learning: The Agent Earns Harder Tasks
129
 
130
+ I didn't just throw the agent at concurrency bugs on day one.
131
 
132
+ CodeRevieInv implements adaptive curriculum learning. The agent starts on easy Python bugs. When it averages above 0.75 for three consecutive episodes, the environment automatically promotes it to medium difficulty. Then hard. Then security tasks.
133
 
134
  ```
135
  Episode 1-20: easy Python bugs β†’ agent builds pattern recognition
 
150
 
151
  ## GRPO Training: The Agent Actually Learns
152
 
153
+ I trained **QIn2.5-Coder-7B-Instruct** using GRPO (Group Relative Policy Optimization) on an A100 GPU, with CodeRevieInv as the reward signal.
154
 
155
  500 episodes. 2 hours 43 minutes. One A100.
156
 
 
167
 
168
  ## The Result That Surprised Us
169
 
170
+ I expected the trained model to be better than the baseline. I didn't expect *this*:
171
 
172
+ | Task | Groq llama-3.3-**70B** (Baseline) | QIn2.5-Coder-**7B** (After GRPO) | Change |
173
  |------|----------------------------------|-----------------------------------|--------|
174
  | easy | 0.95 | 1.13 | ↑ +0.18 |
175
  | medium | 0.90 | 1.28 | ↑ +0.38 |
 
212
 
213
  ---
214
 
215
+ ## What I Learned Building This
216
 
217
  **1. Ground truth is everything.**
218
+ The reason this works is that code bugs have verifiable ground truth. The grader always knows the right ansIr. This is what separates code review from tasks like "write a good essay" β€” there's no ambiguity in whether line 25 has a race condition.
219
 
220
  **2. Reward hacking appears faster than you expect.**
221
+ Within 20 episodes of our first training run, the model discovered it could output `request_changes` with no comments and get +0.10 every time. I saw reward plateau at exactly +0.08. The anti-hacking checks Ire not optional β€” they Ire essential.
222
 
223
+ **3. Curriculum matters more than I thought.**
224
+ Without curriculum, reward variance was so high the model couldn't find a learning signal. With curriculum, the training curve became smooth and consistent. This one change was the difference betIen a confused model and a learning one.
225
 
226
  **4. A small model with good training beats a big model without it.**
227
  This is the most important lesson. The 70B model has seen more code than our 7B model ever will. But it has never been corrected 500 times for missing a race condition. Practice beats knowledge.
 
276
  This is version 1. Here's what version 2 looks like:
277
 
278
  - **Multi-step episodes** β€” agent reviews, gets feedback, revises. Like a real code review conversation.
279
+ - **GitHub Ibhook** β€” environment hooks directly into real pull requests. Agent reviews your actual PRs.
280
  - **More languages** β€” Go, Rust, Java. Because bugs don't care what language you write in.
281
  - **Larger model** β€” 34B with full fine-tuning. If 7B beats 70B, what does 34B do?
282
 
 
284
 
285
  ## One Last Thing
286
 
287
+ I built this in 48 hours. One developer. Limited coding experience. With AI assistance.
288
 
289
  The environment works. The training pipeline works. The agent genuinely improved. The results are real.
290
 
291
  If a beginner can build a system where a 7B model beats a 70B model at code review in 48 hours using open-source tools β€” imagine what a team of engineers can do with more time.
292
 
293
+ That's the promise of OpenEnv. That's why I built CodeRevieInv.
294
 
295
  ---
296
 
 
298
 
299
  *Aditya Sharma (lucifer0077)*
300
 
301
+ *Theme 4: Self-Improving Agent | Theme 3.1: Professional Tasks*