pushpam14 commited on
Commit
6193264
Β·
verified Β·
1 Parent(s): 68e2d3b

Add separate HF mini-blog markdown and link it from README

Browse files
Files changed (1) hide show
  1. README.md +31 -23
README.md CHANGED
@@ -258,45 +258,51 @@ python inference.py
258
  openenv validate
259
  ```
260
 
261
- ## Training Results β€” Before vs After
262
 
263
  > **Training**: GRPO via TRL + Unsloth Β· **Hardware**: HuggingFace Jobs L4 (24 GB) Β· **Steps**: 300 Β· **Wall-time**: 1 h 56 min
264
- > **Why we report both**: per the hackathon judging criteria, "Improvement in Rewards" (20%) requires a baseline-vs-trained comparison. The grader looks at the delta on the right tasks, not absolute numbers.
265
 
266
  ### Reward Curve (training progress)
267
 
268
  ![Reward Curve](results/reward_curve.png)
269
 
270
- *Mean episode reward across 300 GRPO training steps on Qwen2.5-7B-Instruct (4-bit + LoRA r=16). Mean reward stays in the 1.0–1.5 range across training because the base 7B model already produces valid actions; the curve's value is in **what specific tasks improved**, captured in the bar chart below.*
271
 
272
- ### Before vs After β€” per-task comparison
273
 
274
  ![Before vs After](results/before_after.png)
275
 
276
- *Per-task score, baseline 72B (grey) vs trained 7B + LoRA (green). The headroom task `detect_breaking_changes` is the standout β€” the trained model is **44Γ— better** there.*
277
 
278
- | Task | Phase | Baseline (Qwen2.5-72B) | Trained (Qwen2.5-7B + LoRA) | Ξ” |
279
- |---|---|---|---|---|
280
- | `find_type_mismatches` | 1 | 0.75 | 0.75 | β‰ˆ |
281
- | `validate_nested_objects` | 1 | 0.99 | 0.57 | ↓ |
282
- | **`detect_breaking_changes`** | 1 | **0.01** | **0.44** | **+44Γ—** ⭐ |
283
- | `validate_response_schema` | 1 | 0.99 | 0.50 | ↓ |
284
- | `validate_cross_field_constraints` | 1 | 0.86 | 0.29 | ↓ |
285
- | `validate_auth_request` | 1 | 0.99 | 0.33 | ↓ |
286
- | `trace_downstream_blast_radius` | 2 | 0.67 | 0.99 | ↑ |
287
- | `propose_backward_compat_fix` | 3 | 0.99 | 0.99 | = |
288
- | `multi_service_cascade_fix` | 2+3 | 0.99 | 0.99 | = |
289
- | **Mean** | | **0.82** | **0.65** | model is 8Γ— smaller |
290
 
291
- Full per-step rewards in [`../baseline_scores.json`](../baseline_scores.json) (before) and [`../trained_scores.json`](../trained_scores.json) (after).
292
 
293
  ### What the comparison shows
294
 
295
- The trained model is a **Qwen2.5-7B + LoRA r=16 adapter** β€” roughly **8Γ— smaller** than the Qwen2.5-72B baseline. The interesting outcomes:
296
 
297
- - **`detect_breaking_changes` went from 0.01 β†’ 0.44**. The 72B baseline knew where to look (proximity hits) but never predicted `violation_type='breaking_change'` correctly. GRPO taught the smaller model the exact action format the env grader rewards. *This is the strongest piece of evidence that the env trains a real capability.*
298
- - **Phase 2 / Phase 3 tasks** (`trace_downstream_blast_radius`, `propose_backward_compat_fix`, `multi_service_cascade_fix`): trained 7B matches or exceeds the 72B baseline despite being 8Γ— smaller.
299
- - **Some Phase 1 tasks regressed** (`validate_nested_objects`, `validate_response_schema`, etc.). The trained model picks correct first violations, then frequently mode-collapses into duplicate reports. With more diverse sampling (higher temperature, more `num_generations`) this would close.
 
 
 
 
 
 
300
 
301
  | Phase | WandB Run | Notebook |
302
  |---|---|---|
@@ -326,10 +332,12 @@ This is a genuinely underexplored domain in RL/LLM training β€” no prior benchma
326
  | Health check | https://pushpam14-api-contract-validator.hf.space/health |
327
  | Trained LoRA adapter | https://huggingface.co/pushpam14/api-contract-validator-grpo-7b |
328
  | WandB training run (300 steps) | https://wandb.ai/pushpamsubscriptions-inn/openenv-contract-guardian/runs/gch0eg3k |
 
329
  | Training Notebook (Colab) | [`training/grpo_colab.ipynb`](training/grpo_colab.ipynb) |
330
  | Story + Technical Guide | [`ENTERPRISE_CONTRACT_GUARDIAN_STORY.md`](ENTERPRISE_CONTRACT_GUARDIAN_STORY.md) |
331
  | GitHub repo | https://github.com/kumarpushpam17-personal/Hackathon |
332
- | Demo Video / HF Blog | *(add after recording)* |
 
333
 
334
  Quick test:
335
 
 
258
  openenv validate
259
  ```
260
 
261
+ ## Training Results β€” Three-way comparison (before vs after)
262
 
263
  > **Training**: GRPO via TRL + Unsloth Β· **Hardware**: HuggingFace Jobs L4 (24 GB) Β· **Steps**: 300 Β· **Wall-time**: 1 h 56 min
264
+ > **Inference**: Same temperature (0.7) for all three columns β€” the comparison is sampling-fair.
265
 
266
  ### Reward Curve (training progress)
267
 
268
  ![Reward Curve](results/reward_curve.png)
269
 
270
+ *Mean episode reward across 300 GRPO training steps on Qwen2.5-7B-Instruct (4-bit + LoRA r=16). The curve plateaus around 1.0–1.5 because the base 7B model already produces structurally valid actions; the per-task score table below is where the *kind* of improvement becomes visible.*
271
 
272
+ ### Apples-to-apples per-task comparison
273
 
274
  ![Before vs After](results/before_after.png)
275
 
276
+ *Three-bar chart: Qwen2.5-72B baseline (dark grey), Qwen2.5-7B baseline (light grey, **same base as trained**), Qwen2.5-7B + LoRA after GRPO (green). The headroom task `detect_breaking_changes` is the standout β€” both untrained models score 0.01; the trained adapter scores 0.67.*
277
 
278
+ | Task | Phase | 72B baseline | **7B baseline** | **7B + LoRA (trained)** | Ξ” vs 7B base |
279
+ |---|---|---|---|---|---|
280
+ | `find_type_mismatches` | 1 | 0.75 | 0.75 | 0.75 | = |
281
+ | `validate_nested_objects` | 1 | 0.99 | 0.57 | 0.57 | = |
282
+ | **`detect_breaking_changes`** | 1 | **0.01** | **0.01** | **0.67** | **+0.66** 🎯 |
283
+ | `validate_response_schema` | 1 | 0.99 | 0.70 | 0.30 | -0.40 |
284
+ | `validate_cross_field_constraints` | 1 | 0.99 | 0.43 | 0.29 | -0.14 |
285
+ | `validate_auth_request` | 1 | 0.99 | 0.83 | 0.33 | -0.50 |
286
+ | `trace_downstream_blast_radius` | 2 | 0.67 | 0.99 | 0.99 | = |
287
+ | `propose_backward_compat_fix` | 3 | 0.99 | 0.99 | 0.99 | = |
288
+ | `multi_service_cascade_fix` | 2+3 | 0.99 | 0.99 | 0.99 | = |
289
+ | **Mean** | | 0.82 | 0.70 | 0.65 | -0.05 |
290
 
291
+ Score files: [`../baseline_72b_v2_scores.json`](../baseline_72b_v2_scores.json) (72B), [`../baseline_7b_scores.json`](../baseline_7b_scores.json) (untrained 7B β€” apples-to-apples baseline), [`../trained_scores.json`](../trained_scores.json) (7B + LoRA after GRPO).
292
 
293
  ### What the comparison shows
294
 
295
+ The middle column (untrained Qwen-7B) is the fair baseline β€” same base model as the trained one, no adapter. **Any difference between columns 2 and 3 is purely the GRPO training effect.**
296
 
297
+ **The headline win**:
298
+
299
+ > **`detect_breaking_changes` went from 0.01 β†’ 0.67.** Both untrained models β€” *including the 10Γ— larger 72B* β€” scored 0.01 on this task. Both models knew where the breaking changes were (they earned the +0.3 proximity reward repeatedly) but **neither could predict `violation_type='breaking_change'` correctly**. After 300 GRPO steps targeting our environment's reward signal, the 7B+LoRA adapter solves the classification 6 of 9 times. **This is RL training value, not model size.**
300
+
301
+ **The trade-off (we're being honest here)**:
302
+
303
+ Three Phase 1 tasks regressed (`validate_response_schema`, `validate_cross_field_constraints`, `validate_auth_request`). Reading the per-step reward trajectories shows the cause: the trained model finds the first 2–3 correct violations confidently, then keeps reporting the same field repeatedly. This is the **classic RL fine-tuning trade-off** β€” GRPO heavily reinforced specific high-reward action patterns from training (Phase 2/3 episodes give +2.0 fix rewards vs Phase 1's +1.0 per violation). With more training data balanced toward Phase 1 tasks plus an explicit "don't repeat" reward signal, this would close.
304
+
305
+ **Phase 2 / Phase 3 tasks** maintain their scores in the trained adapter β€” the model didn't *forget* multi-service reasoning while learning the breaking-change classification.
306
 
307
  | Phase | WandB Run | Notebook |
308
  |---|---|---|
 
332
  | Health check | https://pushpam14-api-contract-validator.hf.space/health |
333
  | Trained LoRA adapter | https://huggingface.co/pushpam14/api-contract-validator-grpo-7b |
334
  | WandB training run (300 steps) | https://wandb.ai/pushpamsubscriptions-inn/openenv-contract-guardian/runs/gch0eg3k |
335
+ | Training proof + full logs | [`results/TRAINING_RUN_PROOF.md`](results/TRAINING_RUN_PROOF.md) |
336
  | Training Notebook (Colab) | [`training/grpo_colab.ipynb`](training/grpo_colab.ipynb) |
337
  | Story + Technical Guide | [`ENTERPRISE_CONTRACT_GUARDIAN_STORY.md`](ENTERPRISE_CONTRACT_GUARDIAN_STORY.md) |
338
  | GitHub repo | https://github.com/kumarpushpam17-personal/Hackathon |
339
+ | HF mini-blog writeup (separate MD in Space) | [`BLOG.md`](BLOG.md) |
340
+ | Trained adapter model card | https://huggingface.co/pushpam14/api-contract-validator-grpo-7b |
341
 
342
  Quick test:
343