Spaces:
Sleeping
Sleeping
Add submission strengths and data mix summary
Browse files
README.md
CHANGED
|
@@ -38,6 +38,21 @@ Per `themes.md` Theme #3.1: *"environments that require real interaction with to
|
|
| 38 |
- β
**Enterprise workflow nuance**: versioning, deprecation, backward compatibility
|
| 39 |
- β
**Verifiable reward**: every step has a deterministic, objective grader
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## Architecture: Phase 1 β Phase 2 β Phase 3
|
| 42 |
|
| 43 |
| Phase | What the agent does | Task examples |
|
|
|
|
| 38 |
- β
**Enterprise workflow nuance**: versioning, deprecation, backward compatibility
|
| 39 |
- β
**Verifiable reward**: every step has a deterministic, objective grader
|
| 40 |
|
| 41 |
+
## What Makes This Submission Strong
|
| 42 |
+
|
| 43 |
+
This is not a static prompt benchmark. It is a runnable OpenEnv environment with hidden ground truth, stateful episodes, objective rewards, real training, and public proof artifacts.
|
| 44 |
+
|
| 45 |
+
| Area | What is included |
|
| 46 |
+
|---|---|
|
| 47 |
+
| **Environment depth** | 9 tasks across detection, downstream impact tracing, and backward-compatible fix verification |
|
| 48 |
+
| **Episode/data mix** | Seeded synthetic enterprise API scenarios: OpenAPI specs, payloads, version diffs, consumer service graphs, and migration candidates |
|
| 49 |
+
| **Reward richness** | 14 independent reward signals covering correct findings, proximity, duplicates, false positives, missed consumers, malformed patches, broken consumers, and anti-spam |
|
| 50 |
+
| **Training evidence** | 300 GRPO steps on Qwen2.5-7B + LoRA, public WandB run, reward curve, training state JSON, full logs, and trained adapter on Hugging Face |
|
| 51 |
+
| **Before/after evaluation** | Three-way comparison: untrained Qwen2.5-72B, untrained Qwen2.5-7B, and trained Qwen2.5-7B + LoRA |
|
| 52 |
+
| **Headline improvement** | `detect_breaking_changes`: 0.01 β 0.67 for the same 7B base model after GRPO |
|
| 53 |
+
|
| 54 |
+
The environment uses generated, deterministic scenarios rather than a scraped external dataset. That is intentional: every episode has known ground truth, which makes the reward signal auditable and lets judges reproduce the same task with a fixed `seed`.
|
| 55 |
+
|
| 56 |
## Architecture: Phase 1 β Phase 2 β Phase 3
|
| 57 |
|
| 58 |
| Phase | What the agent does | Task examples |
|