docs/STATUS.md

STATUS — live project state

Update this file at the END of every session. Keep it short (≤ 100 lines). For history, see SESSION_LOG.md.

Last updated: 2026-04-26 11:15 IST — Phase 15: Training fundamentals fixed + Run 6 completed. Diagnosed 4 root causes of eval regression across Runs 1-5 (example contamination in prompt, sparse reward signal, missing required-keys hint, train/eval role mismatch). Fixed all 4 in train_grpo.py and inference.py. Launched Run 6 (1.7B, β=1.0, A100-large, Kanan account) — training rewards were non-zero from step 1 (0.12 first step) and peaked at 0.27, dramatically better than any previous run. Run 6 eval: avg_score=0.0607, completion_rate=16% — nearly matches 1.7B base (0.063 on same v5 prompts). All plots regenerated with Run 6 included. ~5.5 h to the 5 PM IST deadline.

Current phase

Phase 14 — submission: hackathon thesis is "KL-anchored GRPO at scale: can RL improve a strong reasoner without overfitting to one task family?" Run 1 (0.6B, β=0), Run 2 (1.7B, β=0), and Run 4 (1.7B, β=0.2) form a 3-point ablation; the 4B base eval marks the ceiling. Run 3 (4B, β=0) was canceled in queue and is logged in blog.md §7b as the natural next experiment (4B + β=0.2 + half-LR).

HF Space: https://huggingface.co/spaces/agarwalanu3103/clarify-rl — LIVE, 64 concurrent sessions. Trained models: https://huggingface.co/agarwalanu3103/clarify-rl-grpo-qwen3-0-6b, https://huggingface.co/agarwalanu3103/clarify-rl-grpo-qwen3-1-7b, https://huggingface.co/2022uec1542/clarify-rl-grpo-qwen3-1-7b (Run 4, β=0.2 KL anchor).

Last completed (this session)

• ✅ Final hackathon-criteria audit against Discord auto-validator gates + Themes & Judging Criteria + FAQ + Help Guide — all P0/P1 gates GREEN
• ✅ README restructured for judging: Judges-60s-Tour at top, Problem · Environment · Results · Why-it-matters arc, 1-line caption under every embedded plot (plots 01/02/03/04/05/06), Wild Card #5 promoted to title block
• ✅ Run 4 model card: added "Weights mirror note for judges" + commented-out fallback repo = "2022uec1542/clarify-rl-grpo-qwen3-1-7b" so judges always have a working from_pretrained() path even if the personalized mirror is still uploading
• ✅ Logged-out smoke test of 25 submission URLs: env Space landing/README, demo Space, GitHub repo + README + blog + slides + trace_demo + checklist + STATUS + notebook + openenv.yaml + rubrics, 3 anurag203 model cards, upstream Run 4 weights HEAD, W&B project, Colab badge target, 6 plot PNGs — all 200/302
• ✅ Env Space functional smoke: /health → {"status":"healthy"}, /reset → real CallToolObservation with family=medical_intake, request="I have a problem.", task_id=medium, max_steps=10, questions_remaining=6
• ✅ Env Space README has all 5 storytelling anchors (Wild Card / Judges-60s-Tour / Problem · Environment / plot has a 1-line caption / epistemic humility)
• ✅ Pre-existing wins kept: 3 trained GRPO runs evaluated, 4B base ceiling, plots regenerated, all docs reconciled, demo Space (Replay + KL-anchor ablation + Live chat tabs) deployed

Headline n=50 v4 numbers (fair) — corrected

Model	Avg score	Completion	Format pass
Random policy	0.0000	0%	0%
Qwen3-0.6B base	0.0000	0%	0%
Qwen3-0.6B GRPO (Run 1, β=0)	0.0076 ↑	2%	0%
Qwen3-1.7B base	0.0669	18%	0%
Qwen3-1.7B GRPO (Run 2, β=0)	0.0286 ↓	6%	0%
Qwen3-1.7B GRPO (Run 4, β=0.2)	0.0560 ✅	14%	0%
Qwen3-4B-Instruct	0.0399	6%	0%
Qwen3-4B base ← REAL CEILING	0.1446	24%	0%
Qwen3-4B GRPO (Run 3)	canceled — queue	.	.

Per-family table — KL anchor verdict in bold

Family	1.7B base	Run 2 (no-KL)	Run 4 (+KL)	4B base	4B-instruct
event_planning μ	0.138	0.000 ❌	0.175 ✅✅	0.340	0.166
event_planning max	0.522	0.000	0.510	0.795	0.757
meeting_scheduling μ	0.153	0.130 ↑	0.064 ↓	0.287	0.000
meeting_scheduling max	0.500	0.725 ↑↑	0.350	0.819	0.000
medical_intake	0.000	0.000	0.000	0.000	0.000
support_triage	0.000	0.000	0.000	0.000	0.000

Refined narrative — this is the hackathon thesis confirmed:

1. GRPO without anchor causes capability collapse. Run 2 (β=0) drove event_planning from 0.138 → 0.000 mean, sacrificing breadth for one peak in meeting_scheduling. Avg_score regressed 0.067 → 0.029.
2. GRPO with KL anchor cleanly improves the protected family. Run 4 (β=0.2, lr=5e-7) on the same model recovered avg to 0.056 AND beat base on event_planning (0.138 → 0.175). The anchor literally fixed Run 2's regression.
3. The cost is peak capability. Run 4 dropped meeting_scheduling max from 0.725 (Run 2's gem) to 0.350. KL prevents the extreme specialization that Run 2 leaned on.
4. GRPO unlocks weak bases: 0.6B couldn't touch event (0 → 0.032 mean, 0 → 0.382 max). Real new capability.
5. Strong base sets the bar: Qwen3-4B base scores 0.1446 without any RL — beats 4B-Instruct (0.0399) on every solvable family. Whatever Qwen3 instruct-SFT did, it weakened reasoning on multi-turn tool-using setups. Open question: does GRPO+KL push 4B above this — Run 3 was canceled before we could find out (logged as future work in blog.md §7b).

In progress (right now)

• ✅ Run 4 weights mirror to anurag203/clarify-rl-run4-qwen3-1.7b-beta0.2 is LIVE. Upload completed 06:33 IST (8147 s = 2 h 16 min total elapsed for snapshot_download + upload_folder); model.safetensors resolves to a 6,882,335,328-byte (~6.41 GiB) presigned cas-bridge.xethub.hf.co URL with x-xet-cas-uid=public. Watcher PID 46586 autopiloted the cleanup at 06:33:03: stripped the fallback block from docs/model_cards/run4-qwen3-1.7b-beta0.2.md, committed edd1efe docs: drop Run 4 weights-mirror fallback note (mirror is live), pushed to GitHub main, and pushed the cleaned README onto the HF Hub model repo. Hero model is now self-contained on the personalized mirror — no fallback path needed.
• ✅ Submission lap CLOSED: 3 trained runs evaluated, 4B base eval as ceiling, all plots regenerated, all docs reconciled, Wild Card #5 + Judges-60s-Tour + plot captions live on README and env Space.
• ✅ All 25 logged-out submission URLs → HTTP 200/302
• ✅ Env Space /health + /reset → real OpenEnv structured responses

Decision dial: scoreboard of expected vs actual

1. ✅ Run 4 mean (0.056) ≈ 1.7B base (0.067) → KL anchor recovered most of the regression. +0.027 vs Run 2 no-KL.
2. ❌ Run 4 max(meeting) = 0.350 < Run 2's 0.725 → KL anchor traded the peak for breadth. Expected.
3. ✅ Run 4 event_planning μ (0.175) > 1.7B base (0.138) → KL not just preserved but improved the family Run 2 destroyed. Surprise upside.
4. ➖ Run 3 (4B GRPO) → not run (HF Jobs queue saturation, canceled at 48 min). Logged as future work; not blocking the submission since Run 4 already validates the central thesis on 1.7B.

Next step (default if user just says "continue")

1. ✅ All P0/P1 hackathon gates GREEN — auto-validator + Themes & Judging Criteria + FAQ + Help Guide cross-checked
2. ✅ Final docs reconciled — README, env Space README, slides, blog, model cards all synced
3. ✅ Logged-out smoke test of 25 URLs + Env Space functional smoke both PASSED
4. NEXT (optional polish before 5 PM IST): (a) capture an assets/demo_chat_screenshot.png from the demo Space "Live chat" tab and embed it in README — improves "Storytelling" 30% gate; (b) confirm Run 4 mirror upload finished (hf_hub_url returns 200 on model.safetensors) and remove the fallback note from the model card; (c) submit through the official Discord submission form once the form opens.
5. HARD STOP: Apr 26, 2026, 5:00 PM IST.

Open questions / blockers

• 🟢 vLLM-in-HF-Job eval pipeline solid (Run-1: 6.7 min, Run-2: 113 sec, Run-4: 7.8 min, 4B base: ✅)
• 🟢 Auto-eval state persisted to outputs/auto_eval_state.json — restart safe
• 🟢 Token plumbing for 3 accounts working
• 🟢 Plots auto-skip Run 3 because eval JSON never landed — no manual cleanup required

Files most recently touched (this session)

• outputs/runs.json — Run 3 marked CANCELED_QUEUE_SATURATION with reason; Run 4 marked completed with eval job id
• outputs/auto_eval_state.json — Run 3 = CANCELED, Run 4 = COMPLETED
• outputs/run4_artifacts/log_history_partial.json — full 300-step log scraped from live job
• outputs/run_artifacts/1.7B-KL/evals/ — Run 4 eval JSON pulled from Hub
• outputs/run_artifacts/4B-base/evals/ — 4B base eval (real ceiling)
• docs/blog.md, docs/trace_demo.md, docs/STATUS.md, README.md — KL-anchor narrative; Run 3 → future work
• plots/01–07_*.png + plots/runs_summary.json — regenerated against the 7-row scoreboard
• scripts/compare_runs.py, scripts/refresh_all_plots.sh, scripts/watch_and_eval.py, scripts/poll_status.sh — orchestration kept (Run 3 specs gated on file existence, no-op if missing)

Locked decisions

• Idea: ClarifyRL — train asking-vs-guessing via RL
• Theme: #5 Wild Card (primary) + #3.2 + #2
• 5 families: coding / medical-intake / support-triage / meeting / event
• Stack: OpenEnv 0.2.2 + MCPEnvironment + TRL GRPO ≥1.0 + Qwen3 family (0.6B / 1.7B / 4B)
• Compute: HF Jobs, 3 successful trained runs + 1 base eval across 3 accounts; spend ≈ Run-1 $1.08 + Run-2 $2.21 + Run-3 v3 (OOM) $0.40 + Run-3 v5 (canceled in queue, $0.00) + Run-4 $1.80 + 4B base eval $0.13 + Run-4 eval $0.20 = ~$5.8 of $120 budget
• Submission format: HF blog post (markdown) + GitHub repo + env Space + demo Space + W&B project + Colab notebook
• MCP tools: ask_question, propose_plan, get_task_info
• Deadline: Apr 26, 2026, 5:00 PM IST (~9.5 hours from this update)