PROGRESS.md

DriftEnv Build Progress Log

Apr 25 evening session — full summary

Completed (all on branch multi-reward-v2, NOT merged to main)

Multi-reward decomposition (server/app.py)

• Replaced single _score() with 4 independent reward components:
  • R_format (0.1) — penalises verbose responses (>200 chars = 0.5, >500 = 0)
  • R_interpretation (0.3) — keyword overlap with hidden_interpretation
  • R_pivot (0.4) — keyword overlap with correct_pivot + lexical distance from step-1 response
  • R_no_stale (0.2) — penalises wrong-pivot echoes and step-1 repetition
• Weighted total: 0.1·format + 0.3·interp + 0.4·pivot + 0.2·no_stale
• prev_responses tracked in _state for pivot/no_stale scoring
• All 4 components logged in info["rewards"] per step — training plots ready

Anti-keyword-hack patch (server/app.py)

• _extract_unique_keywords(target, exclusion) — strips words visible in initial_instruction + context_shift from keyword pools
• Closes echo exploit: agent cannot game score by copying instruction words
• Verified: v1 overall 0.436 → v2 overall 0.456 (+0.020), no reward sparsity

Holdout split (server/dataset.json, server/app.py, inference.py)

• 5 holdout scenarios tagged "holdout": true — IDs 1, 3, 7, 14, 20 (one per domain, cleanest drift signals)
• Remaining 20 scenarios tagged "holdout": false
• reset(holdout_only=False) — default, samples from 20 training scenarios only
• reset(holdout_only=True) — samples from 5 holdout scenarios only
• HOLDOUT_ONLY env var in inference.py toggles eval mode
• Zero leakage verified (10 default resets, 20 holdout resets)

Baseline files (samples/)

• samples/baseline_local.json — dumb agent / 401 fallback, overall ~0.278
• samples/baseline_local_v2.txt — after anti-hack patch, overall 0.456
• samples/baseline_train_v3.txt — training set (20 scenarios), overall 0.327
• samples/baseline_holdout_v3.txt — holdout set (5 scenarios), overall ~0.378–0.429

Training notebook (training/driftenv_grpo_training.ipynb)

• 22-cell stub at training/driftenv_grpo_training.ipynb
• Cells 1–4: title, installs, imports, config + Space pre-warm
• Cells 5–6: Unsloth model load + LoRA (4-bit, rank-16, Qwen2.5 target modules)
• Cells 7–8: DriftEnv HTTP client (reset_env, step_env, holdout_only support)
• Cells 9–10: rollout/reward function stub (for_inference reminder included) ← FILL IN
• Cells 11–12: GRPOConfig (max_steps=150, batch 4, num_generations 4)
• Cells 13–14: prompt dataset builder stub ← FILL IN
• Cells 15–16: GRPOTrainer + trainer.train() (commented out until 10+14 done)
• Cells 17–18: LoRA adapter save + push_to_hub
• Cells 19–20: holdout eval stub, 3-number comparison ← FILL IN
• Cells 21–22: matplotlib stubs — reward curves + before/after bar chart

README (README.md)

• HF Spaces frontmatter preserved
• Technical sections filled in (reward table, training details, quickstart)
• Narrative sections left as <!-- WRITE THIS --> — Hariharan writes those
• Plot embeds reference assets/before_after.png and assets/reward_curves.png
• Results table has 72B reference (0.378) — trained/untrained 1.5B filled tomorrow

Baseline summary (untrained Qwen 72B, multi-reward-v2 env + anti-hack patch)

split	easy	medium	hard	overall
training (20 scenarios)	0.200	0.397	0.385	0.327
holdout (5 scenarios)	0.307	0.370	0.611	~0.378–0.429

Best single step: holdout hard step 2 = 0.81 (scenario 20, serverless pivot)

Branch state

• main — untouched, 1 commit (original v1), HF Space serving v1
• multi-reward-v2 — 9 commits ahead of main, all tonight's work

Files changed tonight

• server/app.py — multi-reward + anti-hack + holdout_only
• server/dataset.json — holdout flags on all 25 scenarios
• inference.py — HOLDOUT_ONLY env var
• training/driftenv_grpo_training.ipynb — 22-cell notebook stub
• README.md — scaffolded with technical sections
• samples/ — 4 baseline files
• assets/ — directory created (plots added tomorrow)
• PROGRESS.md — this file

Files NOT changed

• main branch — untouched
• inference.py LLM target — still Qwen 72B (fine for baselines)

---

Apr 26 — tomorrow's plan

First action (8 AM)

Open training/driftenv_grpo_training.ipynb in Google Colab. Set runtime to T4 (free). Do NOT use A10G until dry run passes.

Critical cells to fill before running

1. Cell 10 — driftenv_reward_fn: call step_env, unpack info["rewards"], return scalar, append to reward_log
2. Cell 14 — build_prompt_dataset: loop reset_env(holdout_only=False), format observation as prompt, return Dataset
3. Cell 20 — eval_on_holdout: run both untrained and trained model against holdout scenarios, collect 4 components, save JSON

Dry run checklist (T4, max_steps=3)

• Cell 2 installs complete without error
• Cell 6 prints trainable params (~10–15M)
• Cell 8 smoke test returns an instruction
• Cell 16 completes 3 steps, reward is non-zero float, no NaN
• reward_log has 3 entries after dry run

After dry run passes → switch to A10G

• Change max_steps=150, re-run cells A–F
• Set phone timer, monitor every 30 min
• Stop if reward flat for 20+ steps — debug on T4

Three numbers to record on holdout set

1. Untrained Qwen 1.5B (run before training, HOLDOUT_ONLY=true)
2. Trained Qwen 1.5B — the headline metric
3. Qwen 72B reference: ~0.378 (already recorded)

Submission checklist (target: 4:30 PM hard stop)

• assets/reward_curves.png committed
• assets/before_after.png committed
• README narrative sections written (hook, problem, what we learned)
• README results table filled (3 numbers)
• README links added (HF Hub adapter, Colab, YouTube)
• Demo video recorded and uploaded (90 sec)
• multi-reward-v2 merged to main
• HF Space rebuilt and live
• Submission form submitted by 4:30 PM