# DriftEnv Build Progress Log

## Apr 25 evening session — full summary

### Completed (all on branch `multi-reward-v2`, NOT merged to main)

#### Multi-reward decomposition (`server/app.py`)
- [x] Replaced single `_score()` with 4 independent reward components:
  - `R_format` (0.1) — penalises verbose responses (>200 chars = 0.5, >500 = 0)
  - `R_interpretation` (0.3) — keyword overlap with `hidden_interpretation`
  - `R_pivot` (0.4) — keyword overlap with `correct_pivot` + lexical distance from step-1 response
  - `R_no_stale` (0.2) — penalises wrong-pivot echoes and step-1 repetition
- [x] Weighted total: `0.1·format + 0.3·interp + 0.4·pivot + 0.2·no_stale`
- [x] `prev_responses` tracked in `_state` for pivot/no_stale scoring
- [x] All 4 components logged in `info["rewards"]` per step — training plots ready

#### Anti-keyword-hack patch (`server/app.py`)
- [x] `_extract_unique_keywords(target, exclusion)` — strips words visible in
  `initial_instruction` + `context_shift` from keyword pools
- [x] Closes echo exploit: agent cannot game score by copying instruction words
- [x] Verified: v1 overall 0.436 → v2 overall 0.456 (+0.020), no reward sparsity

#### Holdout split (`server/dataset.json`, `server/app.py`, `inference.py`)
- [x] 5 holdout scenarios tagged `"holdout": true` — IDs **1, 3, 7, 14, 20**
  (one per domain, cleanest drift signals)
- [x] Remaining 20 scenarios tagged `"holdout": false`
- [x] `reset(holdout_only=False)` — default, samples from 20 training scenarios only
- [x] `reset(holdout_only=True)` — samples from 5 holdout scenarios only
- [x] `HOLDOUT_ONLY` env var in `inference.py` toggles eval mode
- [x] Zero leakage verified (10 default resets, 20 holdout resets)

#### Baseline files (`samples/`)
- [x] `samples/baseline_local.json` — dumb agent / 401 fallback, overall ~0.278
- [x] `samples/baseline_local_v2.txt` — after anti-hack patch, overall 0.456
- [x] `samples/baseline_train_v3.txt` — training set (20 scenarios), overall **0.327**
- [x] `samples/baseline_holdout_v3.txt` — holdout set (5 scenarios), overall **~0.378–0.429**

#### Training notebook (`training/driftenv_grpo_training.ipynb`)
- [x] 22-cell stub at `training/driftenv_grpo_training.ipynb`
- [x] Cells 1–4: title, installs, imports, config + Space pre-warm
- [x] Cells 5–6: Unsloth model load + LoRA (4-bit, rank-16, Qwen2.5 target modules)
- [x] Cells 7–8: DriftEnv HTTP client (`reset_env`, `step_env`, `holdout_only` support)
- [x] Cells 9–10: rollout/reward function stub (`for_inference` reminder included) **← FILL IN**
- [x] Cells 11–12: `GRPOConfig` (max_steps=150, batch 4, num_generations 4)
- [x] Cells 13–14: prompt dataset builder stub **← FILL IN**
- [x] Cells 15–16: `GRPOTrainer` + `trainer.train()` (commented out until 10+14 done)
- [x] Cells 17–18: LoRA adapter save + `push_to_hub`
- [x] Cells 19–20: holdout eval stub, 3-number comparison **← FILL IN**
- [x] Cells 21–22: matplotlib stubs — reward curves + before/after bar chart

#### README (`README.md`)
- [x] HF Spaces frontmatter preserved
- [x] Technical sections filled in (reward table, training details, quickstart)
- [x] Narrative sections left as `<!-- WRITE THIS -->` — Hariharan writes those
- [x] Plot embeds reference `assets/before_after.png` and `assets/reward_curves.png`
- [x] Results table has 72B reference (0.378) — trained/untrained 1.5B filled tomorrow

### Baseline summary (untrained Qwen 72B, multi-reward-v2 env + anti-hack patch)

| split | easy | medium | hard | overall |
|-------|------|--------|------|---------|
| training (20 scenarios) | 0.200 | 0.397 | 0.385 | **0.327** |
| holdout (5 scenarios)   | 0.307 | 0.370 | 0.611 | **~0.378–0.429** |

Best single step: holdout hard step 2 = **0.81** (scenario 20, serverless pivot)

### Branch state
- `main` — untouched, 1 commit (original v1), HF Space serving v1
- `multi-reward-v2` — 9 commits ahead of main, all tonight's work

### Files changed tonight
- `server/app.py` — multi-reward + anti-hack + holdout_only
- `server/dataset.json` — holdout flags on all 25 scenarios
- `inference.py` — HOLDOUT_ONLY env var
- `training/driftenv_grpo_training.ipynb` — 22-cell notebook stub
- `README.md` — scaffolded with technical sections
- `samples/` — 4 baseline files
- `assets/` — directory created (plots added tomorrow)
- `PROGRESS.md` — this file

### Files NOT changed
- `main` branch — untouched
- `inference.py` LLM target — still Qwen 72B (fine for baselines)

---

## Apr 26 — tomorrow's plan

### First action (8 AM)
Open `training/driftenv_grpo_training.ipynb` in Google Colab.
Set runtime to **T4 (free)**. Do NOT use A10G until dry run passes.

### Critical cells to fill before running
1. **Cell 10** — `driftenv_reward_fn`: call `step_env`, unpack `info["rewards"]`,
   return scalar, append to `reward_log`
2. **Cell 14** — `build_prompt_dataset`: loop `reset_env(holdout_only=False)`,
   format observation as prompt, return `Dataset`
3. **Cell 20** — `eval_on_holdout`: run both untrained and trained model against
   holdout scenarios, collect 4 components, save JSON

### Dry run checklist (T4, max_steps=3)
- [ ] Cell 2 installs complete without error
- [ ] Cell 6 prints trainable params (~10–15M)
- [ ] Cell 8 smoke test returns an instruction
- [ ] Cell 16 completes 3 steps, reward is non-zero float, no NaN
- [ ] `reward_log` has 3 entries after dry run

### After dry run passes → switch to A10G
- Change `max_steps=150`, re-run cells A–F
- Set phone timer, monitor every 30 min
- Stop if reward flat for 20+ steps — debug on T4

### Three numbers to record on holdout set
1. Untrained Qwen 1.5B (run before training, `HOLDOUT_ONLY=true`)
2. **Trained Qwen 1.5B — the headline metric**
3. Qwen 72B reference: **~0.378** (already recorded)

### Submission checklist (target: 4:30 PM hard stop)
- [ ] `assets/reward_curves.png` committed
- [ ] `assets/before_after.png` committed
- [ ] README narrative sections written (hook, problem, what we learned)
- [ ] README results table filled (3 numbers)
- [ ] README links added (HF Hub adapter, Colab, YouTube)
- [ ] Demo video recorded and uploaded (90 sec)
- [ ] `multi-reward-v2` merged to `main`
- [ ] HF Space rebuilt and live
- [ ] Submission form submitted by 4:30 PM