Add brains/sovereign-v3/train_notes.md

brains/sovereign-v3/train_notes.md

@@ -0,0 +1,161 @@
+# SOVEREIGN v3 — Training Doctrine (~272M dense)
+
+Exact recipe for **2× T4 (16 GB)**, shared shard format
+`rl/shards_v2/*.npz`. Memory ≈9.4 GB/GPU at bs 16 + fp16 (see
+ARCHITECTURE.md §3). Everything is implemented by `model.py` +
+`data_adapters.py`.
+
+---
+
+## Stage A — Return-conditioned filtered BC + dynamics (offline, ~2–3 epochs)
+
+### A.1 Sample preparation (`ShardPrepper`)
+
+1. **Returns / RTG**: per-episode `R_t = r_t + γR_{t+1}`, γ=0.995;
+   +2 survive / +5 win terminal bonuses; z-scored per shard. **rtg = R̄** is
+   the conditioning input; `rtg_next` shifts within the episode (last
+   frame repeats).
+2. **Weights** `w = clip(exp(R̄/τ), 0.2, 8)` τ=1; ×4 survivor episodes;
+   ×2 ±2-frame kill windows; mean-normalized. (AWR soft filtering.)
+3. **Free-label targets**: `lab64`, `lab64_next` (majority-vote 64×64),
+   `threat`/`expand` intent mirrors, `gate_mask`/`gate_valid`,
+   `win_lab` (survived=1), `nums_next`.
+4. **Arena filtering**: drop >85%-water episodes.
+5. Frame stacks / sequence windows: opt-in (`stack=`, `build_sequences`).
+
+### A.2 Loss (exact)
+
+```
+L = w · [ λk CE(kind; ls=0.05)
+       + λc CE(cell_logits[·,kind], cell)
+       + λp −logBeta(α,β; pct)·1[kind≠bank]
+       + λv Huber(value, R̄)
+       + λw CE(win_logits, win_lab; ls=0.05)
+       + λg BCE(gate, gate_mask)·1[valid]
+       + λs CE(seg64, lab64)
+       + λd CE(next_seg, lab64_next)              # dynamics
+       + λt BCE(threat, threat) + λx BCE(expand, expand)   # intent mirrors
+       + λe smoothL1(econ, cat(nums_next, rtg_next))
+       − λent (H(kind)+H(cell)+H(pct)) ]
+λ = (0.5, 1.0, 0.25, 0.1, 0.1, 0.5, 0.5, 0.4, 0.2, 0.2, 0.05, 0.003)
+```
+
+**Value option (IQL-expectile)**: for the value head only, replace Huber
+with the τ=0.9 expectile loss `|τ−1[δ<0]|·δ²` — more robust to the weak
+farmer buffer. Both are supported; ablation #6 decides.
+
+### A.3 Optimizer & schedule
+
+- AdamW lr **3e-4 → 3e-5** cosine, **300-step warmup** (cortex), wd 1e-4
+  (Linear only), grad clip 5.0, **AMP fp16**, bs **16/GPU × accum 2**.
+- 2–3 epochs. Smoke test first: `python model.py` (prints the 272M
+  breakdown), `python data_adapters.py` (adapter shapes).
+
+---
+
+## Stage B — RPO: return-conditioned PPO (online, in sim)
+
+### B.1 Rollouts
+
+- `model.act(..., rtg=rtg_actual)` per tick; store `logprob`, `value`,
+  `win_prob`. **8 envs/GPU × 256 ticks** on CPU workers (2×T4 ⇒ ~2048
+  samples/round).
+- Memory mode (optional): carry `state` through each episode
+  (truncated-BPTT); the env wrapper sketch:
+
+```python
+# env wrapper sketch (mem mode)
+s = None
+for step in range(ep_len):
+    rgb_t, nums_t = env.obs()
+    a = model.act(rgb_t, nums_t, rtg=rtg_est, state=s)
+    s = a["state"]            # (1, B, 768)
+    obs, r, done = env.step(a["kind"], a["cell"], a["pct"])
+```
+
+### B.2 RPO specifics
+
+- **RTG-consistent advantages**: `A_t = R_t − V(s_t, R_t)` (value is
+  rtg-conditioned — that is the whole point; the critic never has to
+  guess the aspiration).
+- GAE(γ=0.995, λ=0.95) over `A_t`; z-normalize; clip ε=0.2 on the full
+  action logprob.
+- Value loss: 0.5×clipped Huber (double clip ±0.2 in return space) +
+  **win head CE on episode outcomes** + optional cell-value auxiliary
+  (Huber vs per-cell utility bootstrap — PPO-friendly spatial critic).
+- 4 epochs/round, minibatch 64/GPU, AdamW lr 3e-4 + cosine tail, grad
+  clip 5.0, **AMP fp16 + fp32 master**.
+- **KL-to-pretrain guard β=0.1** (frozen Stage-A prior); roll-back if mean
+  KL > 0.15 for 3 rounds (β→0.02). ICLR 2025 evidence: pretrain-init +
+  guard, no offline replay retention — do NOT mix shards into the PPO
+  batches.
+
+### B.3 Aspiration at play time
+
+Eval/deploy with `rtg = +1.5` z-units (above-survival ambition; tune over
+{0.5, 1.0, 1.5, 2.0} — ablation #1). The BC stage never saw rtg>2σ, so
+stay inside the trained range.
+
+### B.4 Reward shaping (flag the sim edit)
+
+`+0.004/tick alive, +0.4/kill, +2 survive timeout, +8 win`. Keep
+intermediates small; the win is the judged objective.
+
+### B.5 League-lite (fictitious self-play, cheap)
+
+Keep the top-3 checkpoints by win-rate. Every 10 rounds, evaluate the
+current policy against the pool; promote if it wins >55%. Once promoted,
+the sim runs our policy vs 7 bots + 2 frozen past selves — the only
+"league" a 2×T4 budget can afford, and it directly targets the weakness
+of the last-survivor objective (myopic bot-exploit policies).
+
+---
+
+## Stage C — Evaluation + wake-sleep loop
+
+- 100+ episodes vs **9 mixed bots**, fixed seeds; metric **win-rate +
+  mean rank**; save best on rank; report binomial CI.
+- **Wake-sleep distill**: after PPO plateaus, distill the 272M policy into
+  a ~30M dense student (same heads, distilled heat/kind/pct/value with KL;
+  `return_all=True` provides all teacher targets). Use the student for
+  2× faster rollouts and richer league self-play data; the big teacher
+  trains from it. Repeat (cap 3 cycles).
+
+## Curriculum
+
+Epoch 1 of Stage A on early-game frames (tick_frac < 0.35) → full data →
+×1.5 weight on tick_frac > 0.6 in the final pass.
+
+## Ablations (run before believing anything)
+
+| # | Question | Change | Resolves |
+|---|---|---|---|
+| 1 | Aspiration rtg at play time | {0.5, 1.0, 1.5, 2.0} | the RPO knob |
+| 2 | Return conditioning itself | rtg≡0 (frozen) vs conditioned | Bet 1's real value |
+| 3 | Intent mirrors | λt=λx=0 vs default | opponent modeling |
+| 4 | Dynamics head | λd=0 vs default | world-model value |
+| 5 | Memory | GRU off vs on (seq BPTT) | temporal credit |
+| 6 | Value objective | Huber vs τ=0.9 expectile | robustness |
+| 7 | Win head | on/off in Stage B loss | endgame behavior |
+| 8 | League-lite | pool vs no pool | self-play payoff |
+| 9 | Wake-sleep | 0 vs 3 distill cycles | scaling loop |
+
+## Expected compute (2× T4, fp16)
+
+- Stage A: ~8–14 h for the current GB-scale shard set (272M, bs 16/accum 2).
+- Stage B: ~2–3 updates/min at minibatch 128 → one full experiment per
+  ~1–2 days. That is the honest cost of the mandate; the sanctioned trims
+  (in order): `head_hidden` 3072→2048, `cortex_layers` 6→4, `enc_ch` top
+  832→640. Each keeps the total >200M.
+
+## Risk register
+
+| Risk | Mitigation |
+|---|---|
+| fp16 NaN in attention | fp32 master, grad clip 5, warmup; fall back fp32 for Stage A |
+| Unlearning the BC prior | KL guard + roll-back (B.2); no offline replay retention |
+| Aspiration OOD (rtg>2σ) | clamp rtg to ±2.5; ablation #1 |
+| Threat/expand labels are imbalanced | BCE weights 8:1 positive:negative |
+| GRU rollout complexity | default off; env sketch included |
+| Activation memory at bs 16 | `grad_ckpt=True` halves the heavy stages |
+| Color-identity prior (red=me) | sim-specific; documented |