docs/replay_research.md

Replay research — why 6 adaptations failed the both-worlds gate, and the fix

Date: 2026-08-09. Situation: 25.4M liquid head, fluent TinyStories-era base, domain SFT/LoRA/DPO all collapse free-form fluency or verdicts. Gate: coherent free-form AND verdict accuracy.

The measured pattern (our own data)

• LoRA run i (domain-only 902 rows, 2 epochs, r16): ppl canary held 8.46 (base distribution intact) but free-form = soup, verdicts = true-collapse (0/12), full battery main 0.080 / researcher 0.056 vs DPO3@200 champion 0.286/0.167.
• Collaborator read: "not forgetting (ppl held), not capacity (DPO3 proves it) — adapter learned template shape, not reasoning substance; SFT/LoRA optimizes next-token, not verdict; DPO is the proven path."

What the research says (fresh sources, 2026-08-09)

1. arXiv 2502.06042 — "Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection": finetuning on LIMITED target data (i) overfits and (ii) drifts from the base. KEY: injecting PRETRAINING data into the finetuning mixture prevents both; "as little as 1% of pretraining data in the finetuning data mixture prevents the [drift]".
2. arXiv 2401.05605 — "Scaling Laws for Forgetting When Fine-Tuning LLMs": LoRA STILL suffers catastrophic forgetting; strong INVERSE LINEAR relation between fine-tune performance and forgetting; forgetting grows with update steps; "cannot be avoided through early stopping or varying the number of parameters fine-tuned." => higher LoRA rank / different adapter family does NOT fix the tradeoff; REPLAY (data mixing) is the lever.
3. Local lessons (cookbook/lessons_from.md): "balanced mixes + window-shuffle" fixed the same drift during PRETRAIN; full-param SFT destroyed fluency at 7.8M; LoRA adapts style but not reasoning at tiny scale.

What we were doing wrong (root cause)

• Adaptation ran on DOMAIN-ONLY data (902 gold rows ~= 115K assistant tokens). Tiny target set => overfit (format memorized, true-collapse) + drift (soup). No pretraining-data replay in the mixture. This is exactly the failure both papers quantify.
• Secondary: 8k tokenizer fragments rare domain words ("Stepartment", "Stepublication") — recorded as future 16k re-tokenize + continue-pretrain.

The fix (applied as LoRA run ii)

• Mix fluent pretraining tokens into every epoch: --replay-bin (train_phase2b.bin, the real pretrain corpus, NOT the val canary => guard stays honest), --replay-ratio 0.5, injected as raw full-loss items (train_lora.py).
• Keep: frozen base, r16/a32, KL anchor (bump 0.05 -> 0.1), ppl guard 60, early stop on best.pt (guard + sft val loss).
• Next gate: free-form sample vs base, verdict battery. If probes improve, adapter-only DPO (3,004 pairs) is unblocked (the champion path).