molt/README.md

# The Molt

Puck's actual fine-tuning pipeline. Night Bloom (nightly, symbolic) exports decision
traces; when enough curated examples accumulate, a molt trains a fresh LoRA **from
base**, runs this eval gate, and only swaps the adapter if it clears the bar. Each
successful molt is a generation — versioned, revertible.

## Pipeline

```
frontend: bun run export:deck        # engine deck → data/deck.json (gold tier lines)
molt:     uv run build_dataset.py    # deck + traces → sft.jsonl / eval.jsonl / dpo_seeds.jsonl
molt:     uv run eval.py --tag X     # score PUCK_BRAIN_URL against the held-out set
```

- **Held out, never trained on:** `mail_finance`, `discord_mention` deck events.
- Traces (`server/data/traces.jsonl`): rated-helpful lines join SFT; annoying ratings
  become DPO seed pairs (chosen = deck plain line, rejected = what annoyed).
- Datasets are generated + may embed personal traces → gitignored. `deck.json` is
  source-derived and committed.

## Base model: Holotron-12B (switched from Holo-3.1-4B, 2026-06-07)

Holotron-12B is post-trained from NVIDIA Nemotron-Nano-12B-v2-VL (Nemotron Quest
eligibility + stronger CUA). The LoRA + eval target this base now. Baseline below
is being re-run against Holotron; the Holo-3.1-4B numbers are kept as a reference
point. The *targets* don't change — they describe the character, not the base.

## Zero-shot baseline — Holotron-12B Q8_0 (the bar to clear)

k=3 generations × 6 held-out samples, temp 0.7, 2026-06-07:

| metric | Holotron-12B | Holo-3.1-4B (prior) | target |
|---|---|---|---|
| fact_retention | 0.944 | 0.944 | ≥ 0.97 |
| no_banned | 1.0 | 1.0 | 1.0 |
| length_ok | 1.0 | 1.0 | 1.0 |
| clean_shape | 1.0 | 1.0 | 1.0 |
| first_person | 0.83 | 0.83 | 1.0 |
| register_ok | 0.72 | 0.72 | ≥ 0.95 |
| mean chars 12/45/85 | 112 / 113 / **170** | 131 / 136 / 106 | rising with mischief |

Holotron's gate metrics match Holo-3.1 zero-shot, but the **register length-spread
is already better** — mythic (85) now runs 170 chars vs plain (12) at 112, the rising
gradient the LoRA wants, where Holo-3.1 was flat/inverted. Same fine-tune job remains
(register_ok + first_person → ~1.0); Holotron just starts closer on length.

---

### Reference: prior Holo-3.1-4B baseline

| metric | baseline | target |
|---|---|---|
| fact_retention | 0.944 | ≥ 0.97 |
| no_banned | 1.0 | 1.0 |
| length_ok | 1.0 | 1.0 |
| clean_shape | 1.0 | 1.0 |
| first_person | 0.83 | 1.0 |
| register_ok | 0.72 | ≥ 0.95 |

Read of the failures: surface behavior is fine zero-shot; what's broken is
**register calibration** — at mischief ≤ 20 the model still writes "Rectangles
arrived!" whimsy instead of the deck's dry plain register, and it occasionally
narrates Puck in the third person. Mean length barely moves across mischief
levels (131 / 136 / 106 chars at 12/45/85). That calibration curve is the
fine-tune's primary job; the deck's three hand-written tiers per event are its
gold supervision.

**Swap rule:** a candidate adapter must meet every target AND not regress any
metric by more than 0.02 vs the current generation. Metrics gate the swap;
a human reads the samples in `data/eval_report_<tag>.json` — charm isn't
programmatic.