molt/README.md

The Molt

Puck's actual fine-tuning pipeline. Night Bloom (nightly, symbolic) exports decision traces; when enough curated examples accumulate, a molt trains a fresh LoRA from base, runs this eval gate, and only swaps the adapter if it clears the bar. Each successful molt is a generation — versioned, revertible.

Pipeline

frontend: bun run export:deck        # engine deck → data/deck.json (gold tier lines)
molt:     uv run build_dataset.py    # deck + traces → sft.jsonl / eval.jsonl / dpo_seeds.jsonl
molt:     uv run eval.py --tag X     # score PUCK_BRAIN_URL against the held-out set

• Held out, never trained on: mail_finance, discord_mention deck events.
• Traces (server/data/traces.jsonl): rated-helpful lines join SFT; annoying ratings become DPO seed pairs (chosen = deck plain line, rejected = what annoyed).
• Datasets are generated + may embed personal traces → gitignored. deck.json is source-derived and committed.

Base model: Holotron-12B (switched from Holo-3.1-4B, 2026-06-07)

Holotron-12B is post-trained from NVIDIA Nemotron-Nano-12B-v2-VL (Nemotron Quest eligibility + stronger CUA). The LoRA + eval target this base now. Baseline below is being re-run against Holotron; the Holo-3.1-4B numbers are kept as a reference point. The targets don't change — they describe the character, not the base.

Zero-shot baseline — Holotron-12B Q8_0 (the bar to clear)

k=3 generations × 6 held-out samples, temp 0.7, 2026-06-07:

metric	Holotron-12B	Holo-3.1-4B (prior)	target
fact_retention	0.944	0.944	≥ 0.97
no_banned	1.0	1.0	1.0
length_ok	1.0	1.0	1.0
clean_shape	1.0	1.0	1.0
first_person	0.83	0.83	1.0
register_ok	0.72	0.72	≥ 0.95
mean chars 12/45/85	112 / 113 / 170	131 / 136 / 106	rising with mischief

Holotron's gate metrics match Holo-3.1 zero-shot, but the register length-spread is already better — mythic (85) now runs 170 chars vs plain (12) at 112, the rising gradient the LoRA wants, where Holo-3.1 was flat/inverted. Same fine-tune job remains (register_ok + first_person → ~1.0); Holotron just starts closer on length.

---

Reference: prior Holo-3.1-4B baseline

metric	baseline	target
fact_retention	0.944	≥ 0.97
no_banned	1.0	1.0
length_ok	1.0	1.0
clean_shape	1.0	1.0
first_person	0.83	1.0
register_ok	0.72	≥ 0.95

Read of the failures: surface behavior is fine zero-shot; what's broken is register calibration — at mischief ≤ 20 the model still writes "Rectangles arrived!" whimsy instead of the deck's dry plain register, and it occasionally narrates Puck in the third person. Mean length barely moves across mischief levels (131 / 136 / 106 chars at 12/45/85). That calibration curve is the fine-tune's primary job; the deck's three hand-written tiers per event are its gold supervision.

Swap rule: a candidate adapter must meet every target AND not regress any metric by more than 0.02 vs the current generation. Metrics gate the swap; a human reads the samples in data/eval_report_<tag>.json — charm isn't programmatic.