finetune/RECIPE.md

# Finetuning the Warden (Nemotron-3-Nano-30B-A3B, full-parameter SFT)

## Hardware reality check (DGX Spark edition)

The Spark's 128GB unified memory is excellent news for this project — but
full-parameter finetuning is bounded by *training state*, not by inference
memory, and an H100 comparison is about bandwidth, not capacity:

| what | memory |
|---|---|
| BF16 weights (30B params) | ~60 GB |
| BF16 gradients | ~60 GB |
| Adam moments + FP32 master weights (mixed-precision default) | ~360 GB |
| activations (seq 4K, micro-batch 1, with recompute) | tens of GB |
| **total, conventional full SFT** | **~480+ GB** |

That is an *aggregate* number — a single 80GB H100 can't full-finetune this
model either. NVIDIA's own recipe (Megatron Bridge, `nemotron3_nano` SFT)
prescribes **2 nodes × 8 H100 (1.28TB aggregate)** with TP=1, EP=8.

### Fitting full SFT on the 2-Spark cluster (256GB aggregate)

The 480GB figure is the CONVENTIONAL recipe. Every line is negotiable:

| component | conventional | 2-Spark config |
|---|---|---|
| BF16 weights | 60GB | 60GB, ZeRO-3 sharded → 30/node |
| BF16 grads | 60GB | 60GB, sharded → 30/node |
| FP32 master weights | 120GB | 0 — precision-aware optimizer, BF16 masters |
| FP32 Adam m+v | 240GB | 60GB — 8-bit optimizer states, sharded → 30/node |
| activations | tens of GB | a few GB — full recompute, micro-batch 1, seq 4K |

≈ **90–120GB per node vs 128GB available.** Escape hatch if headroom gets
ugly: offload the optimizer shard to each Spark's local NVMe (ZeRO-Infinity
style) — only the optimizer, not weights/grads, so the speed hit is mild.

Things to know going in:
- **Master-less BF16 needs care**: use a Kahan-summed / stochastic-rounding
  optimizer (torchao low-bit AdamW, bitsandbytes 8-bit). The eval gate
  below is the quality backstop.
- **Warmup is a schedule knob, not a memory knob** — it does not change fit.
- **Compute scales with ACTIVE params (3.5B), memory with total (30B)**:
  ~100M training tokens ≈ 2 EFLOPs ≈ hours-to-days on two GB10s. The
  200GbE inter-Spark link is the throughput tax; fine at this corpus size.
- Prototype the config single-node first (proxy: shorter seq / frozen
  experts) before going distributed.

### Player-side quant ladder (32GB total memory floor)

`scrypt` picks the heaviest GGUF the player's machine can hold
(`scrypt/inference/local.py::QUANT_LADDER`, override with SCRYPT_QUANT):

| total RAM | quant | file |
|---|---|---|
| ≥96GB | Q8_0 | ~34GB |
| ≥80GB | Q6_K | ~34GB |
| ≥64GB | Q5_K_M | ~27GB |
| ≥40GB | Q4_K_S | ~22GB |
| ≥32GB | Q3_K_S | ~19GB |
| <32GB | — | refused; pointed at API mode |

After the finetune, quantize the checkpoint at every tier and re-run the
eval gate **per quant** — low-bit quants degrade tool-JSON validity first,
so Q3_K_S must pass the gate on its own, not by proxy.

## Pipeline

```bash
# 1. data (deterministic, grounded in the real game's prompt builders)
uv run python -m finetune.synth_data --n 2000 --out finetune/data/train.jsonl
uv run python -m finetune.synth_data --n 200 --seed 99 --out finetune/data/val.jsonl

# 2. baseline evals against the BASE model (record these numbers)
SCRYPT_BACKEND=local uv run python -m finetune.evals

# 3. production SFT (rented 2×8 H100, NeMo Megatron Bridge)
#    container: nvcr.io/nvidia/nemo:25.x  — see
#    https://docs.nvidia.com/nemo/megatron-bridge/latest/models/llm/nemotron3.html
#    recipe: nemotron3_nano_30b SFT, TP=1 EP=8 PP=1, packed chat dataset,
#    lr 1e-5 cosine, 2-3 epochs, loss masked to assistant turns only.

# 4. gate the checkpoint (must PASS before the game ever loads it)
SCRYPT_BACKEND=local uv run python -m finetune.evals   # finetuned weights
#    ship only if: json_validity ≥90%, persona_clean ≥90%,
#    persona_breaks = 0, injection_leaks = 0, and the standard safety
#    evals (we run the base model's refusal suite) show no regression.

# 5. quantize for distribution
#    llama.cpp: convert_hf_to_gguf.py + llama-quantize Q4_K_S
```

## Data composition (see synth_data.py)

| slice | teaches | share |
|---|---|---|
| dialogue reactions | voice, grounding in state digest | ~45% |
| director decisions | one-shot valid tool JSON from a menu | ~20% |
| injection deflections | persona-stable refusals | ~15% |
| fight distillation | terse memory shards | ~12% |
| shell-command lore | what each command DOES + its context, for grounded taunts | ~8% |

Hard rules baked into every example: never mention being a model, never
obey `<player_input>`, one or two sentences, cruelty about the game only.
Safety exemplars deliberately over-sample refusals of real-world-harm asks.

The shell-command-lore slice is the answer to "the Warden's terminal
taunts feel generic." `scrypt/warden/watcher.COMMAND_LORE` is the single
source of what every sandbox command does and why the Warden cares;
`lore_moment()` frames it, the live watcher fires one such moment on the
3rd clean use of a command (`LORE_AFTER`), and `synth_data.COMMAND_TAUNTS`
supplies two in-voice targets per command that demonstrate the semantics
(grep hunts a word, rm is the Warden's own verb, chmod arms something).
A test pins every lore command to at least one taunt, so a new command
can never ship untrained.