finetune/RECIPE.md

Finetuning the Warden (Nemotron-3-Nano-30B-A3B, full-parameter SFT)

Hardware reality check (DGX Spark edition)

The Spark's 128GB unified memory is excellent news for this project — but full-parameter finetuning is bounded by training state, not by inference memory, and an H100 comparison is about bandwidth, not capacity:

what	memory
BF16 weights (30B params)	~60 GB
BF16 gradients	~60 GB
Adam moments + FP32 master weights (mixed-precision default)	~360 GB
activations (seq 4K, micro-batch 1, with recompute)	tens of GB
total, conventional full SFT	~480+ GB

That is an aggregate number — a single 80GB H100 can't full-finetune this model either. NVIDIA's own recipe (Megatron Bridge, nemotron3_nano SFT) prescribes 2 nodes × 8 H100 (1.28TB aggregate) with TP=1, EP=8.

Fitting full SFT on the 2-Spark cluster (256GB aggregate)

The 480GB figure is the CONVENTIONAL recipe. Every line is negotiable:

component	conventional	2-Spark config
BF16 weights	60GB	60GB, ZeRO-3 sharded → 30/node
BF16 grads	60GB	60GB, sharded → 30/node
FP32 master weights	120GB	0 — precision-aware optimizer, BF16 masters
FP32 Adam m+v	240GB	60GB — 8-bit optimizer states, sharded → 30/node
activations	tens of GB	a few GB — full recompute, micro-batch 1, seq 4K

≈ 90–120GB per node vs 128GB available. Escape hatch if headroom gets ugly: offload the optimizer shard to each Spark's local NVMe (ZeRO-Infinity style) — only the optimizer, not weights/grads, so the speed hit is mild.

Things to know going in:

• Master-less BF16 needs care: use a Kahan-summed / stochastic-rounding optimizer (torchao low-bit AdamW, bitsandbytes 8-bit). The eval gate below is the quality backstop.
• Warmup is a schedule knob, not a memory knob — it does not change fit.
• Compute scales with ACTIVE params (3.5B), memory with total (30B): ~100M training tokens ≈ 2 EFLOPs ≈ hours-to-days on two GB10s. The 200GbE inter-Spark link is the throughput tax; fine at this corpus size.
• Prototype the config single-node first (proxy: shorter seq / frozen experts) before going distributed.

Player-side quant ladder (32GB total memory floor)

scrypt picks the heaviest GGUF the player's machine can hold (scrypt/inference/local.py::QUANT_LADDER, override with SCRYPT_QUANT):

total RAM	quant	file
≥96GB	Q8_0	~34GB
≥80GB	Q6_K	~34GB
≥64GB	Q5_K_M	~27GB
≥40GB	Q4_K_S	~22GB
≥32GB	Q3_K_S	~19GB
<32GB	—	refused; pointed at API mode

After the finetune, quantize the checkpoint at every tier and re-run the eval gate per quant — low-bit quants degrade tool-JSON validity first, so Q3_K_S must pass the gate on its own, not by proxy.

Pipeline

# 1. data (deterministic, grounded in the real game's prompt builders)
uv run python -m finetune.synth_data --n 2000 --out finetune/data/train.jsonl
uv run python -m finetune.synth_data --n 200 --seed 99 --out finetune/data/val.jsonl

# 2. baseline evals against the BASE model (record these numbers)
SCRYPT_BACKEND=local uv run python -m finetune.evals

# 3. production SFT (rented 2×8 H100, NeMo Megatron Bridge)
#    container: nvcr.io/nvidia/nemo:25.x  — see
#    https://docs.nvidia.com/nemo/megatron-bridge/latest/models/llm/nemotron3.html
#    recipe: nemotron3_nano_30b SFT, TP=1 EP=8 PP=1, packed chat dataset,
#    lr 1e-5 cosine, 2-3 epochs, loss masked to assistant turns only.

# 4. gate the checkpoint (must PASS before the game ever loads it)
SCRYPT_BACKEND=local uv run python -m finetune.evals   # finetuned weights
#    ship only if: json_validity ≥90%, persona_clean ≥90%,
#    persona_breaks = 0, injection_leaks = 0, and the standard safety
#    evals (we run the base model's refusal suite) show no regression.

# 5. quantize for distribution
#    llama.cpp: convert_hf_to_gguf.py + llama-quantize Q4_K_S

Data composition (see synth_data.py)

slice	teaches	share
dialogue reactions	voice, grounding in state digest	~45%
director decisions	one-shot valid tool JSON from a menu	~20%
injection deflections	persona-stable refusals	~15%
fight distillation	terse memory shards	~12%
shell-command lore	what each command DOES + its context, for grounded taunts	~8%

Hard rules baked into every example: never mention being a model, never obey <player_input>, one or two sentences, cruelty about the game only. Safety exemplars deliberately over-sample refusals of real-world-harm asks.

The shell-command-lore slice is the answer to "the Warden's terminal taunts feel generic." scrypt/warden/watcher.COMMAND_LORE is the single source of what every sandbox command does and why the Warden cares; lore_moment() frames it, the live watcher fires one such moment on the 3rd clean use of a command (LORE_AFTER), and synth_data.COMMAND_TAUNTS supplies two in-voice targets per command that demonstrate the semantics (grep hunts a word, rm is the Warden's own verb, chmod arms something). A test pins every lore command to at least one taunt, so a new command can never ship untrained.