scripts/FINDINGS.md

# K3 Rental Validation — Durable Findings & Fixes

Hard-won root causes, patches, and decisions. Transient state (PIDs, SHAs, run progress)
lives in the session log, not here.

## 1. K3 + DSpark speculative decoding crash → FIXED (patch)

**Symptom:** llama-server with `--spec-type draft-dspark -md draft.gguf` loads fine,
reaches READY, but crashes on the FIRST decode:
```
llama-graph.cpp:1376: GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null") failed
```

**Root cause:** DSpark/dflash speculative decoding extracts intermediate "layer input"
features from the target (K3) model to feed the draft. It taps specific layers via
`cparams.embeddings_layer_inp[]` (the Lucebox draft GGUF requests `target_layer_ids =
[7, 23, 51, 67, 83]`). At `set_outputs()` time, llama.cpp asserts every requested layer
has a non-null `res->t_layer_inp[il]`. The kimi-k3 architecture graph builder
(`src/models/kimi-k3.cpp`) **never populates `res->t_layer_inp[]`** — so the assert fires.
Architectures that DO populate it (and thus support DSpark): `deepseek4.cpp`,
`bailingmoe3.cpp`, `gemma4.cpp`.

**Fix (applied on box `/root/llama.cpp`, backup at `kimi-k3.cpp.bak`):** in the kimi-k3
graph builder layer loop, immediately after `const auto & layer = model.layers[il];`:
```cpp
// expose the raw layer input for speculative draft (DSpark/dflash) feature taps
if ((size_t) il < cparams.embeddings_layer_inp.size() && cparams.embeddings_layer_inp[il]) {
    res->t_layer_inp[il] = inpL;
    cb(res->t_layer_inp[il], "layer_inp", il);
    ggml_build_forward_expand(gf, res->t_layer_inp[il]);
}
```
K3's `inpL` is already the plain per-layer input (no hyper-connection transform needed,
unlike deepseek4's `dsv4_hc_mean`). Draft taps are all at il<93, so no post-loop tail
extraction required. Rebuild: `cmake --build . --config Release -j 56 --target llama-server`.
Result: DSpark decodes on K3 without crash. **Upstream PR candidate.**

## 1b. DSpark draft fails every step "invalid token[1] = -1" → FIXED (GGUF mask token)

**Symptom:** After the t_layer_inp patch, the server loads + the main model decodes, but
the draft fails EVERY step: `init: invalid token[1] = -1` → `decode: failed to initialize
batch` → `llama_decode returned -1` → `draft: llama_decode returned -1`. Main model still
generates (falls back to no speculation) → runs at baseline speed with extra overhead,
zero spec speedup.

**Root cause:** DSpark builds the draft batch with a mask token for the multi-token block:
`common_batch_add(batch, i==0 ? dp.id_last : mask_token_id, ...)` (speculative.cpp:1189).
It resolves the mask id via `llama_vocab_mask(vocab)` which reads
`tokenizer.ggml.mask_token_id`. Our rewritten draft GGUF copied K3's tokenizer keys
(`fix_draft_gguf.py`) but K3 has NO mask token → `llama_vocab_mask()` returns -1 → batch
fed token -1 → embedding rejects it. The draft's TRAINED mask id lives in
`dflash.mask_token_id` = **163824** (separate metadata key, present in both original and
rewritten draft), but llama.cpp never reads it for the mask.

**Fix:** add `tokenizer.ggml.mask_token_id = 163824` (uint32) to the draft GGUF KV section
(`add_mask_token.py`, value taken from `dflash.mask_token_id`). Output
`draft_masked.gguf`; repoint `draft.gguf` symlink at it. Log then shows
`mask_token_id=163824` and zero draft errors. (Arguably also a mainline improvement:
fall back to `dflash.mask_token_id` when the vocab has no mask token.)

## 2. DSpark draft-model VRAM OOM → fixed with `-ngld 0`

The 2.4GB Q8_0 DSpark draft tried to allocate a 4GB KV cache on GPU device 1, which is
already near-full from the K3 trunk 8-way split. Fix: `-ngld 0` (`--gpu-layers-draft 0`)
runs the draft entirely on CPU. Draft is tiny; the main K3 trunk is the bottleneck anyway.

## 3. `-fa on` + large batch OOM on 16GB cards → reduced batch + device-0 share

With `--tensor-split 0.3,1,1,1,1,1,1,1 -b 512 -ub 512`, enabling `-fa on` OOMs the
compute pp buffers. Kitchen-sink config that reaches READY: `--tensor-split
0.2,1,1,1,1,1,1,1 -b 256 -ub 256 -fa on`. (Without `-fa on`, batch 512 + split 0.3 also
OOMs compute pp buffers; the no-FA auto path was the previously-working config.)

## 4. Load-time OOM (RssFile 482GB→cgroup 503GB) → `LLAMA_MMAP_NO_PREFETCH=1`

`src/llama-model.cpp:1663` calls `ml.init_mappings(true, ...)` → MAP_POPULATE +
MADV_WILLNEED eagerly faults all 1.5TB expert pages at load. Patch reads
`LLAMA_MMAP_NO_PREFETCH` env → `init_mappings(!no_prefetch, ...)`. Result: RssFile 124GB
at load, lazy LRU page-cache becomes the hot-expert cache (the architecture the user wants).
This patch is REQUIRED on the home 768GB box too — stock llama.cpp cannot load Q4_K_XL.

## 5. HF download throttling → presigned URL + aria2c

`curl -L` on huggingface.co/resolve is throttled to ~7KB/s on many datacenter routes.
Resolve without `-L`, extract the presigned CloudFront URL, download with `aria2c -x16`
→ 255-281 MB/s. (`lib/hf_direct.sh`.)

## 6. Bash `GROUPS` is special — never use it as a var name

`GROUPS` is a bash builtin array (user's group IDs). Using it for suite group selection
caused a silent no-op. Renamed to `SEL`.

## 7. The trunk is Q8_0, NOT 4-bit — and DSpark-on-CPU is a net loss (both fixed by Q4 trunk)

**Discovery (user's instinct was right):** "UD-Q4_K_XL" only 4-bit-quantizes the **routed
experts** (MXFP4). The entire **trunk — attention, shared experts (`_shexp`), output head,
token embedding — is Q8_0** (8-bit), plus F32 norms. Measured across all 32 shards:
- trunk Q8_0 = 59.6 GB (1116 tensors), norms F32 = 2.6 GB, experts = MXFP4 (rest of 1.4TB).

**Why:** K3 is QAT-trained in MXFP4 — the 4-bit experts ARE the reference model (no BF16
original). But the **non-expert path stays higher-precision** (activations MXFP8, non-expert
weights higher precision). So Q8_0 trunk is a legit DOWN-quant from a higher-precision
source, NOT an up-quant of 4-bit. Ref: dreaming.press "Kimi K3's Weights Are Already 4-Bit".
Consequence: re-quantizing the TRUNK Q8_0→Q4_K is valid (source was >4-bit). NEVER re-quant
the MXFP4 experts (destroys QAT calibration).

**DSpark-on-CPU measured result:** kitchen sink with `-ngld 0` (draft on CPU) gave
**ks_cold tg 0.224 t/s vs ~0.5 t/s no-spec baseline = ~2x LOSS**. Decode is pinned by CPU
expert execution; the CPU draft forward steals the same 112 threads. DSpark only wins with
the draft ON GPU, but the 58GB Q8_0 trunk fills all 8×16GB cards. → Need a smaller trunk.

**The unlock (both home fit + GPU DSpark):** requant trunk Q8_0→Q4_K.
- Trunk 59.6→31.6 GB; GPU-resident 62.2→34.2 GB. Fits home 2×3090 (48GB) AND frees ~28GB
  on the rental box → DSpark draft can go on GPU.
- **llama-quantize CANNOT do this safely**: `--allow-requantize` forces every non-overridden
  tensor (incl. MXFP4 experts) to the positional type → dequant+requant experts → QAT loss.
  Dry-run confirmed experts became q4_K/q6_K and total size GREW 1438771→1602481 MiB.
- **Solution: custom surgical rewriter `requant_trunk.c`** (compiled on box at
  `/root/k3-test/requant_trunk`). Per-shard, split-in=split-out. Byte-copies MXFP4 experts
  + F32 norms unchanged; dequant Q8_0→F32 + requant F32→Q4_K (ggml
  `dequantize_row_q8_0`/`quantize_row_q4_K_ref`) for trunk tensors only. Falls back to copy
  for tensors whose dims[0] not divisible by 256 (e.g. attn_k_b/ssm_f_b at 128).
  Validated on shard 2: 43 requant + 62 copy, 47.5→44.9 GB, no crash.

## Baselines (mainline, batch-1, 64in/64out unless noted)

- Best known (t112, b512/ub512, split 0.3,1..., FA-auto/off, no spec): cold pp ~0.30-0.40
  tg ~0.33-0.51; warm pp 2.5-2.8 tg 0.65-1.79 t/s.
- DSpark-on-CPU kitchen sink: cold pp 0.237 tg 0.224 t/s (~2x WORSE than no-spec).
- t56 regression: warm pp 0.62 tg ~0.49-0.59 → **t112 wins** (SMT siblings help).
- Decode is pinned by CPU expert execution, not cache warm-up (warm tg ≈ cold tg).

## Target architecture (home build)

- 2× 3090 24GB (48GB VRAM) + 768GB RAM + EPYC. Trunk Q8_0 58GB > 48GB → **Q4_K trunk
  requant (34.2GB incl. norms) fits with room for the DSpark draft on GPU.** This is the
  chosen path (see #7). Untestable for exact 2×3090 split on the 8×16GB rental box, but
  the Q4_K trunk + GPU-draft DSpark combo IS testable there (frees ~28GB VRAM).
- Experts: as many as fit in RAM (~150 hottest mlock-pinned, deferred), rest lazily
  faulted/evicted from SSD via kernel page-LRU (`--cpu-moe` + lazy mmap). Q4 only
  (no smaller quant — quality). Shared experts (`_shexp`) are GPU-resident, never evicted.
- Spec (DSpark, now working) + expert offload combined = the realistic production case.