---
license: other
tags:
- kimi-k3
- dflash
- dspark
- speculative-decoding
- mtp
- gguf
- llama-cpp
---
# K3-Stuff

Artifacts + tooling for running **Kimi K3** (2.78T MoE, 896 routed experts, MXFP4 QAT) with DSpark/DFlash speculative decoding in llama.cpp, validated on rented 8× RTX 5060 Ti hardware before committing to a 2× RTX 3090 home build.

## Draft model (root)
- **K3-DSpark-draft-Q8_0.gguf** — K3 DSpark/DFlash MTP draft, Q8_0, rewritten for llama.cpp:
  - arch renamed `dflash-draft` → `dflash`, tensor names fixed, K3 tokenizer keys added
  - `tokenizer.ggml.mask_token_id=163824` added (from `dflash.mask_token_id`) — required by llama.cpp `speculative.cpp`
- **fix_draft_gguf.py** — original GGUF rewrite (KV/tensor renames, tokenizer keys from K3)
- **add_mask_token.py** — adds `mask_token_id` to the rewritten draft

## `llama-cpp-patches/` — modified llama.cpp (apply with `git apply` or `patch -p1`)
Three changes, base commit ~b10630 (`d222767c7`):
1. **`llama-arch.cpp`** — accept `dflash-draft` as an alias for the `dflash` arch.
2. **`llama-model.cpp`** — **`LLAMA_MMAP_NO_PREFETCH=1`** env disables `MAP_POPULATE`/prefetch at load. Without it llama.cpp faults in the whole 1.4TB model and OOMs when RAM < model size. With it, the page cache becomes the hot-expert cache (lazy fault-in + kernel LRU eviction). **Required for any sub-full-RAM K3 box.**
3. **`kimi-k3.cpp`** — populate `res->t_layer_inp[]` for the layers DSpark taps (7/23/51/67/83 via `cparams.embeddings_layer_inp`). Without this, DSpark crashes at `llama-graph.cpp: GGML_ASSERT(t_layer_inp[il] != nullptr)`. Mirrors deepseek4/bailingmoe3. **Upstream PR candidate.**

## `scripts/` — full experiment + tooling suite
Highlights:
- `requant_trunk.c` — **surgical GGUF rewriter**: requants trunk Q8_0→Q4_K while byte-preserving MXFP4 experts + F32 norms (llama-quantize can't do this safely — it requants the experts and destroys QAT calibration). K3's trunk is Q8_0 (59.6GB); Q4_K trunk = 34.2GB GPU-resident → fits 2×3090 with room for a GPU-resident DSpark draft. Build: `gcc -O2 -c requant_trunk.c -I<llama.cpp>/ggml/include -I<llama.cpp>/ggml/src` then link with `g++ ... libggml-base.a libggml-cpu.a libggml.a -lm -lpthread -fopenmp`.
- `30_requant_all.sh` — driver: requant all 32 shards in parallel with write→verify→delete-source (for disk-constrained boxes).
- `measure_quants.py` — walk all shards, report byte/tensor split by quant category (this is how we discovered the trunk was Q8_0 not 4-bit).
- `28_kitchen_sink.sh` — main perf experiment (FA on, cpu-moe, DSpark spec, tensor-split).
- `20/21_*` — baseline perf suites; `25_expert_residency.sh`, `26_draft_sweep.sh` — staged follow-ups.
- `FINDINGS.md` / `HANDOFF.md` — full root-cause log + session handoff (the durable record).

## Key facts
- K3 is **QAT-trained in MXFP4** — the 4-bit experts ARE the reference model. Never re-quant them.
- Usage: `LLAMA_MMAP_NO_PREFETCH=1 llama-server -m <K3 shard1> -ngl 999 --tensor-split ... --cpu-moe -fa on -t 112 -md K3-DSpark-draft-Q8_0.gguf -ngld 999 --spec-type draft-dspark`