mredmond/Kimi-K2.7-Code-DFlash-kimi-ft-v3

Kimi-K2.7-Code-DFlash — Hex fine-tune (kimi-ft-v3)

A DFlash speculative-decoding draft model for moonshotai/Kimi-K2.7-Code (served as nvidia/Kimi-K2.7-Code-NVFP4), fine-tuned on 3,071 real agent-harness conversations from Hex's coding agent.

The stock NVIDIA draft is trained on generic text; Hex agent traffic is not generic — every conversation carries a large system prompt, notebook and schema context, and long tool-call exchanges. Fine-tuning the draft on that distribution makes it measurably better at predicting the target exactly where it is deployed. Speculative decoding is output-lossless: the draft affects speed only, never output quality.

Full details in tech-report.pdf (methodology, training curves, verified sweeps, and an EAGLE3 comparison).

Headline results

Measured deployed, against the stock draft, under a conversation-paired protocol (28–29 held-out conversations x 3–6 repeats, verified warm caches, single-container servers, stall-robust decode TPS):

metric (concurrency 1)	stock draft	this model	Δ
accepted tokens per 8-token block	3.10	3.65	+17.5%
paired median decode speedup	—	—	+11.7% (95% CI [+5.0, +20.7], n=84 pairs)
high-power replication (4,096-token decodes, n=174)	—	—	+10.2% (95% CI [+5.2, +19.6])

The accept-length advantage holds at every concurrency level (+2–13% at c=2–8); the TPS speedup concentrates at low concurrency, as expected for speculative decoding. Against an off-the-shelf EAGLE3 draft for the same target (AQ-MedAI/Kimi-K2.7-Code-eagle3, comparable to the production Baseten deployment), this model is +106% on paired decode TPS at concurrency 1.

Serving (SGLang v0.5.14)

Serve the NVFP4 target with this repo as the speculative draft:

python -m sglang.launch_server \
  --model-path nvidia/Kimi-K2.7-Code-NVFP4 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path mredmond/Kimi-K2.7-Code-DFlash-kimi-ft-v3 \
  --speculative-dflash-block-size 8 \
  --speculative-draft-attention-backend fa4 \
  --attention-backend trtllm_mla \
  --kv-cache-dtype bf16 \
  --moe-runner-backend flashinfer_trtllm \
  --mem-fraction-static 0.85 \
  --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 \
  --trust-remote-code --tp 8

Notes (hard-won; see the report and specdec/serving/variants.py):

• bf16 KV cache is required: the draft worker inherits the global --kv-cache-dtype and its fa4 kernel asserts bf16 q/k/v, which also rules out the fp8-only tokenspeed_mla target backend — use trtllm_mla.
• Do not pass --quantization explicitly: the draft worker deep-copies the target's server args, and an explicit modelopt_fp4 makes SGLang try to quantize the (bf16, unquantized) draft on the fly. Left unset, the target auto-detects NVFP4 from its own hf_quant_config.json and the draft loads as plain bf16.
• Block size 8 is this checkpoint's native block size — the geometry it was trained (and fine-tuned) at.

Training

warm start	converted nvidia/Kimi-K2.7-Code-DFlash (target embeddings baked in)
data	67,208 request/response pairs → 3,071 training conversations (one per thread, every assistant turn supervised), 29 held out
context lengths	median ~48k tokens, p99 ~126k; max_seq_length 98,304 (fits 96.9% of threads)
recipe	TorchSpec, D-PACE loss, last_turn_loss_only=false, LR 2e-5, 1 epoch = 383 steps at global batch 8
hardware	3x Modal B200:8 (8-GPU FSDP trainer + 4x TP=4 SGLang engines streaming aux hidden states over Mooncake), ~8.4 h
final held-out sim accept length	3.43 (still improving at epoch end)

The draft trains against the target's intermediate hidden states (6 auxiliary layers x 7,168 dims). Fine-tuning helps most at deep block positions — exactly what gates whole-block acceptance.

Files

• model.safetensors — 69 tensors, bf16 (~7.0 GB). Same architecture as the base draft (6 layers, hidden 7168, YaRN rope); embed_tokens / lm_head are borrowed from the target at serving time, as with the stock checkpoint.
• config.json — unchanged from the base draft.
• tech-report.pdf — the kimi-ft-v3 technical report (July 2026).

Caveats

• Fine-tuned for Hex agent-harness traffic; on generic text expect roughly stock-draft behavior, not gains.
• Evaluations above use temperature 1.0 with ignore_eos on held-out conversations collected against the stock-draft server (if anything, a bias against the fine-tune).
• Exported from training checkpoint iter_0000384 (step 383) of the kimi-ft-v3 run; provenance (input hashes, version pins) is recorded in the run manifest in the specdec repo.