scripts/HANDOFF.md

# K3 Rental Validation — Session Handoff

Read `FINDINGS.md` first for all root causes / patches / decisions. This file = live state
+ exact next actions. Started fresh after context compaction.

## Box access
- vast.ai German host: `ssh -o StrictHostKeyChecking=accept-new root@158.181.52.18 -p 43516`
- ed25519 key, passphrase loaded via askpass (do NOT echo it).
- Hardware: 1× EPYC 7663 (56C/112T), 503GB RAM, 8× RTX 5060 Ti 16GB, 2.5TB NVMe (1.1TB free).
- Local repo `/Users/danielleensign/prog/chat/k3-test/` ⇄ box `/root/k3-test/` via rsync.
  rsync clobbers +x — `chmod 755` scripts on box after sync.

## What's DONE and verified
1. K3 Q4_K_XL loads + generates via `LLAMA_MMAP_NO_PREFETCH=1` (lazy mmap patch, FINDINGS #4).
2. K3+DSpark arch crash FIXED: `kimi-k3.cpp` patched to populate `t_layer_inp` (FINDINGS #1).
   Rebuilt llama-server on box. Patch backup: `/root/llama.cpp/src/models/kimi-k3.cpp.bak`.
3. DSpark draft invalid-token FIXED: added `tokenizer.ggml.mask_token_id=163824` to draft
   GGUF (FINDINGS #1b). `draft.gguf` → symlink → `draft_masked.gguf`.
4. Confirmed trunk is Q8_0 (not 4-bit); only experts are MXFP4 (FINDINGS #7).
5. Measured DSpark-on-CPU = 2x LOSS (tg 0.224 vs 0.5 baseline). Need draft on GPU.
6. Built `requant_trunk.c` → `/root/k3-test/requant_trunk` (FINDINGS #7). Requants trunk
   Q8_0→Q4_K, byte-preserves MXFP4 experts + F32 norms. Validated logic on shard 2
   (43 requant + 62 copy, 47.5→44.9 GB, no crash).

## IN FLIGHT (check first)
- Shard-2 requant test: `ssh box 'cat /tmp/requant_test.log; ls -la /tmp/shard2_q4.gguf'`
  (was still writing output, PID was 45320). Verify it completed and output parses.

## NEXT ACTIONS (in order)
1. **Verify shard-2 output correctness.** Confirm `/tmp/shard2_q4.gguf` completed, trunk
   tensors are Q4_K, experts still MXFP4. Quick parse check with a GGUF reader.
2. **Run requant_trunk across all 32 shards.** Output to a new dir
   `/root/models/Kimi-K3-GGUF/Q4TRUNK-Q4_K_XL/` with the SAME filenames (so split loading
   works). Process per-shard: `for i in 01..32: ./requant_trunk in_shard_i out_shard_i`.
   Shard 1 is metadata-only (n_tensors=0) — just byte-copy it as-is (tool handles 0 tensors,
   but verify). NOTE: shard filenames use `-00001-of-00032.gguf` etc. ~47GB/shard, I/O bound,
   consider running 2-4 shards in parallel (NVMe + 112 threads; requant is single-threaded
   per tensor but ggml quantize_row_q4_K_ref may use OpenMP). ~28GB total shrink.
3. **Reload + test** the Q4-trunk model: kitchen sink with the DSpark draft now able to go
   ON GPU (drop `-ngld 0`, or set `-ngld 999`). Expect trunk 34GB → ~94GB free VRAM across
   8 cards. Re-test DSpark spec speedup vs baseline. This is the key validation.
4. If DSpark-on-GPU wins → that's the home architecture. Then expert pinning
   (`--moe-pin`, ~150 hottest) + residency measurement (`25_expert_residency.sh`).

## Key box paths
- llama.cpp: `/root/llama.cpp` (b10630 + lazy-mmap patch + kimi-k3 t_layer_inp patch).
  Build: `cd /root/llama.cpp/build && cmake --build . --config Release -j 56 --target llama-server`
- llama-server: `/root/llama.cpp/build/bin/llama-server`; llama-quantize also built.
- requant tool: `/root/k3-test/requant_trunk` (src `requant_trunk.c`)
- Model: `/root/models/Kimi-K3-GGUF/UD-Q4_K_XL/` (32 shards, 1.4TB)
- Draft: `/root/models/k3-draft/draft.gguf` → `draft_masked.gguf` (2.4GB Q8_0, mask fixed)
- Scripts: `28_kitchen_sink.sh` (main experiment), logs `/root/ks.log`,
  `/root/k3-test/logs/{metrics.tsv,server_ks.log,progress.log}`

## Current best kitchen-sink invocation (in 28_kitchen_sink.sh)
```
LLAMA_MMAP_NO_PREFETCH=1 llama-server --host 127.0.0.1 --port 8899 \
  -m <shard1> -ngl 999 --tensor-split 0.2,1,1,1,1,1,1,1 --cpu-moe -fa on \
  -t 112 -b 256 -ub 256 -np 1 -md <draft> -ngld 0 --spec-type draft-dspark
```
(With Q4 trunk: change split back toward 0.3, drop `-ngld 0` to put draft on GPU.)

## Constraints / prefs (from user)
- Max tok/s, batch-1, 64in/64out probes (box is expensive). Q4-only (no smaller quant).
- Spec + expert-offload combined is the target. ~150 experts pinned + LRU rest.
- Pipe output to files; background jobs that NOTIFY (no busy polling); kill stale jobs.
- Budget target $15-25, hard cap $60/day.