| # K3 Rental Validation — Session Handoff |
|
|
| Read `FINDINGS.md` first for all root causes / patches / decisions. This file = live state |
| + exact next actions. Started fresh after context compaction. |
|
|
| ## Box access |
| - vast.ai German host: `ssh -o StrictHostKeyChecking=accept-new root@158.181.52.18 -p 43516` |
| - ed25519 key, passphrase loaded via askpass (do NOT echo it). |
| - Hardware: 1× EPYC 7663 (56C/112T), 503GB RAM, 8× RTX 5060 Ti 16GB, 2.5TB NVMe (1.1TB free). |
| - Local repo `/Users/danielleensign/prog/chat/k3-test/` ⇄ box `/root/k3-test/` via rsync. |
| rsync clobbers +x — `chmod 755` scripts on box after sync. |
|
|
| ## What's DONE and verified |
| 1. K3 Q4_K_XL loads + generates via `LLAMA_MMAP_NO_PREFETCH=1` (lazy mmap patch, FINDINGS #4). |
| 2. K3+DSpark arch crash FIXED: `kimi-k3.cpp` patched to populate `t_layer_inp` (FINDINGS #1). |
| Rebuilt llama-server on box. Patch backup: `/root/llama.cpp/src/models/kimi-k3.cpp.bak`. |
| 3. DSpark draft invalid-token FIXED: added `tokenizer.ggml.mask_token_id=163824` to draft |
| GGUF (FINDINGS #1b). `draft.gguf` → symlink → `draft_masked.gguf`. |
| 4. Confirmed trunk is Q8_0 (not 4-bit); only experts are MXFP4 (FINDINGS #7). |
| 5. Measured DSpark-on-CPU = 2x LOSS (tg 0.224 vs 0.5 baseline). Need draft on GPU. |
| 6. Built `requant_trunk.c` → `/root/k3-test/requant_trunk` (FINDINGS #7). Requants trunk |
| Q8_0→Q4_K, byte-preserves MXFP4 experts + F32 norms. Validated logic on shard 2 |
| (43 requant + 62 copy, 47.5→44.9 GB, no crash). |
| |
| ## IN FLIGHT (check first) |
| - Shard-2 requant test: `ssh box 'cat /tmp/requant_test.log; ls -la /tmp/shard2_q4.gguf'` |
| (was still writing output, PID was 45320). Verify it completed and output parses. |
| |
| ## NEXT ACTIONS (in order) |
| 1. **Verify shard-2 output correctness.** Confirm `/tmp/shard2_q4.gguf` completed, trunk |
| tensors are Q4_K, experts still MXFP4. Quick parse check with a GGUF reader. |
| 2. **Run requant_trunk across all 32 shards.** Output to a new dir |
| `/root/models/Kimi-K3-GGUF/Q4TRUNK-Q4_K_XL/` with the SAME filenames (so split loading |
| works). Process per-shard: `for i in 01..32: ./requant_trunk in_shard_i out_shard_i`. |
| Shard 1 is metadata-only (n_tensors=0) — just byte-copy it as-is (tool handles 0 tensors, |
| but verify). NOTE: shard filenames use `-00001-of-00032.gguf` etc. ~47GB/shard, I/O bound, |
| consider running 2-4 shards in parallel (NVMe + 112 threads; requant is single-threaded |
| per tensor but ggml quantize_row_q4_K_ref may use OpenMP). ~28GB total shrink. |
| 3. **Reload + test** the Q4-trunk model: kitchen sink with the DSpark draft now able to go |
| ON GPU (drop `-ngld 0`, or set `-ngld 999`). Expect trunk 34GB → ~94GB free VRAM across |
| 8 cards. Re-test DSpark spec speedup vs baseline. This is the key validation. |
| 4. If DSpark-on-GPU wins → that's the home architecture. Then expert pinning |
| (`--moe-pin`, ~150 hottest) + residency measurement (`25_expert_residency.sh`). |
| |
| ## Key box paths |
| - llama.cpp: `/root/llama.cpp` (b10630 + lazy-mmap patch + kimi-k3 t_layer_inp patch). |
| Build: `cd /root/llama.cpp/build && cmake --build . --config Release -j 56 --target llama-server` |
| - llama-server: `/root/llama.cpp/build/bin/llama-server`; llama-quantize also built. |
| - requant tool: `/root/k3-test/requant_trunk` (src `requant_trunk.c`) |
| - Model: `/root/models/Kimi-K3-GGUF/UD-Q4_K_XL/` (32 shards, 1.4TB) |
| - Draft: `/root/models/k3-draft/draft.gguf` → `draft_masked.gguf` (2.4GB Q8_0, mask fixed) |
| - Scripts: `28_kitchen_sink.sh` (main experiment), logs `/root/ks.log`, |
| `/root/k3-test/logs/{metrics.tsv,server_ks.log,progress.log}` |
|
|
| ## Current best kitchen-sink invocation (in 28_kitchen_sink.sh) |
| ``` |
| LLAMA_MMAP_NO_PREFETCH=1 llama-server --host 127.0.0.1 --port 8899 \ |
| -m <shard1> -ngl 999 --tensor-split 0.2,1,1,1,1,1,1,1 --cpu-moe -fa on \ |
| -t 112 -b 256 -ub 256 -np 1 -md <draft> -ngld 0 --spec-type draft-dspark |
| ``` |
| (With Q4 trunk: change split back toward 0.3, drop `-ngld 0` to put draft on GPU.) |
|
|
| ## Constraints / prefs (from user) |
| - Max tok/s, batch-1, 64in/64out probes (box is expensive). Q4-only (no smaller quant). |
| - Spec + expert-offload combined is the target. ~150 experts pinned + LRU rest. |
| - Pipe output to files; background jobs that NOTIFY (no busy polling); kill stale jobs. |
| - Budget target $15-25, hard cap $60/day. |
|
|
