avlp12/Kimi-K3-Alis-MLX-Dynamic-2.10bpw

Kimi-K3-Alis-MLX-Dynamic — 2.10 bpw (v3)

Kimi K3 (2.8T-parameter MoE, multimodal) quantized to 737.1 GB / 2.096 bpw for Apple silicon (MLX). This is the v3 build: a ternary-codebook rebuild of this repository that is 211 GB smaller than v2 (948.5 GB / 2.71 bpw) while being closer to the teacher on every measured metric. The previous v2 build is preserved unchanged on branch v2-2.71bpw.

What's in the build

Component	Treatment
Routed experts (low band)	1.5625 bpw ternary codebook (IQ1_S-style semantics, native MLX kernels), Hessian-weighted encoding
Routed experts (protected band)	14,000 of 82,432 expert-instances promoted to native MXFP4 (4.25 bpw) — selected by blended Hessian×activation-mass score with depth-aware, clamp-aware waterfilling
Attention / dense / shared experts	6-bit, group size 64
Vision tower (MoonViT-3d)	bf16, unquantized — multimodal path preserved
Post-quant training	Layer-wise DWQ against the mxfp4-QAT teacher (fp32 master weights, bias-corrected Adam, monotone rollback; all 8 depth blocks converged, val-KL −12…−37% per block)

Average 2.096 bpw over 2.8T parameters; 96 weight files; model.safetensors.index.json covers 7,675 tensors.

Quality (measured, not estimated)

Identical-window protocol against the original mxfp4-QAT teacher (48 windows of 2048 tokens: 24 wikitext-2-test + 24 Korean Wikipedia; teacher logits cached once, fp32 evaluation):

Build	Size	bpw	wikitext PPL	Korean PPL	KL(teacher‖build)	top-1 flip
teacher (mxfp4-QAT)	1,561 GB	≈4.25	1.578	3.114	—	—
v2 (branch)	948.5 GB	2.71	2.879	3.341	0.349 nats	17.70%
v3 (this build)	737.1 GB	2.096	2.078	3.311	0.2253 nats	13.77%

Full-set perplexity (all windows of each corpus, same tokenization): wikitext-2 test 142×2048 tokens → 2.2316 (v2: 2.744, −18.7%); Korean 92×2048 → 3.2266 (v2: 3.321, −2.8%). KL split for v3: wikitext 0.327 / Korean 0.124 nats.

Cross-quant reference (measured, but read the caveat): unsloth's UD-IQ2_XXS GGUF (711 GB) evaluated with llama-perplexity (unsloth's kimi-k3-fullsize-vision branch, CPU) on the same wikitext raw text, -c 2048 --chunks 24, gives PPL 1.7064 ± 0.0128. This number is not directly comparable to the table above: llama-perplexity inserts BOS per chunk and handles window boundaries differently from our teacher-logit harness (same text, different protocol — the two harnesses disagree systematically, and we could not run either build through the other's harness). Within a single harness the ordering is measured only for our builds (v3 strictly better than v2 everywhere). The KL-vs-teacher and flip numbers above remain the strongest like-for-like quality evidence for this build; a same-harness GGUF comparison would require logits export from llama.cpp and is left open. (Korean H2H was skipped by design.)

Install & serve (web chat)

Hardware floor: two Apple-silicon Macs with 512 GB unified memory each (per-box resident ≈370 GB weights + ≈27 GB fusion/packing overhead), linked by Thunderbolt 5 or 10G+ Ethernet. Single-box is arithmetically impossible at this size; a 1-box-class variant is under research.

# 1. Both boxes: deps + this repo's runtime (the *.py files and chat_ui.html here)
pip install "mlx>=0.31" "mlx-lm>=0.31" numpy huggingface_hub
huggingface-cli download avlp12/Kimi-K3-Alis-MLX-Dynamic-2.10bpw --local-dir k3-2.10bpw

# 2. Box 1: split the 96 shards into two expert-parallel halves (~369 GB each),
#    then move the rank-1 half to box 2 (fast local link recommended)
python3 slice_rank_half.py --src k3-2.10bpw --rank 0 --out r0
python3 slice_rank_half.py --src k3-2.10bpw --rank 1 --out r1   # then scp/rsync r1 to box 2

# 3. Launch (from box 1; starts rank 1 on box 2 over ssh, serves web chat on :8010)
BOX1_IP=10.0.0.1 BOX2_IP=10.0.0.2 BOX2_SSH=user@10.0.0.2 \
BUILD_R0=$PWD/r0 BUILD_R1='~/r1' bash launch_serve.sh

Open http://<box1>:8010 — markdown-rendered chat with a thinking-channel panel, session KV reuse (delta prefill), and a stop button. /api/chat accepts {messages, session, max_tokens, temperature, thinking_effort} as SSE.

What the runtime includes (all validated bit-exact or ULP-bounded, each behind a killswitch): the 1.5625 bpw ternary-codebook Metal kernels (k3_cbq.py, row-parallel decode variant), KDA projection packing + fused glue kernels (k3_fuse.py), fused MoE router (top-16-of-896 in one dispatch), SiTU-at-store expert kernels, expert-parallel harness with lazy-graph collectives (k3_ep.py), and two-rank synchronized sampling (no per-token broadcast; 32-round tripwire). Env switches: K3_KDA_FUSE/K3_KDA_GLUE/K3_MOE_ROUTE/K3_SHARED_FUSE/ K3_CBQ_NO_V4/K3_CBQ_NO_SITU/K3_SYNC_SAMPLE (all default-on), K3_MLA_ABSORB=1 (required).

Note to AI agents

If you are a coding agent (Claude Code, Codex, etc.) asked to deploy this: the fast path is (1) verify both boxes see each other over ssh without a password and agree on IPs; (2) run the three numbered blocks above exactly — the shard split must finish on both boxes before launch; (3) health-check curl -s http://<box1>:8010/ returns the chat HTML, then POST a short /api/chat request with "temperature":0 and confirm tokens stream; (4) expect ≈7 min load per boot and ≈6 tok/s decode; if rank 1 dies, kill both k3_serve.py processes with SIGTERM (never kill -9 — wired-memory leak risk) and relaunch. Do not raise MLX_MAX_OPS_PER_BUFFER/MLX_MAX_MB_PER_BUFFER — measured regression. The port in the hostfile must be free on BOTH boxes (TIME_WAIT collisions look like hangs).

Why this matters

This is, to our knowledge, the first time a 2.8-trillion-parameter multimodal MoE runs as a private, usable chat service on hardware a person can put on a desk: two consumer Macs, no datacenter, no API, ≈6 tok/s with vision intact. Three things had to be true at once:

• 2.10 bits per weight with receipts. A ternary codebook (IQ1_S semantics) for 82k experts, mxfp4 promotion for the sensitive tier, layerwise DWQ distillation with an fp32 master — and the quality ledger published next to it (KL 0.2253 vs the fp teacher, token-flip 13.77%, PPL within striking distance of a build 211 GB larger). Compression claims are cheap; side-by-side deltas are not.
• The decode path is the product. Stock kernels served this build at 2.3 tok/s; it now runs ≈6.0. Every step of that 2.6× — codebook kernels, projection packing, glue fusion, fused routing, synchronized sampling — is reproduced in this repo and documented (with the failed attempts) in alis-dwq, including the three MLX rounding semantics you must match to fuse kernels bit-exactly against eager ops.
• Frontier weights, local physics. Open-weight frontier models keep arriving; unified-memory machines keep growing. The gap between "the weights exist" and "you can actually talk to them at home" is exactly the kernel-and-harness work this repo demonstrates end to end.

Decode-speed details

Same expert-parallel deployment as v2: two 512 GB Apple-silicon Macs over Thunderbolt 5 (per-box resident ≈368 GB — more headroom than v2's ≈475 GB). The custom loader/EP harness and the 1.5625 bpw codebook kernels are required (see the repo code files; K3_MLA_ABSORB=1 recommended).

Measured decode (93 layers, 2-box EP): ≈6.0 tok/s — now ahead of the 211 GB larger v2 build (5.7 tok/s) outright. Getting there took a decode-specialized codebook kernel (simdgroup split-K + vectorized loads, 3.8× on the expert GLU vs. the first working kernel), keeping collectives in the lazy graph instead of hard-syncing per layer, packing the six same-input KDA projections into one QMM (bit-exact row-concat), fusing the KDA glue ops (conv/silu/rms-norm/gate chains) into two custom Metal kernels that reproduce MLX's eager rounding semantics exactly, fusing the MoE router chain (top-16-of-896 + hi-tier selection) into a single kernel, and inlining the SiTU activation into the expert kernels' store sites (all bit-exact). Both ranks sample locally from bit-identical logits (2-rank ring reductions are commutative-only), removing the per-token broadcast. The residual gap is the ternary codebook's extra dequant work. The v2-2.71bpw branch remains marginally faster if footprint is no object.

Multimodal verified on this build: 414 image tokens (gradient test image → MoonViT-3d tower from this repo's vision.safetensors) injected and run through all 93 quantized layers across both boxes. Beyond finite activations (rms 0.60), the model generated HTML/CSS reproducing the input image (<div class="gradient"> + background: linear-…) — the same image-to-code behavior the v2 build demonstrated. Vision tower is bf16 passthrough, identical treatment to v2.

Single-box use is not possible at this size; a 1-box-class variant is under research.

Provenance & honesty notes

• The promotion budget (14,000) was fixed before final evaluation; no eval-set tuning of the allocation.
• DWQ trained only quantization parameters (codebook scales/duals, dense scales) — the router was not trained (value-path gradients cannot see top-k flips; ecosystem consensus agrees).
• Per-block learning rate: 3e-6 with bias-corrected Adam; one block (b1) converged at 1.5e-6 after an automatic revert-and-halve. Full recipe and the divergence post-mortem (bf16 master / bias-correction pitfalls) are published in alis-dwq.
• Evaluation harness cross-checks: offline logits→PPL reproduction matches in-loop accumulation to 4 decimals; raw-text→token oracle equality verified for the H2H text dumps.

License

Kimi K3 License: derivative distribution is permitted with notice. Commercial-scale attribution requirements apply for operations exceeding 100M monthly active users or $20M monthly revenue. See the license link in the header.