Kimi-K3-REAP-384experts

A 57% expert-pruned, quality-unvalidated research artifact derived from moonshotai/Kimi-K3.

This is not Kimi K3. 512 of the 896 routed experts have been deleted from every MoE layer and there has been no recovery training, so none of the base model's published benchmark results carry over — assume they do not hold here. If you want a working K3, use the original weights.

⚠️ Read this before the numbers

1. No benchmark has been run on this checkpoint. None.

Not SWE-bench, not MMLU, not perplexity. Everything below is either a property of the surgery (verifiable from the files) or an expectation (not verified). We are not aware of a published result where a MoE survives this much expert pruning without a healing phase and keeps its scores. Expect degradation.

2. This tier exists for a hardware reason, not a research one.

Our 50% build, Kimi-K3-REAP-448experts, is 837 GB and needs 8 GPUs. 8×H200 and 4×B300 nodes were unobtainable for us across secure, community and spot pools over repeated attempts, and 6×H200 leaves only ~24 GiB after the 448 weights — not enough for KV cache and activations.

384 is the largest expert count that fits a 6×H200 node with real headroom and divides by 6, so expert parallelism stays available. It is a worse checkpoint than the 448 build in exchange for being one we can actually serve. If you have 8 GPUs, use the 448 build instead.

3. Chinese is expected to be better than our 224 build, and is still unverified.

This uses the same Chinese-inclusive saliency (v2) as the 448 build. Our 224-expert build was selected with an English-only corpus and collapses in Chinese generation in a specific, reproducible way — coherent English reasoning in the same forward pass, then the Chinese output degenerating into a repeated token. v2 was built to fix that cause. Whether it actually does has not been measured on any checkpoint yet.

What the surgery did

Routed experts per MoE layer	896 → 384 (57.1% removed)
MoE layers touched	92 (layer 0 is dense)
Tensors	497,220 → 214,596 (= 497,220 − 92 × 512 × 6, exactly)
Size	~734 GB across 96 shards
Format	Unchanged — native MXFP4 (mxfp4-pack-quantized, group size 32)
Config delta	num_experts: 896 → 384. Nothing else.

Pruning is a byte-level operation: no dequantisation happens anywhere. MXFP4 packed weights and their group-of-32 scales are copied verbatim, so a kept expert is bit-identical to the same expert in the base model.

Component	Treatment
Routed experts experts.{0..895}.w{1,2,3}	Pruned to 384, renumbered 0..383, packed weights copied verbatim
Router gate.weight, gate.e_score_correction_bias	Rows sliced to kept experts (the bias is frozen at inference, so no requantile pass is needed)
Shared experts, latent routed_expert_up/down_proj	Untouched
KDA / MLA attention, AttnRes projections	Untouched
Norms, embeddings, lm_head, vision tower	Untouched

How the experts were chosen

REAP saliency — router-weighted expert activation, averaged over routed tokens — computed on the base model with a layer-streaming driver, then the top 384 per layer kept.

Coverage	92 layers × 896 experts, all finite, zero never-routed
Calibration tokens	1.92 M
Support per expert	median 30,012
Corpus	74% English (code instructions, reasoning, agentic coding traces), 26% Chinese (code instructions, reasoning, technical and general Q&A)
Sequence length	32,768, left-truncated

Saliency mass retained at this tier, and at the neighbours, from the same scores:

Experts kept	Pruned	Total saliency kept	Structured-syntax channel	Shallowest 11 layers
672	25.0%	83.4%	82.2%	85.2%
448	50.0%	62.5%	60.7%	65.1%
384	57.1%	55.7%	54.0%	58.5%
224	75.0%	37.0%	35.6%	39.5%

⚠️ Saliency retention is a proxy, not a validated predictor of quality. It is reported because it is measurable, not because we know how it maps to behaviour. Note in particular that this figure was nearly unchanged (62.6% → 62.5%) by the calibration change that did measurably alter which experts survive — so retention percentages are blind to at least one real failure mode.

Serving

~734 GB of weights. Fits 6×H200 (846 GiB) with roughly 120 GiB left for KV cache and activations, or 8×H200 / 4×B300 with more. Requires vLLM 0.27+ and trust_remote_code. At the time of writing the only working x86_64 path is the official image; nightly wheels are aarch64-only.

docker run --gpus all -p 8000:8000 vllm/vllm-openai:kimi-k3-x86_64 \
  --model runrunway/Kimi-K3-REAP-384experts \
  --served-model-name k3-reap-384 \
  --tensor-parallel-size 6 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

384 divides by 6 and by 8, so --enable-expert-parallel is available at either width if the pure-tensor-parallel path misbehaves on quantised MoE matrices. 96 attention heads and 96 KDA heads divide by 6 and 8 but not by 5 or 7, so those widths will not load at all.

Operational notes, carried over from bringing the 224-expert build up on 4×H200 — this build has not yet been served, so treat them as starting points:

• The host NVIDIA driver must be ≥ 580 (CUDA 13.x). The image ships torch 2.13+cu130 and fails late, after the weights have downloaded, on older drivers.
• --enable-prefix-caching must be passed explicitly; it is off by default for Kimi K3.
• On Hopper the MXFP4 MoE dispatches to the Marlin W4A16 path, so weights stay 4-bit in memory rather than being upconverted. On Blackwell (SM100/SM103) the backend differs; pass --moe-backend marlin if you need the Hopper numerics.
• If startup dies in flashinfer_autotune or cudagraph capture with cp_world_size must be positive, add --no-enable-flashinfer-autotune and --attention-config '{"backend": "FLASHMLA"}'.

Other pruning ratios

Saliency is decoupled from the ratio: the same scores cut any tier without recomputation, and the pruning itself is CPU-only. 672 experts (25% pruned) discards only 16.6% of the saliency mass and is the conservative choice if you have the memory for ~1.2 TB.

License and attribution

Derived from Kimi K3, © 2026 Moonshot AI, under the Kimi K3 License (included as LICENSE), which permits derivative works and their distribution provided the copyright and permission notice are retained. The base model, tokenizer, chat template, modeling code and all of the capability that survives here are Moonshot AI's work; this repository contributes only the expert selection and the surgery that applies it, and the defects described above are ours, not theirs.

Pruning criterion: REAP, from Cerebras' work on router-weighted expert activation pruning.

Provided as is, without warranty of any kind — see Section 5 of the license.