Kimi-K3-REAP-448experts

A 50% expert-pruned, quality-unvalidated research artifact derived from moonshotai/Kimi-K3.

This is not Kimi K3. Half of the routed experts have been deleted and there has been no recovery training, so none of the base model's published benchmark results carry over — assume they do not hold here. If you want a working K3, use the original weights.

Compared with our earlier Kimi-K3-REAP-224experts, this checkpoint prunes half as aggressively and was selected with a calibration corpus that includes Chinese. Both changes are expected to help. Neither has been measured.

⚠️ What we do and do not know

1. No benchmark has been run on this checkpoint. None.

We have not measured SWE-bench, MMLU, perplexity, or anything else on these weights. Everything below is either a property of the surgery (verifiable) or an expectation (not verified). Do not read "50% is safer than 75%" as "50% is safe".

What is known from the literature and from our own smaller-scale runs: we are not aware of a published result where a MoE survives 50% expert pruning without a healing phase and keeps its scores. Expect degradation.

2. There is no knee in the saliency curve at this ratio

Pruning to 448 experts discards 37.5% of the measured saliency mass (at 672 experts it would be 16.6%). The marginal cost per pruned expert rises monotonically through this region — 50% is a hardware-driven choice (it is what fits one 8×H200 node), not a point the measurements recommend.

3. Non-English: improvement is intended but unverified

Our 224-expert checkpoint is severely broken in Chinese — asked for a short story in Chinese it locks into a single-token loop, while its English reasoning in the same forward pass stays coherent. We traced that to the calibration corpus being >99% English: REAP scores an expert by its mean router-weighted activation over the tokens it actually sees, so experts specialising in an unseen language score near zero and are pruned first.

This checkpoint was selected with Chinese data added to the corpus (26% of calibration tokens), which measurably changes which experts survive — see below. Whether that restores usable Chinese generation has not been tested. Treat non-English as unvalidated, not as fixed.

4. Multimodal path is untouched but untested

Only text MoE experts were pruned; the vision tower is intact and we have never run an image through it. Assume nothing.

The calibration-language effect, measured

This is the part of this artifact we think is genuinely worth publishing. Adding a Chinese corpus does not merely nudge the ranking — it changes which experts are kept, far beyond what adding the same amount of data can explain.

We compared three expert rankings computed from the same model and pipeline:

Ranking	Corpus
A	0.70 M tokens, English
B	1.42 M tokens, English (A plus a second English shard)
C	1.92 M tokens, English + Chinese (B plus 0.50 M Chinese tokens)

At 448 kept experts per layer, over all 92 MoE layers (41,216 kept slots):

Change	Slots whose keep/drop flips
A → B: +0.72 M English	1,222	2.96% — the noise floor
B → C: +0.50 M Chinese	3,971	9.63% — 3.2× the floor, from less added data

The Chinese-driven flips are not random churn near the cut point: 70% of the promoted experts are routed at least 1.5× more often (per token) by the Chinese corpus than by the English one, and their English support is well below the corpus median — that is, they were being under-measured rather than genuinely unimportant. The same pattern holds at 25% and 75% pruning.

Note what does not change: retained saliency mass at 448 is 62.6% under ranking B and 62.5% under C. The corpus changes which experts are kept, not how much mass survives. A language-blind corpus does not produce a visibly worse-looking pruning decision — which is precisely why this failure mode is easy to ship by accident.

We are not aware of this effect being documented in the expert-pruning literature, where evaluation is generally monolingual English.

What was and wasn't touched

Each of the 92 MoE layers keeps the 448 highest-saliency routed experts out of 896. Because K3 ships in native MXFP4 (mxfp4-pack-quantized, group_size=32), removing an expert is pure tensor surgery on the packed weights — nothing is dequantized or requantized, and the per-group scales of surviving experts stay bit-for-bit valid, since only whole experts are ever dropped.

Component	Treatment
Routed experts experts.{0..895}.w{1,2,3}	Pruned to 448, renumbered 0..447, packed weights copied verbatim
Router gate.weight, gate.e_score_correction_bias	Rows sliced to kept experts (bias is frozen at inference, so no requantile pass is needed)
Shared experts, latent routed_expert_up/down_proj	Untouched
KDA / MLA attention, AttnRes projections	Untouched
Norms, embeddings, lm_head, vision tower	Untouched

Kept experts stay in ascending original order, so router rows and expert indices remain consistent. K3 uses topk_group=1 (no grouped routing), so no group-balance bookkeeping is affected.

How the saliency was computed

REAP (router-weighted expert activation, mean-pooled over routed tokens). Scores are accumulated in the latent space (ℓ=3584): K3 applies routed_expert_down_proj before the per-expert loop, and the shared up-projection is identical across experts in a layer, so it cannot change their relative ranking.

Coverage	92 MoE layers (layer 0 is dense) × 896 experts, all finite, zero never-routed
Calibration tokens	1.92 M
Support per expert	median 30,012
Corpus	74% English (code-instruction, reasoning, agentic coding traces), 26% Chinese (code instructions, reasoning, technical and general Q&A)
Sequence length	32,768, left-truncated

Because a token reaches only 16 of 896 experts, per-expert support is thinner than we would like; matching the coverage of our 256-expert experiments would need ~3.7 M tokens. Read the ranking as well-resolved at the top and noisier near the cut.

Retained saliency mass by tier, from the same scores: 25% pruned → 83.4%, 50% (this model) → 62.5%, 62.5% → 50.4%, 75% → 37.0%.

Serving

837 GB of weights — this needs 8×H200 (1128 GB) or equivalent; it does not fit a 4×H200 node. Requires vLLM 0.27+ and trust_remote_code. At the time of writing the only working x86_64 path is the official image — the nightly wheels are aarch64-only.

docker run --gpus all -p 8000:8000 vllm/vllm-openai:kimi-k3-x86_64 \
  --model runrunway/Kimi-K3-REAP-448experts \
  --served-model-name k3-reap-448 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --trust-remote-code \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Notes carried over from bringing the 224-expert build up on 4×H200 (this build has not yet been served, so treat these as starting points):

• The host NVIDIA driver must be ≥ 580 (CUDA 13.x) — the image ships torch 2.13+cu130 and fails late, after the weights download, on older drivers.
• --enable-prefix-caching must be passed explicitly; it is off by default for Kimi K3.
• Expert parallelism is preferable to pure tensor parallelism here — it keeps each MXFP4 expert whole on one rank instead of sharding quantized matrices across ranks.
• On Hopper the MXFP4 MoE dispatches to the Marlin W4A16 path, so weights stay 4-bit in memory rather than being upconverted.
• If startup dies in flashinfer_autotune or cudagraph capture with cp_world_size must be positive, pass --no-enable-flashinfer-autotune and --attention-config '{"backend": "FLASHMLA"}'.

Other pruning ratios

Saliency is decoupled from the ratio — the same scores cut any tier without recomputation, and the pruning itself is CPU-only. If you want more headroom, 672 experts (25% pruned) discards only 16.6% of the saliency mass.

License and attribution

Derived from Kimi K3, © 2026 Moonshot AI, under the Kimi K3 License (included as LICENSE), which permits derivative works and their distribution provided the copyright and permission notice are retained. The base model, tokenizer, chat template, modeling code and all of the capability that survives here are Moonshot AI's work; this repository contributes only the expert selection and the surgery that applies it, and the defects described above are ours, not theirs.

Pruning criterion: REAP, from Cerebras' work on router-weighted expert activation pruning.

Provided as is, without warranty of any kind — see Section 5 of the license.