mlx-community/Kimi-K3-mlx-reap160-2bit

Kimi-K3 MLX: 160 experts, 2-bit (181 GB)

An MLX build of moonshotai/Kimi-K3 (2.78 T total, 104 B active) pruned to 160 of 896 routed experts per layer and quantized to 2-bit at group 128, so it runs resident on a 192 GB Apple Silicon machine. In short, about half the memory of the smallest previously published MLX tier.

Read this before downloading

This is a code-capable model with degraded factual recall. It is not a general-purpose K3.

Measured with greedy decoding, 40 new tokens:

prompt	output	verdict
def merge_intervals(intervals):	intervals = sorted(intervals, key=lambda x: x[0]) → merged = [] → for interval in intervals: → if not merged or merged[-1][1] < interval[0]:	correct
The capital of France is	the capital of France is the capital of France is…	fails
机器学习的基本原理是	理解数据，从数据中学习，学习是，学习，学习… (starts coherently, then loops)	degrades

If you need factual recall or Chinese, use a larger tier. If you want a K3 that writes code on a 192 GB Mac, this is the only build we know of that fits.

size	181.2 GB (57 files)
experts	160 of 896 kept per layer (82% pruned), 2-bit affine, group 128
shared experts	both kept, never pruned
non-expert	per-component: attention 6-bit, embeddings/router 8-bit
load time	53 s
decode	3.93 tok/s (measured on a CUDA node; Apple Silicon will differ)
routing	top-16 of 160 = 10.0% density (base 1.8%)

Why the factual recall is gone, precisely

A token's MoE output combines its top-k experts with gate weights renormalized to sum to 1, so the model cannot down-weight an expert it dislikes. With p_j = P(expert j is in a token's top-k), a kept set S serves Σ_{j∈S} p_j / top_k of the router's demand. We measured this on a 32k-token calibration set:

kept	random	AIMER (weight-only)	REAP (this build)	ideal
160	17.9%	22.3%	51.7%	54.3%
376	42.0%	48.2%	77.5%	79.6%

At 160 experts a token reaches ~8.3 of the 16 experts it wanted. That is enough for code and not for facts. Recovering full capability needs ~77% retained mass ≈ 376 experts ≈ 366 GB, which does not fit 192 GB at MLX's 2-bit floor. So this limitation is arithmetic, not a tuning failure.

Things we tried that did not help, so you don't repeat them:

• Lower top_k (16 → 12 → 8 → 6): the France prompt loops identically at every value, and Chinese gets worse. The damage is missing knowledge, not misweighted survivors.
• A weight-only criterion (AIMER): only 1.25× better than random. At 137 experts it retained 19.4% and every prompt looped.
• Shared-basis + low-rank factorization (keep all 896 experts as W_base + A_jB_j): closed by measurement — the shared mean alone has 99.9% relative error and rank 256 still 86%. K3's experts do not share a basis.

Usage

# pip install mlx mlx-lm
# this repo bundles kimi_k3.py; register it with mlx_lm first:
#   cp kimi_k3.py $(python -c "import mlx_lm.models,os;print(os.path.dirname(mlx_lm.models.__file__))")/
from mlx_lm import load, generate
model, tokenizer = load("./Kimi-K3-mlx-reap160-2bit")
print(generate(model, tokenizer, "def merge_intervals(intervals):", max_tokens=128))

Raise the Metal wired limit — it defaults to ~75% of RAM, below this build:

sudo sysctl iogpu.wired_limit_mb=185000

What has and has not been checked

check	status
structural: config, shard index, module coverage	done
loads and generates	done
code generation	done — correct
factual recall	done — fails
Chinese	done — degrades
perplexity / benchmarks	not done
Apple Silicon decode speed	not done (3.93 tok/s measured on a CUDA node)

The prune plan and the REAP saliency ship with the weights, so the selection is auditable and the retained-mass numbers above are reproducible.

Attribution

• Base model: moonshotai/Kimi-K3.
• Loader and streaming converter vendored from PipeNetwork/kimi-k3-mlx.
• REAP expert saliency: arXiv 2510.13999.
• Prior art at 512 GB: pipenetwork/Kimi-K3-REAP80-MLX-mxfp4-q8.