Kimi-K3-GGUF

GGUF quantizations of moonshotai/Kimi-K3, mirrored at AtomicChat/Kimi-K3.

Kimi K3 is a ~2.9T parameter (104B active) mixture of experts model: 93 layers, 896 experts with 16 active per token, 7168 hidden size, 1M context, trained natively in MXFP4. The original only ships in MXFP4, so everything here comes from a direct MXFP4 to Q8_0 conversion, and the lower quants are requantized from that Q8_0.

Honesty first: this thing is enormous

Let's be upfront. This is a monster. The smallest quant we make, IQ1_S at ~1.56 bits, still weighs about 540 GB. That already blows past the 512 GB that even well specced machines top out at. To actually run any of these you need enough fast memory (RAM + VRAM combined) to hold the whole file. In practice that means a large RAM server or a serious multi GPU rig, not a laptop and not most workstations.

We know most people won't be able to run these locally, and that's okay. We made them for the few setups that can, for people who want to poke at the model, and as an archival base so the quants already exist the day the hardware or the tooling catches up. Rule of thumb: grab the biggest quant your memory can actually hold.

Does it even work?

Yes. After computing a quant, we run a full perplexity pass using llama-perplexity, and Q3_K_S came out at PPL 1.6066 +/- 0.02582. That is a healthy number. It means the quantized model is coherent and not broken (a broken quant gives perplexity in the tens or hundreds, or NaN).

Usage

We added text and vision support for kimi_k3 in our fork. It lives in this PR: AtomicBot-ai/atomic-llama-cpp-turboquant#58.

build with -DGGML_CUDA=ON if you have enough GPU for this to work!

git clone https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
cd atomic-llama-cpp-turboquant
git fetch origin pull/58/head:kimi-k3 && git checkout kimi-k3
cmake -B build -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-mtmd-cli

Ask a model anything:

./build/bin/llama-cli -m ./Q3_K_S/Q3_K_S-00001-of-000NN.gguf \
    --jinja --chat-template-file models/templates/Kimi-K3.jinja \
    -p "What is the capital of France?"

Use Vision:

Download a projector, and a picture:

wget -c https://huggingface.co/AtomicChat/Kimi-K3-GGUF/resolve/main/mmproj-F16.gguf
wget -O /tmp/test.jpg https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/mtmd/test-1.jpeg

Then, run a model with projector using --mmproj flag and include image with --image:

./build/bin/llama-mtmd-cli \
    -m ./Q3_K_S/Q3_K_S-00001-of-000NN.gguf \
    --mmproj ./kimi-k3/mmproj-F16.gguf \
    --image your_image.jpg -p "Describe this image in detail." \
    --jinja --chat-template-file models/templates/Kimi-K3.jinja

Plain mainline llama.cpp will reject these files with an unknown architecture error until Kimi K3 support lands upstream (tracking: ggml-org/llama.cpp#26185).

Vision works now!

Kimi K3 is a native vision language model, and as of this update the vision path works end to end in our fork: MoonViT-V2 support in clip/mtmd plus the converted vision tower, published right here as mmproj-F16.gguf and mmproj-BF16.gguf at the repo root. One small mmproj pairs with any text quant above, so nothing has to be re-downloaded. Verified with Q3_K_S: the model reads real images and describes them accurately.

Two important flags to not forget. --jinja is required: the Kimi K3 chat template is a real jinja program and the legacy template matcher cannot handle it. --chat-template-file models/templates/Kimi-K3.jinja (the file ships in the fork checkout) is needed because Moonshot publishes no chat template inside the checkpoint, so current GGUFs carry none. The upcoming v2 quants embed the template, after which only --jinja remains.

In progress: proper 1-bit

We are rebuilding the low-bit ladder: a much deeper importance matrix and dynamic quants that keep attention, gates, latent projections and shared experts at 5 to 6 bit while compressing only the routed experts, which hold 97 percent of the parameters. The target is a ~1.6 bit IQ1_S that stays coherent and fits a single 640 GB 8xH100 node or just a large RAM+CPU server. Existing quants already received an in-place fix of the eos token (163586, the chat turn terminator; the original conversion carried the document terminator, which broke chat turn ends). If you downloaded a quant before this note, re-download its first shard.

Quants

Each quant is a folder of 45 GB shards. Point llama.cpp at the first shard (...-00001-of-000NN.gguf) and the rest load on their own. Sizes for quants that aren't up yet are estimates.

Folder	Bits	Size	imatrix	Status	Notes
IQ1_S	-	~555 GB	yes	bad, improving	smallest, extreme, quality degraded
IQ1_M	-	~610 GB	yes	bad, improving
IQ2_XXS	-	~715 GB	yes	bad, improving	best size to quality in the 2 bit tier
Q2_K	2.9	~1009 GB	no	mediocre, improving
Q3_K_S	3.4	~1196 GB	no	"ok", uploaded
Q8_0	8.5	~2955 GB	no	uploaded	conversion base, for archival and requant
imatrix.dat	n/a	small	n/a	uploaded	importance matrix in gguf, see below

We deliberately skip the middle i-quants (IQ2_XS, IQ2_S, IQ3, IQ4). At roughly 0.8 to 1.2 TB they land right on top of Q2_K and Q3_K_S, which already cover that range and are simpler. The set above spans the useful spectrum without pointless duplicates.

Measured quality (perplexity)

Perplexity on the wikitext-2 test set (ctx 512, 50 chunks). Lower is better. Read these only relative to each other: absolute PPL is only comparable within the same model, dataset and context. We fill this in as each quant finishes.

quant	bits	PPL (wikitext-2, ctx 512)
IQ1_S	1.60	improving
IQ1_M	1.75	improving
IQ2_XXS	2.06	improving
Q2_K	2.9	4.5250 +/- 0.11565
Q3_K_S	3.4	1.6066 +/- 0.02582

About the imatrix

imatrix.dat was computed with the calibration_datav3 corpus and is applied to every i-quant (IQ1 and IQ2).

Being straight about how it was made:

• The source is the Q3_K_S quant, not the full Q8_0. We don't have 3TB vRAM clusters available for this job, so 3 TB Q8 doesn't fit, whereas Q3_K_S (about 1.2 TB) fits in 8xb200 and also runs cleanly on CPU. Computing an imatrix from a quantized model is standard practice, and its effect on the final 1 to 2 bit outputs is second order next to the quantization itself.
• With 896 experts and only 16 firing per token, a handful of rarely used experts stay slightly undersampled and fall back to default quantization for those weights.

Attribution and license

This is a derivative of Kimi K3 by Moonshot AI, redistributed under the Kimi K3 License. The original copyright notice and license are kept in LICENSE.

Kimi K3 License
Copyright (c) 2026 Moonshot AI

Quantized and published by AtomicChat.