| --- |
| base_model: moonshotai/Kimi-K3 |
| base_model_relation: quantized |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - nvfp4 |
| - fp4 |
| - quantized |
| - moe |
| - modelopt |
| - sglang |
| --- |
| |
| # Kimi-K3-NVFP4 |
|
|
| NVFP4 conversion of [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3), |
| produced by a **closed-form, bit-exact MXFP4 → NVFP4 cast** with |
| [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/Model-Optimizer). |
|
|
| Tooling: [`patronus-ai/kimi-k3-nvfp4`](https://github.com/patronus-ai/kimi-k3-nvfp4) |
|
|
| | | | |
| | --- | --- | |
| | Cast fidelity | **85,085,650,944 / 85,085,650,944 blocks bit-exact (100.0000%)**, `max_abs_err 0.0` | |
| | Layers cast | 247,296 (routed MoE experts) | |
| | **GPQA Diamond** | **92.96%** (pass@1 avg-of-16) | |
| | **IFBench** | **75.27%** (pass@1 avg-of-5, prompt_loose) | |
| | **SciCode** | **23.08%** problem / **57.29%** subtask | |
| | Verified on | SGLang `kimi-k3` branch, 2 nodes × 8×B200, TP16 | |
| | Size | ~1.6 TB, 96 shards | |
| |
| ## What this is |
| |
| | | Source (Kimi-K3) | This checkpoint | |
| | --- | --- | --- | |
| | Weight format | MXFP4 (E2M1, block 32, E8M0 scale) | **NVFP4** (E2M1, block 16, E4M3 scale + FP32 per-tensor scale) | |
| | Weight values | — | **bit-identical to the source** | |
| | Scope | routed MoE experts (`w1`/`w2`/`w3`) | routed MoE experts (`w1`/`w2`/`w3`) | |
| |
| Attention, shared experts, dense MLP projections, `lm_head` and vision modules are |
| carried through unquantized. |
|
|
| ## Bit-exactness |
|
|
| The cast is **data-free**: NVFP4 scales are derived from the source E8M0 exponents |
| rather than re-estimated from data. E8M0 has no mantissa, so every source scale is |
| already an exact power of two. With `m = k_max - 8`: |
|
|
| ``` |
| weight_scale_2 = 2^m (per tensor, FP32) |
| per_block_scale = 2^(k_j - m) (per 16 elements, E4M3) |
| => product = 2^k_j exactly |
| ``` |
|
|
| E4M3 represents `2^k` exactly for `k ∈ [-9, 8]`, so any block within 17 binades of |
| the tensor max reconstructs bit-for-bit. On Kimi-K3 that covered **every** block. |
| No calibration data was used, and none is needed. |
|
|
| > **Note on precision.** Because the per-block scale is always pinned to an |
| > exact power of two, this cast deliberately uses none of NVFP4's precision |
| > advantages at the weight level — neither the finer block-16 granularity nor |
| > E4M3's 3 mantissa bits. |
| > That is the cost of bit-exactness, and it is the right trade: quantization is |
| > irreversible, so a higher-resolution container cannot recover information MXFP4 |
| > already discarded. NVFP4's accuracy edge is only reachable by re-quantizing from |
| > BF16, which for 2.8T params means materializing ~5.6 TB. What the conversion buys |
| > is access to the NVFP4 kernel and tooling ecosystem. |
|
|
| ## Deployment (SGLang) |
|
|
| Verified on the SGLang **`kimi-k3` branch** (not `main`), 2 nodes × 8×B200, TP16. |
| Run the identical command on every node, varying only `--node-rank`. |
|
|
| ```bash |
| # --- per-node environment --- |
| export SGLANG_HOST_IP=$(ip -o -4 addr show ens1 | awk '{print $4}' | cut -d/ -f1) |
| export GLOO_SOCKET_IFNAME=ens1 NCCL_SOCKET_IFNAME=ens1 |
| export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_9,mlx5_12,mlx5_13 |
| export NCCL_IB_GID_INDEX=7 NCCL_IB_DISABLE=0 NCCL_NVLS_ENABLE=0 NCCL_CROSS_NIC=0 |
| export PATH=/path/to/venv/bin:$PATH # ninja must resolve from SHARED storage |
| |
| python -m sglang.launch_server \ |
| --model-path /path/to/kimi-k3-nvfp4 \ |
| --trust-remote-code \ |
| --tp-size 16 --nnodes 2 --node-rank ${NODE_RANK} \ |
| --dist-init-addr ${HEAD_IP}:20000 \ |
| --moe-runner-backend marlin \ |
| --mem-fraction-static 0.85 \ |
| --mamba-full-memory-ratio 7.21 \ |
| --disable-flashinfer-autotune \ |
| --watchdog-timeout 3600 --dist-timeout 3600 \ |
| --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \ |
| --model-loader-extra-config '{"enable_multithread_load": true}' \ |
| --host 0.0.0.0 --port 30000 |
| ``` |
|
|
| `SGLANG_HOST_IP` is mandatory for multi-node — without it the ranks never rendezvous |
| and startup hangs with no error. |
|
|
| ### Five non-obvious requirements |
|
|
| Each of these independently prevents the model from serving: |
|
|
| 1. **`--moe-runner-backend marlin`.** Kimi-K3 uses the `situ` (SiTuGlu) activation, |
| which is **not implemented** in SGLang's NVFP4 FlashInfer runner |
| (`_SUPPORTED_FP4_ACTIVATIONS = {silu, relu2, gelu}`). Marlin supports gated |
| `{silu, situ}` and forwards both SiTu parameters |
| (`gemm1_alpha`=β=4.0, `clamp_limit`=linear_β=25.0). |
| 2. **Patch `modelopt_quant.py`**: `_SUPPORTED_ACT_STRS += ("situ",)`. The assert at |
| `:2552` runs *before* the per-backend dispatch at `:2556`, so it rejects `situ` |
| even when the selected backend supports it. Every backend already applies its own |
| stricter check afterwards. |
| 3. **Raise `UNBALANCED_MODEL_LOADING_TIMEOUT_S`** (`load_model_utils.py`, default |
| 480 s, no CLI flag). 1.6 TB over network storage means per-node load skew |
| routinely exceeds 8 minutes and aborts an otherwise healthy run. |
| 4. **`--dist-timeout 3600`.** Marlin's `prepare_moe_nvfp4_layer_for_marlin` repacks |
| 247k expert weights, exceeding NCCL's 600 s default; the first rank to finish then |
| times out alone at `SeqNum=1 _ALLGATHER_BASE`. |
| 5. **Omit `--dcp-size`** unless FlashInfer is new enough — `cutedsl_mla_backend` |
| passes `enable_dcp`/`cp_world` to `trtllm_batch_decode_with_kv_cache_mla`, which |
| flashinfer 0.6.15 does not accept. |
|
|
| Patches 2–3 are scripted in |
| [`serving/patch_situ_nvfp4.py`](https://github.com/patronus-ai/kimi-k3-nvfp4/blob/main/serving/patch_situ_nvfp4.py). |
|
|
| ### Querying |
|
|
| K3 is a reasoning model — it emits thinking tokens before the answer. Use a generous |
| `max_tokens` (≥512) and read `choices[0].message.content`; the thinking is in |
| `reasoning_content`. `reasoning_effort` accepts `low` / `high` / `max`. |
|
|
| ```bash |
| curl $URL/v1/chat/completions -H "Content-Type: application/json" -d '{ |
| "model": "k3", |
| "messages": [{"role":"user","content":"What is 17 * 23? Reply with only the number."}], |
| "max_tokens": 1200, "temperature": 0, "reasoning_effort": "low"}' |
| ``` |
|
|
| With too small a budget you get `content: ""` and `finish_reason: "length"` — that is |
| truncated thinking, not a broken model. |
|
|
| ### This is a W4A16 deployment |
|
|
| Marlin dequantizes the 4-bit weights to BF16 **inside the GEMM**, so weights stay |
| 4-bit in HBM (the memory and bandwidth win is kept) but the math runs at BF16 |
| tensor-core rate. Activations are never quantized. |
|
|
| Consequences: no activation calibration is needed or used and this deployment setup |
| does not deliver FP4 compute throughput. Accuracy should be at least that of native |
| MXFP4, since the weights are bit-identical and the activations are *higher* precision |
| than the source's MXFP8. |
|
|
| ## Evaluation |
|
|
| All three are judge-free and scored with the official harnesses. |
|
|
| | Benchmark | Score | Setup | |
| | --- | --- | --- | |
| | **GPQA Diamond** | **92.96%** | pass@1 avg-of-16, sd 1.24; 198 questions | |
| | **IFBench** | **75.27%** | pass@1 avg-of-5 `prompt_loose`, sd 0.86; 300 prompts | |
| | **SciCode** | **23.08%** / 57.29% | problem / subtask; 65 problems, 288 subtasks | |
|
|
| GPQA per-seed: 96.0 / 93.4 / 91.9 / 91.4 / 92.9 / 93.4 / 90.9 / 93.9 / |
| 92.4 / 91.4 / 93.9 / 92.4 / 92.9 / 93.9 / 93.4 / 92.9. |
|
|
| IFBench: `prompt_loose` counts a response only if every constraint on it is met; |
| `prompt_strict` was 71.07% and instruction-level loose 78.20%. |
|
|
| SciCode: AA test split, `prompt_config=eval/scicode/background`, greedy. |
|
|
| Harness: nemo-skills with prompt `eval/aai/mcq-4choices` and |
| `++eval_type=multichoice`, matching the |
| [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking); |
| 198 Diamond questions, greedy for seed 0 and `temperature=0.6, top_p=0.95` thereafter. |
|
|
| `tokens_to_generate=65536` is load-bearing: at 16384 the model exhausted its budget |
| inside `reasoning` on ~10% of questions and emitted an empty answer, scoring as wrong |
| for a purely harness reason — worth ~8.6 points on the greedy pass (87.4% → 96.0%). |
| Truncation fell from 9.6% to 0.58%. |
|
|
| ## Layout |
|
|
| Per quantized module `<base>`: |
|
|
| ``` |
| <base>.weight uint8 (out, in/2) NVFP4-packed E2M1 nibbles |
| <base>.weight_scale E4M3 (out, in/16) per-block scale |
| <base>.weight_scale_2 fp32 scalar per-tensor scale |
| ``` |
|
|
| `config.json` reports `format: nvfp4-pack-quantized`, `quant_method: modelopt`, |
| `quant_algo: NVFP4`, `group_size: 16`, and an `exclude_modules` list of 55 globs |
| derived from the modules that actually kept an unpacked 2-D float `.weight`. |
|
|
| > **Branches.** `main` is weight-only (`input_activations: null`). `w4a4` adds |
| > `input_scale` placeholders of 1.0; they are unused by the marlin path. |
|
|
| ## Bundled fix |
|
|
| `modeling_kimi_linear.py` here includes a fix to `KimiDeltaAttention`: `A_log` is |
| stored at `head_dim` (128) but the upstream release builds it at `num_heads` (96), so |
| `from_pretrained` fails to load its own weights. |
|
|
| The checkpoint holds **96 trained decay values zero-padded to 128** — verified across |
| all 69 `A_log` tensors here: elements `[96:128]` are exactly zero in every one. The |
| FLA kernel indexes by `num_heads`, so the padding is inert. |
|
|
| The bundled file follows |
| [discussion #150](https://huggingface.co/moonshotai/Kimi-K3/discussions/150): it sizes |
| `A_log` at `num_heads` and narrows the checkpoint tensor in `_load_from_state_dict`, |
| raising if the discarded tail is ever non-zero. |
|
|
| An earlier version of this file followed our own |
| [discussion #144](https://huggingface.co/moonshotai/Kimi-K3/discussions/144), which |
| enlarges the parameter to `head_dim` instead. That approach loads but cannot run: |
| `fla/ops/kda/gate.py` computes `A_log.view(H, 1)` with `H = g.shape[-2]` (the head |
| count), so a 128-element parameter raises |
| `shape '[96, 1]' is invalid for input of size 128` on the first forward. If you pulled |
| this checkpoint before 2026-07-31, re-fetch `modeling_kimi_linear.py`. |
|
|
| This matters only for `transformers`-based loading — SGLang narrows to the first |
| `num_heads` elements independently, which the zero-padding makes lossless. |
|
|
| ## Known limitation: the FP4 fast path |
|
|
| The trtllm-gen cubin pool ships SiTu kernels at NVFP4's 16-element scale block |
| (57 × BF16-activation, 45 × E2M1-activation), but they are unreachable from SGLang's |
| NVFP4 runner, which resolves activations via FlashInfer's `ActivationType` (no SiTu |
| member) and reads a different cubin store. Wiring the dispatch to |
| `sglang.kernels.ops.moe.trtllm_gen_moe` reaches the JIT build stage and then fails on |
| CUDA 12.8 with `namespace "cuda" has no member "maximum"` — `cuda::maximum` requires |
| CUDA 13's CCCL. **Untested beyond that point.** |
|
|
| ## Reproducing |
|
|
| ```bash |
| python cast_mxfp4_to_nvfp4_offline.py \ |
| --source_ckpt /path/to/Kimi-K3 \ |
| --output_ckpt /path/to/Kimi-K3-NVFP4 --verify |
| ``` |
|
|
| Runs on CPU without instantiating the model (~7 h, bounded by shard I/O). |
|
|
| ## License |
|
|
| Derivative of `moonshotai/Kimi-K3`; the upstream model's license and terms apply. |
|
|
