Inkling-2-Small-NVFP4-CSF
Multimodal checkpoint of Inkling 2 Small with NVFP4 routed experts in all 48 decoder layers, calibrated static activation scales, losslessly compressed expert block scales, and MXFP8 DSpark feed-forward weights.
| Component | Stored precision |
|---|---|
| Decoder routed experts | NVFP4; E2M1 values, E4M3 block scales per 16 weights, FP32 global scale and FP32 calibrated input scale per projection; 48 layers, 384 experts, 55,296 matrices |
| Decoder attention projections | BF16 |
| Router weights and biases, short-convolution weights, attention-residual queries | FP32; 368 tensors |
| Embedding, output head, normalization and other decoder tensors | BF16 |
| Vision tower | BF16 |
DSpark feed-forward weights (mtp.safetensors) |
MXFP8; E4M3 values, UE8M0 scales per 32 weights; 15 matrices |
| Other DSpark tensors | BF16; 69 tensors, including attention, normalization, fusion, and the Markov head |
The decoder has 1,418 tensors outside the routed experts: 1,050 BF16 and 368 FP32. The FP32 router and short-convolution weights are exact upcasts of the source BF16 values.
Purpose
The upstream NVFP4 release, thinkingmachines/Inkling-2-Small, suits expert-parallel serving better than tensor-parallel serving, and it mixes two expert formats:
- Its NVFP4 experts require per-token activation quantization: each token's scale comes from the maximum of that token's own activations. With expert parallelism every expert runs whole on one device, so that maximum is local. With tensor parallelism each device holds a slice of every expert's intermediate channels, so the devices must exchange per-token maxima inside every expert layer before the down projection can run.
- Its experts are NVFP4 in decoder layers 0 through 39 and MXFP8 in layers 40 through 47. A runtime needs two expert formats and two sets of kernels, and many local inference stacks support only one of them.
This checkpoint uses the layout most local runtimes already serve: NVFP4 experts in every decoder layer, in the standard Model Optimizer format, with calibrated static activation scales. No scale depends on a token's activations at run time, so tensor-parallel devices run their expert slices independently.
Source and conversion
The pinned weight source is thinkingmachines/Inkling-2-Small-MXFP8 at revision 086a047e8b7ff2ac902dd52355328ddb66b16914. The activation scales are model-inputscales.safetensors from local-inference-lab/Inkling-2-Small at revision 1b77722adf8327b12b57af05cd2f370669b62657, stored unchanged.
Each source expert is decoded from MXFP8 to BF16, with its w13 rows de-interleaved into gate (even rows) and up (odd rows) projections, and packed as NVFP4 by NVIDIA Model Optimizer at commit cc6963dea0a6f01d90d66c7699baf3884df5f64a. Each expert has its own global scale: gate and up share max(|gate|, |up|) / 2688, and down uses max(|down|) / 2688. Block scales are block_max / (6 * global_scale) rounded to E4M3. No scale search, AWQ, or smoothing is applied. Each packed byte holds the even input column in its low nibble.
Each input_scale equals the calibrated input maximum divided by 2688. Gate and up share one value per layer; each down projection has its own value. All other decoder and vision tensors retain their source values. The 15 DSpark feed-forward matrices are converted from BF16 to MXFP8 by Model Optimizer; the other 69 DSpark tensors match the source byte-for-byte.
Tensors use the parameter names of Inkling2ForConditionalGeneration in Transformers 5.23.0, with routed experts stored per expert as model.language_model.layers.L.mlp.experts.E.{gate_proj,up_proj,down_proj}. Three groups keep their source spelling: model.language_model.embed_norm.weight, model.language_model.layers.L.self_attn.{k,v}_sconv.weight_proj on the eight global-attention layers, and model.vision_tower.embeddings.position_embedding.
Storage contract
The checkpoint contains 48 safetensors shards of at most 9.1 GB, the input-scale shard model-inputscales.safetensors, and the DSpark file mtp.safetensors. model.safetensors.index.json maps 443,786 stored tensors, holding 327,558,040,424 bytes, to the shards and the input-scale shard. mtp.safetensors holds 99 tensors and is not indexed.
Scale encoding: weight_scale_encoding: "csf_32x2_v1". The 55,296 compressed matrices are exclusively the routed-expert E4M3 block scales. Each logical weight_scale tensor is represented by five planes: uint8 .csf32_bases, uint8 .csf32_codes, uint32 .csf32_page_bases, uint16 .csf32_offsets, and uint16 .csf32_exceptions. The planes occupy 19,883,694,952 bytes; the scales they encode occupy 36,691,771,392 bytes. Packed weights, global scales, input scales, and DSpark MXFP8 scales remain uncompressed.
Runtime requirements
config.json and hf_quant_config.json use quant_method: "modelopt", quant_algo: "NVFP4", group_size: 16, and weight_scale_encoding: "csf_32x2_v1". The ignore list excludes the output head, the vision tower, and every decoder layer's router, attention, and short-convolution modules. hf_quant_config.json declares the DSpark precision groups under mtp.parameter_groups: a BF16 group of 69 tensors and an mxfp8_nearest group of 15 tensors with block shape [1, 32].
A runtime must reconstruct the E4M3 block scales from the csf32 planes, or execute them directly, before running the routed experts. The static input scales support NVFP4 activation quantization without per-token scale computation.
Validation
Every compressed scale plane passed lossless reconstruction against the uncompressed E4M3 block scales with a decoder independent of the encoder. All other tensors in the numbered shards match the uncompressed conversion byte-for-byte. For that conversion, sampled experts in decoder layers 2, 40, and 47 reproduce Model Optimizer's packed values, block scales, and global scales exactly from an independently decoded source; all non-expert tensors match the cast source tensors; the stored input scales are byte-identical to the published file; and the DSpark MXFP8 matrices equal Model Optimizer's conversion.
License
LICENSE, NOTICE, and LICENSES/ hold the license and attribution materials for this work, including the upstream model's Apache License, Version 2.0.
- Downloads last month
- 78