Inkling 2 Small — NVFP4

Experimental full NVFP4 quant of thinkingmachines/Inkling-2-Small for use on RTX PRO 6000s.

Modified: MoE expert weights of layers 40-47 requantized from MXFP8 to NVFP4. Those weights are GPTQ-calibrated. The earlier round-to-nearest (RTN) version is kept at revision rtn-baseline; all other tensors are identical between the two.

Calibration

  • 4,096 samples (about 4.2M tokens): evol-codealpaca-v1 and C4, 25% wrapped in the chat template.
  • GPTQ per expert, layers in order (each layer calibrated on the output of the already-quantized layers before it), damping 0.05, block order by activation energy, per-block scale search.
  • Experts that saw under 0.5% of the calibration tokens keep RTN (55-89 of 384 per layer, about 2% of routed tokens).

Measurements

Torch reference of the model, layers 40-47 in place, 67,593 held-out positions, compared with the original MXFP8 layers:

top-1 agreement mean KL (nats) perplexity
original — — 4.438
RTN 0.9917 0.00119 4.433
GPTQ 0.9940 0.00062 4.439

Engine check (vLLM, 6-layer copy with layer 40's experts placed in layer 5): RTN and GPTQ tie (top-1 0.9888 both; same-weights noise floor 0.9944). That check feeds the experts another layer's activations, so it confirms the shards load and run but cannot show a calibration gain; the reference gives the same tie in that setup (0.9840 vs 0.9841).

License: Apache-2.0 (LICENSE). Use is also subject to Thinking Machines Lab's Model Acceptable Use Policy (ACCEPTABLE_USE_POLICY.md).

Downloads last month
80
Safetensors
Model size
337B params
Tensor type
F32
·
BF16
·
F8_E4M3
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for btbtyler09/Inkling-2-Small-NVFP4

Quantized
(21)
this model