Inkling 2 Small — NVFP4
Experimental full NVFP4 quant of thinkingmachines/Inkling-2-Small for use on RTX PRO 6000s.
Modified: MoE expert weights of layers 40-47 requantized from MXFP8 to NVFP4. Those weights are GPTQ-calibrated. The earlier round-to-nearest (RTN) version is kept at revision rtn-baseline; all other tensors are identical between the two.
Calibration
- 4,096 samples (about 4.2M tokens): evol-codealpaca-v1 and C4, 25% wrapped in the chat template.
- GPTQ per expert, layers in order (each layer calibrated on the output of the already-quantized layers before it), damping 0.05, block order by activation energy, per-block scale search.
- Experts that saw under 0.5% of the calibration tokens keep RTN (55-89 of 384 per layer, about 2% of routed tokens).
Measurements
Torch reference of the model, layers 40-47 in place, 67,593 held-out positions, compared with the original MXFP8 layers:
| top-1 agreement | mean KL (nats) | perplexity | |
|---|---|---|---|
| original | — | — | 4.438 |
| RTN | 0.9917 | 0.00119 | 4.433 |
| GPTQ | 0.9940 | 0.00062 | 4.439 |
Engine check (vLLM, 6-layer copy with layer 40's experts placed in layer 5): RTN and GPTQ tie (top-1 0.9888 both; same-weights noise floor 0.9944). That check feeds the experts another layer's activations, so it confirms the shards load and run but cannot show a calibration gain; the reference gives the same tie in that setup (0.9840 vs 0.9841).
License: Apache-2.0 (LICENSE). Use is also subject to Thinking Machines Lab's Model Acceptable Use Policy (ACCEPTABLE_USE_POLICY.md).
- Downloads last month
- 80
Model tree for btbtyler09/Inkling-2-Small-NVFP4
Base model
thinkingmachines/Inkling-2-Small