BF16 when?
let the world have it
If I understand correctly, there's no BF16 version. It's natively trained in MXFP4, not just quantized down.
If I understand correctly, there's no BF16 version. It's natively trained in MXFP4, not just quantized down.
not possible, bro.
also, in their tech report, it says, "In addition, we apply quantization-aware training (QAT) from the SFT stage onward, with MXFP4 weights and MXFP8 activations (§ 4.1.4)."
There is no FP16 model per se, but some component were Fp16. They probably could release all of those components below, but that would be risky, allowing competition to catch-up quite quickly.
"MXFP4 Quantization-Aware Post-Training To reduce memory footprint and serving cost at deployment, we
quantize the MoE expert weights — which dominate the model’s parameter memory — to MXFP4 [104], with
activations computed in MXFP8, while all non-expert components (attention projections, latent MoE projections, shared
experts, and MoE routers) remain in higher precision. We perform quantization-aware training (QAT) [50] throughout
the entire post-training stage, covering both SFT and RL, so that the model adapts to quantization-induced precision
loss. During RL, rollout and training share the same quantization scheme — eliminating the train–inference mismatch"