Model-SafeTensors/MiniMax-M3-MoE-FP8-128x128

MiniMax-M3 — MoE-only block-FP8 (128×128)

Block-FP8 quantization of MiniMaxAI/MiniMax-M3 applied only to the MoE expert weights. Everything else stays BF16.

• 408 GB vs 796 GiB for the BF16 original (~102 GB/GPU at TP4, vs 199 GB)
• Quantized directly from BF16 in a single rounding step
• Vision tower intact — still the full multimodal MiniMaxM3SparseForConditionalGeneration

What is and isn't quantized

Quantized (block-FP8 128×128)	Left in BF16 / FP32
routed experts w1 / w2 / w3	all attention, incl. index_q_proj / index_k_proj (sparse indexer)
shared experts gate/up/down_proj	router gate (FP32 — precision-critical for expert selection)
	dense MLP layers 0–2
	vision tower, lm_head, embed_tokens

Routed experts are ~96.6% of the parameters, so MoE-only quantization captures essentially all of the memory saving while leaving the accuracy-sensitive and long-context-critical paths untouched. Shared experts are included because they run for every token (unlike top-4-of-128 routed experts) and account for ~1/5 of MoE decode traffic.

22,059 tensors were quantized (21,888 routed = 57 layers × 128 experts × 3, plus 171 shared = 57 × 3).

Format

weight_scale_inv follows the DeepSeek-V3 block-FP8 contract — despite the name it is a multiplier:

dequantized = fp8_weight * weight_scale_inv

Weights are float8_e4m3fn; scales are FP32 with shape [ceil(N/128), ceil(K/128)]. Activations are quantized dynamically in 1×128 groups (per token, along K) — hence activation_scheme: dynamic, since block-wise FP8 cannot use static activation scales.

Mean round-trip error vs the original BF16 is ~2.4%.

Usage (SGLang)

python -m sglang.launch_server \
  --model-path Model-SafeTensors/MiniMax-M3-MoE-FP8-128x128 \
  --trust-remote-code \
  --tp-size 4 \
  --moe-runner-backend triton \
  --context-length 98304

Requires an SGLang build with MiniMax-M3 support (merged 2026-07-11).

⚠️ Known serving issues on AMD / ROCm

Both are properties of the serving stack, not of these weights. Both produce fluent-looking garbage with no error, so smoke-test before trusting output.

1. Do not use --moe-runner-backend aiter with MiniMax-M3. M3's MoE uses a clamped SwiGLU (gemm1_alpha=1.702, gemm1_clamp_limit=7.0):

   gate = gate.clamp(max=limit); up = up.clamp(-limit, limit)
   out  = gate * sigmoid(gate * alpha) * (up + 1)
   

   aiter's fused_moe has no alpha argument and its ActivationType offers only {No, Gelu, Silu, Swiglu}, so the runner silently computes a plain SiLU. This affects BF16 too, not just FP8.

2. SGLANG_USE_AITER=1 together with --moe-runner-backend triton can corrupt output. On builds where the block-FP8 post-load path gates its aiter (16,16) weight pre-shuffle on the SGLANG_USE_AITER env var rather than on the selected MoE runner, weights are shuffled for aiter and then consumed un-shuffled by triton. Workaround: set SGLANG_USE_AITER=0, or use a build whose plain-FP8 branch gates the shuffle on the runner (the mxfp8 → block-fp8 branch already does).

Why MoE-only, and why 128×128

• 128×128 blocks are the granularity that reaches the optimized block-FP8 kernels (DeepSeek-V3 style), rather than per-tensor or per-channel scaling.
• Single rounding. An MXFP8 checkpoint served on gfx942 (CDNA3, which has no MX matmul hardware) is converted to block-FP8 at load — a dequant→requant round trip that drops from 1×32 scales to 128×128, a 512× reduction in scale count, on top of already-quantized values. Quantizing straight from BF16 avoids that second rounding.
• BF16 attention keeps the sparse-attention indexer exact, which matters most at long context where attention dominates the decode step.

Provenance

Quantized from the official BF16 release using the DeepSeek-V3 block-FP8 formula (amax/448 per 128×128 tile, clamp(min=1e-4)), verified bit-identical to SGLang's own bf16_to_block_fp8_128 reference implementation.

Evaluation

Not formally evaluated. Validated by generation smoke tests and the ~2.4% weight round-trip error only — no GSM8K / MMLU / perplexity numbers were produced. Benchmark before production use.

License

Inherits the MiniMax Model License from the base model. See LICENSE.