mph/BigBang-v1-MXFP8

BigBang-v1-MXFP8

MXFP8 quantization of endless-frontier/BigBang-v1, a Qwen3.6-35B-A3B finetune (Qwen3_5MoeForConditionalGeneration), produced with NVIDIA TensorRT Model Optimizer (ModelOpt) v0.47.0.

The source model has 40 layers in a repeating pattern of 3 Gated DeltaNet layers and 1 full-attention layer, 256 routed experts (top-8) plus a 512-wide shared expert, and a 262,144-token context. Total parameters are about 36B, with about 3B active per token.

What is quantized

MXFP8 stores weights and activations as FP8 (E4M3) values with a shared power-of-two (E8M0) scale for every block of 32 elements. This checkpoint is dynamic MXFP8 (MXFP8_DEFAULT_CFG, no calibration). Routed experts are the bulk of the parameter count, so the weight files are about 37 GiB.

The following modules are left in BF16, as listed in hf_quant_config.json:

• Language-model embeddings and lm_head
• Gated DeltaNet conv1d, in_proj_a, and in_proj_b
• MoE routers (mlp.gate) and shared_expert_gate
• All vision modules (model.visual*)
• The MTP head (mtp*), including its routed experts
• The KV cache

Quantized modules are the language-model linears: full-attention q/k/v/o, Gated DeltaNet in_proj_qkv / in_proj_z / out_proj, shared-expert gate/up/down, and routed-expert gate/up/down. Packed expert tensors are split into per-expert linears on export.

Hardware requirements

• Native MXFP8 compute needs NVIDIA Blackwell (SM100/SM120) GPUs.
• Weights take about 37 GiB, so one 48 GB GPU can hold the weights. Add headroom for the KV cache and vision activations. Use more GPUs with tensor parallelism for long contexts or high concurrency.

Serving with vLLM

#!/usr/bin/env bash
set -euo pipefail

vllm serve mph/BigBang-v1-MXFP8 \
    --port 8000 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-auto-tool-choice \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --gpu-memory-utilization 0.95 \
    --attention-backend FLASHINFER \
    --mm-encoder-tp-mode data \
    --quantization modelopt_mxfp8 \
    --dtype bfloat16 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Serving with SGLang

#!/usr/bin/env bash
set -euo pipefail

python -m sglang.launch_server \
    --model-path mph/BigBang-v1-MXFP8 \
    --port 8000 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mem-fraction-static 0.95 \
    --attention-backend flashinfer \
    --mm-enable-dp-encoder \
    --dtype bfloat16 \
    --speculative-algo NEXTN \
    --speculative-draft-model-path mph/BigBang-v1-MXFP8 \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4