mph/Nanbeige4.3-33B-MXFP8

Nanbeige4.3-33B-MXFP8

MXFP8 quantization of Nanbeige/Nanbeige4.3-33B, produced with NVIDIA TensorRT Model Optimizer (ModelOpt) v0.47.0.

The base checkpoint is a text-only LlamaForCausalLM: 52 decoder layers, hidden size 7168, and a native context of 524,288 tokens. The chat template supports reasoning (enable_thinking, on by default) and tool calls (tool_call_format, XML by default).

What is quantized

MXFP8 stores weights and activations as FP8 (E4M3) values with a shared power-of-two (E8M0) scale for every block of 32 elements. This gives roughly half the weight memory of BF16 and keeps accuracy close to it, because the scaling is fine-grained. Scales are dynamic, so the export uses no calibration set.

The following modules are left in BF16, as listed in hf_quant_config.json:

• Language-model embeddings (model.embed_tokens) and lm_head

RMSNorms stay in BF16 as well, and the KV cache stays BF16 (kv_cache_quant_algo is null).

Quantized layers are the decoder linears: attention q_proj, k_proj, v_proj, and o_proj, and MLP gate_proj, up_proj, and down_proj.

Hardware requirements

• Native MXFP8 compute needs NVIDIA Blackwell (SM100/SM120) GPUs.
• Weights take about 34 GB, so one 40 GB+ GPU can hold the weights. Add headroom for the KV cache. Use more GPUs with tensor parallelism for long contexts or high concurrency.

Serving with vLLM

#!/usr/bin/env bash
set -euo pipefail

vllm serve mph/Nanbeige4.3-33B-MXFP8 \
    --port 8000 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-auto-tool-choice \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --gpu-memory-utilization 0.95 \
    --attention-backend FLASHINFER \
    --quantization modelopt_mxfp8 \
    --dtype bfloat16

Serving with SGLang

#!/usr/bin/env bash
set -euo pipefail

python -m sglang.launch_server \
    --model-path mph/Nanbeige4.3-33B-MXFP8 \
    --port 8000 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mem-fraction-static 0.95 \
    --attention-backend flashinfer \
    --dtype bfloat16