Ling-3.0-flash-NVFP4

NVFP4 (4-bit) quantization of inclusionAI/Ling-3.0-flash (BailingMoeV3, 42-layer hybrid MoE + MTP) produced with NVIDIA ModelOpt using the DeepSeek-V4 experts-only recipe, served natively by the inclusionAI vllm-ling-v3 fork (FlashInfer-CUTLASS NVFP4 MoE).

• Size: 81.3 GB (24 shards) — 3.14x compression over the 255 GB BF16 source
• Quantization: weight-only NVFP4 (E2M1 packed 2-per-byte, E4M3 block scales, FP32 per-tensor scale), activation_scheme: dynamic
• Scope: the 61,440 routed-expert projections (layers 2–41) are NVFP4; attention (KDA + MLA), shared experts, routers, norms, embeddings, lm_head and the full MTP layer remain BF16
• Hardware: designed for NVIDIA GB10 / SM120/SM121-class GPUs

Quickstart

# vllm-ling-v3 fork @ 92c104112 (see the reproducibility repo for the container)
git clone --branch ling_3_0 https://github.com/inclusionAI/vllm-ling-v3
cd vllm-ling-v3 && git checkout 92c104112
VLLM_USE_PRECOMPILED=1 uv venv --python 3.12
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
uv pip install "flashinfer==0.6.15.post1" \
  https://github.com/flashinfer-ai/flashinfer/releases/download/v0.6.15.post1/flashinfer_cubin-0.6.15.post1-py3-none-any.whl

vllm serve r0b0tlab/Ling-3.0-flash-NVFP4 \
  --trust-remote-code --kv-cache-dtype fp8 --mamba-cache-mode align \
  --max-model-len 24576 --enable-auto-tool-choice --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Two project-local fork patches are required on SM121 (Triton MLA decode smem and NVFP4 input-scale defaults) — see github.com/r0b0tlab/ling-3.0-flash-nvfp4 (patches/apply_patches.py).

Note on chat_template.jinja: this checkpoint ships with enable_thinking defaulting to off (harness-compatible; matches the published benchmark rows). Pass chat_template_kwargs: {"enable_thinking": true} per request for reasoning traces, or restore the upstream template from inclusionAI/Ling-3.0-flash for the native thinking-on default.

Quality (r0b0bench core-subset, official harnesses)

lane	score
GSM8K 0-shot (200)	94.5 %
ARC-Easy test (400)	96.0 %
IFEval (lightweight, 200)	83.0 %
HumanEval pass@1 (164)	65.9 %
BFCL v4 multi-turn base (200)	62.0 % (76.5 % with thinking on)
BFCL v4 AST micro (600)	30.7 %
NIAH long-context	PASS (all depths)
Canary	5/5

Performance (1x GB10 / SM121, MTP scale 1, CUDA graphs)

• Decode: 33.5 tok/s C1 (21.4 base AR graphs, 13.8 base AR eager)
• MTP acceptance: 73.2 %, mean acceptance length 1.73
• FP8 KV, 24K context, mamba cache align

Full sweep + methodology: github.com/r0b0tlab/ling-3.0-flash-nvfp4

Reproducibility

• BF16 source: inclusionAI/Ling-3.0-flash @ eb04638493fe9eac066ede636090a9425272c481 (SHA-256 manifest in the reproducibility repo)
• Converter + container: r0b0tlab/ling-3.0-flash-nvfp4 (docker run ... convert reproduces this checkpoint)
• Bench harness: r0b0tlab/r0b0bench (results ledger entries for this model)

License

MIT (mirrors the source model). Weights derived from inclusionAI/Ling-3.0-flash (MIT, © inclusionAI) — see the upstream model card for any additional terms. Quantization and validation by r0b0tlab.