lukealonso

/

MiniMax-M3-NVFP4

like 22

The model card notes that a vLLM container for SM120 is forthcoming and that the model may be difficult to run "due to the specialized kernels required." After spending time waiting for upstream instructions I worked out a working configuration for 4x DGX Spark (GB10, sm_121) and wanted to share what worked so the next person does not have to start from scratch.

Requirements

1. A custom container image. The model needs:

   • The chthonic vLLM base (ghcr.io/tonyd2wild/vllm-m3-chthonic:nccl230u1), which bundles the ModelOpt NVFP4 runtime, the B12X MiniMax M3 sparse attention path, and the B12X NVFP4 MoE path
   • ray==2.55.1 installed (multi-node vLLM)
   • b12x from the GitHub commit that ships copy_runtime_metadata (commit 08e980c303b0b6291700a6b85aa09aa874fc27cb is what I used; without that commit signature vLLM aborts during init)
   • NCCL 2.30.7 forced from /opt/nccl230/build/lib/libnccl.so.2. The default NCCL that ships in the chthonic Python environment does not work for multi-node TP. Force it via VLLM_NCCL_SO_PATH=/opt/nccl230/build/lib/libnccl.so.2 and LD_PRELOAD on the same path, and symlink the pip-installed nvidia/nccl/lib/libnccl.so.2 to that path so any transitive loader picks it up. Verify with NCCL_DEBUG=VERSION at startup; you want NCCL version 2.30.7 in the logs.

2. Multi-node launch must be --no-ray (PyTorch distributed), not Ray. With Ray the chthonic build does not initialize the MiniMax M3 parsers cleanly. The launcher is launch-cluster.sh --no-ray --master-port 29501.

3. The vLLM serve flags that worked for me on 4 nodes:

   • --quantization modelopt_fp4
   • --attention-backend B12X_ATTN
   • --moe-backend b12x
   • -cc.mode=VLLM_COMPILE
   • -cc.cudagraph_mode=FULL (slightly better per-request throughput than PIECEWISE, longer warmup)
   • --load-format safetensors
   • --kv-cache-dtype auto (FP8 KV cache works but I left it off for quality)
   • --max-model-len 524288
   • --max-num-batched-tokens 8192
   • --max-num-seqs 5
   • --gpu-memory-utilization 0.85
   • --enable-chunked-prefill --enable-prefix-caching --skip-mm-profiling
   • --mm-encoder-tp-mode data
   • --reasoning-parser minimax_m3
   • --tool-call-parser minimax_m3 --enable-auto-tool-choice
   • --trust-remote-code
   • --tensor-parallel-size 4

   With these, vLLM reports the model as Detected ModelOpt NVFP4 checkpoint, uses B12X MiniMax M3 MSA attention, and the B12X NvFp4 MoE backend.

4. NCCL/RoCE environment. On DGX Spark with a CX7 fabric, the values that mattered were:

   • NCCL_IB_DISABLE=0, NCCL_NET=ib, NCCL_NET_PLUGIN=none
   • NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 (the "twin" RoCE HCAs on the f0 port)
   • NCCL_IB_GID_INDEX=<your RoCEv2 GID index for the fabric subnet>; discover with show_gids rocep1s0f0 1 and pick the row whose TYPE is v2 and whose IP is on your fabric subnet. The exact index depends on your host's GID table and must be re-discovered per host.
   • NCCL_IB_MERGE_NICS=1, NCCL_IB_QPS_PER_CONNECTION=8, NCCL_IB_SPLIT_DATA_ON_QPS=1, NCCL_IB_PCI_RELAXED_ORDERING=1
   • NCCL_NET_GDR_LEVEL=LOC, NCCL_CUMEM_ENABLE=0, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_ASYNC_ERROR_HANDLING=1
   • NCCL_SOCKET_IFNAME/GLOO_SOCKET_IFNAME/UCX_NET_DEVICES set to the fabric Ethernet interface (e.g. enp1s0f0np0)
   • Fabric MTU 4200 end to end (host interface, switch port L2MTU, switch port MTU), with active_mtu 4096 reported by ibv_devinfo -d rocep1s0f0

5. The --no-ray flag in step 2 is what makes the b12x runtime metadata check pass. If you skip it you get a startup error related to B12XPagedAttentionScratchCaps.__init__ missing copy_runtime_metadata.

Startup Checks

You should see all of the following in the head node logs:

Detected ModelOpt NVFP4 checkpoint
Using B12X MiniMax M3 MSA attention
Using 'B12X' NvFp4 MoE backend
FORCED_NCCL_VERSION 23007
vLLM is using nccl==2.30.7
NCCL version 2.30.7+cuda13.2
Application startup complete

If FORCED_NCCL_VERSION is not 23007, the LD_PRELOAD/symlink chain did not take effect and multi-node collectives will hang.

Notes

• Cold startup is long... roughly 10 minutes for shard loading plus several more minutes for profiling, KV cache allocation, B12X MSA warmup, and CUDA graph capture (FULL mode is the slowest to capture). Multimodal warmup adds another 2 to 3 minutes. Total cold start on the head node was about 15 to 18 minutes on my setup.
• Multimodal... send images as data URIs in image_url.url to avoid the model server fetching external URLs. Multimodal warmup completes successfully with --skip-mm-profiling on the chthonic build.
• Performance snapshot from my llama-benchy run https://spark-arena.com/benchmark/sub1781926492932:
  • Generation tokens/sec, mean plus or minus std, concurrency 1: roughly 9 to 10 t/s across depths
  • Concurrency 2: 14 to 18 t/s for depths up to 32768 (higher depths skipped because KV budget exceeded)
  • Concurrency 5: roughly 26 t/s at depth 0
  • Prompt processing (pp2048) ranges from about 5000 t/s at depth 0 down to about 600 t/s at depth 262144