Hy3 NVFP4 on 2× NVIDIA GB10 (DGX Spark) — vLLM serving recipe

Serve LibertAIDAI/Hy3-NVFP4 — a weight-only NVFP4 quant of Tencent's 295B HYV3 MoE — on a 2-node GB10 / DGX-Spark cluster (aarch64, compute capability sm_121a, ~120 GB unified memory/node) with vLLM tensor-parallel over RoCE.

This recipe is the result of ~10 failed attempts; the notes below exist so you don't repeat them. The GB10 (sm_121) NVFP4-MoE kernel situation is a minefield — the exact image + backend + memory settings here are load-bearing.

TL;DR (what actually works)

Setting	Value	Why
Image	`vllm/vllm-openai:hy3-arm64-cu130`	CUDA-13 aarch64 vLLM 0.23.1 with `HYV3ForCausalLM`. CUDA 13 is mandatory for the FP4 MoE kernels; the `cu129` image fails.
MoE backend	`--moe-backend marlin`	The only NVFP4-MoE backend that runs on sm_121 today (dequant FP4→FP16). See the dead-ends table.
Stability	`--enforce-eager` + docker `mem_limit` + an OOM guard	Prevents marlin's incremental repack / graph capture from overcommitting the 120 GB unified pool and swap-wedging the box.
Fabric	`GLOO_SOCKET_IFNAME` = the RoCE iface (=`NCCL_SOCKET_IFNAME`)	vLLM 0.23 multinode Gloo coord binds `127.0.0.1` otherwise → `Gloo connectFullMesh Connection refused`.
Parallelism	`--tensor-parallel-size 2 --nnodes 2 --distributed-executor-backend mp`	169 GB checkpoint doesn't fit one 120 GB node.

Hardware / prerequisites

2× GB10 / DGX Spark (or MSI EdgeXpert GB10 etc.), aarch64, sm_121a, ~120 GB unified, NVIDIA driver ≥ 580 (CUDA-13 capable), Docker + docker compose.
A direct RoCE link between the nodes (QSFP DAC) with static point-to-point IPs (e.g. 10.10.10.1/.2) and NCCL_IB_GID_INDEX verified (show_gids → the RoCEv2 IPv4 GID; often 5, sometimes 3 — verify per firmware).
The 169 GB NVFP4 checkpoint present on both nodes at the same path (each TP rank reads all shards to extract its slice). rsync it to both; a fast cipher helps (-e "ssh -c aes128-gcm@openssh.com").

Deploy

Pull the CUDA-13 image on both nodes:

docker pull vllm/vllm-openai:hy3-arm64-cu130

Put the files (docker-compose.hy3.yml, .env.hy3, start-hy3.sh, stop-hy3.sh, oom-guard.sh) in the same dir on the head node; edit .env.hy3 (fabric IPs/iface, HY3_MODEL_DIR, model name). The start script SCPs the compose+env to the worker.

Start (worker-first):

./start-hy3.sh          # brings up worker (rank 1, headless) then head (rank 0), waits for /v1/models
./oom-guard.sh &        # optional but recommended: stops the container if free RAM enters the thrash zone

First start takes ~11 min (weight load + marlin incremental repack + profiling).

Smoke test:

curl -s localhost:8888/v1/chat/completions -H 'Content-Type: application/json' -d \
  '{"model":"hy3-preview-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
# -> "Paris"

Memory math (the binding constraint)

At TP=2 each node holds ~84.5 GB of weights (169 GB / 2). That leaves ~35 GB of the 120 GB unified pool for the KV cache pool, activations, marlin repack scratch, and (if not eager) CUDA-graph capture. Keep it safe:

--enforce-eager (skip graph capture — the biggest transient) — costs throughput but is the difference between "serves" and "swap-wedges the whole box".
--kv-cache-dtype fp8 + a bounded --max-model-len / --max-num-seqs so the KV pool fits --gpu-memory-utilization (0.85 works; KV is sharded across TP so 64K×4-seqs fits).
mem_limit: 112g on the container so an overcommit gets the container OOM-killed, not the whole box (which needs a hard reboot when it swap-wedges — pingable but SSH-dead).

Dead-ends (do NOT waste time on these)

Attempt	Result
Custom dspark fusion image (cu13.2)	any FP4 kernel JIT → `cudaErrorUnsupportedPtxVersion` (toolkit > driver). Dead end.
Stock `v0.23.0-aarch64-cu129`, default (flashinfer-cutlass) MoE	`cudaErrorNoKernelImageForDevice` — flashinfer ships zero sm_120/121 cubins; CUTLASS grouped FP4 GEMM is broken on sm120/121.
`--moe-backend flashinfer_cutedsl`	rejected: that kernel is gated to `family(100)` (datacenter sm_100), not sm_121.
`--moe-backend flashinfer_b12x` on `cu129`	`b12x fused MoE requires CUDA 13 or later. Current: 12.9`.
`--moe-backend flashinfer_b12x` on `hy3-arm64-cu130` (cutlass-dsl 4.5.2)	selects + loads, then livelocks in the CuTe-DSL JIT (4.5.x breaks sm121 PTX codegen).
Downgrade `nvidia-cutlass-dsl==4.4.2`	breaks flashinfer 0.6.13 import (`OperandMajorMode` missing). b12x is version-locked to a custom flashinfer build that isn't published.
`--moe-backend marlin` on cu129/dspark	kernel works but the marlin repack + graph capture swap-wedged the box (needed a reboot ×2).
`--moe-backend marlin` on `hy3-arm64-cu130` + `--enforce-eager` + `mem_limit`	✅ works. Marlin repacks incrementally per-shard (stable memory), serves coherently.

The upstream fix for a native fast FP4 MoE path on sm_121 is vLLM's b12x / CuTe-DSL backend (PR #40082) — but it currently needs a from-source flashinfer + a specific nvidia-cutlass-dsl built for sm_121, which no published image provides yet. When that lands, --moe-backend flashinfer_b12x on a CUDA-13 image is the intended (faster) replacement for marlin here.

The checkpoint

LibertAIDAI/Hy3-NVFP4 — weight-only NVFP4 on the 192 routed-expert FFNs (46,080 tensors, ~98% of the model); attention, shared expert, routers, dense/MTP MLP, embeddings, lm_head, norms kept BF16. Per-expert round-trip cos ≈ 0.995. Produced with a memory-frugal shard-streaming ModelOpt pass (no calibration data). config.json's quantization_config.ignore must use vLLM's fused module names (*.self_attn.qkv_proj, *.mlp.gate_up_proj, …) or vLLM mis-quantizes BF16 layers and crashes on a shape assert.

Quantized & documented by LibertAI. Not affiliated with Tencent. Subject to the Tencent Hy Community License.

Boot persistence (systemd)

hy3-stack.service (oneshot, RemainAfterExit, waits for the worker to be SSH-able, then worker-first start) + hy3-watchdog.service (After=hy3-stack; polls /health, restarts worker-first on death with an 11-min cooldown). Install on the head node:

sudo cp hy3-stack.service hy3-watchdog.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now hy3-stack.service hy3-watchdog.service

To replace an existing DeepSeek/DSpark stack on the same :8888, disable its units first (sudo systemctl disable --now dspark-stack dspark-watchdog) so both don't fight for the port. A front proxy (nginx/Caddy) pointed at :8888 needs no change if the served-model-name advertises the old name too (--served-model-name hy3-preview-nvfp4 <old-name>).

Tuning & benchmarks (2× GB10, TP=2, marlin, fp8 KV)

Production config: --enforce-eager --max-model-len 262144 --max-num-seqs 32 --gpu-memory-utilization 0.90 --max-num-batched-tokens 16384 --kv-cache-dtype fp8 --enable-prefix-caching → KV cache 277,376 tokens (a full 256K request fits), stable under load with ~2–3 GB host-free.

Concurrency	1	4	8	16	32
tok/s aggregate	13.3	41.7	65.2	99.8	165.8

Prefix cache: ~92% latency cut on a repeated ~6K-word prefix (3.95 s → 0.32 s).
256K long-context works (~120K-token prefill ≈ 138 s).

⚠️ CUDA graphs HANG with marlin on sm_121 — dropping --enforce-eager either fails the KV check (graph memory + 256K KV don't fit: needs 20 GiB, ~14 GiB available → vLLM refuses) or, at smaller ctx, livelocks during graph capture (stuck in the "No shm broadcast block" coordination loop indefinitely). So --enforce-eager is mandatory with the marlin MoE backend here — single-stream tops out ~13 tok/s; aggregate throughput scales fine with concurrency. A native b12x path (when its flashinfer/cutlass-dsl build lands for sm_121) would lift the single-stream ceiling.