# Hy3 NVFP4 on 2× NVIDIA GB10 (DGX Spark) — vLLM serving recipe

Serve **[LibertAIDAI/Hy3-NVFP4](https://huggingface.co/LibertAIDAI/Hy3-NVFP4)** — a weight-only NVFP4 quant of Tencent's 295B `HYV3` MoE — on a **2-node GB10 / DGX-Spark cluster** (aarch64, compute capability **sm_121a**, ~120 GB unified memory/node) with vLLM tensor-parallel over RoCE.

This recipe is the result of ~10 failed attempts; the notes below exist so you don't repeat them. **The GB10 (sm_121) NVFP4-MoE kernel situation is a minefield** — the exact image + backend + memory settings here are load-bearing.

## TL;DR (what actually works)

| Setting | Value | Why |
|---|---|---|
| Image | **`vllm/vllm-openai:hy3-arm64-cu130`** | CUDA-13 aarch64 vLLM 0.23.1 with `HYV3ForCausalLM`. **CUDA 13 is mandatory** for the FP4 MoE kernels; the `cu129` image fails. |
| MoE backend | **`--moe-backend marlin`** | The only NVFP4-MoE backend that runs on sm_121 today (dequant FP4→FP16). See the dead-ends table. |
| Stability | **`--enforce-eager`** + docker **`mem_limit`** + an OOM guard | Prevents marlin's incremental repack / graph capture from overcommitting the 120 GB unified pool and swap-wedging the box. |
| Fabric | **`GLOO_SOCKET_IFNAME` = the RoCE iface** (=`NCCL_SOCKET_IFNAME`) | vLLM 0.23 multinode Gloo coord binds `127.0.0.1` otherwise → `Gloo connectFullMesh Connection refused`. |
| Parallelism | `--tensor-parallel-size 2 --nnodes 2 --distributed-executor-backend mp` | 169 GB checkpoint doesn't fit one 120 GB node. |

## Hardware / prerequisites

- **2× GB10 / DGX Spark** (or MSI EdgeXpert GB10 etc.), aarch64, sm_121a, ~120 GB unified, NVIDIA driver **≥ 580** (CUDA-13 capable), Docker + `docker compose`.
- A **direct RoCE link** between the nodes (QSFP DAC) with static point-to-point IPs (e.g. `10.10.10.1/.2`) and `NCCL_IB_GID_INDEX` verified (`show_gids` → the RoCEv2 IPv4 GID; often 5, sometimes 3 — **verify per firmware**).
- The **169 GB** NVFP4 checkpoint present on **both** nodes at the same path (each TP rank reads all shards to extract its slice). `rsync` it to both; a fast cipher helps (`-e "ssh -c aes128-gcm@openssh.com"`).

## Deploy

1. **Pull the CUDA-13 image on both nodes:**
   ```bash
   docker pull vllm/vllm-openai:hy3-arm64-cu130
   ```
2. **Put the files** (`docker-compose.hy3.yml`, `.env.hy3`, `start-hy3.sh`, `stop-hy3.sh`, `oom-guard.sh`) in the same dir on the **head** node; edit `.env.hy3` (fabric IPs/iface, `HY3_MODEL_DIR`, model name). The start script SCPs the compose+env to the worker.
3. **Start (worker-first):**
   ```bash
   ./start-hy3.sh          # brings up worker (rank 1, headless) then head (rank 0), waits for /v1/models
   ./oom-guard.sh &        # optional but recommended: stops the container if free RAM enters the thrash zone
   ```
   First start takes ~11 min (weight load + marlin incremental repack + profiling).
4. **Smoke test:**
   ```bash
   curl -s localhost:8888/v1/chat/completions -H 'Content-Type: application/json' -d \
     '{"model":"hy3-preview-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
   # -> "Paris"
   ```

## Memory math (the binding constraint)

At TP=2 each node holds ~**84.5 GB** of weights (169 GB / 2). That leaves ~**35 GB** of the 120 GB unified pool for the KV cache pool, activations, marlin repack scratch, and (if not eager) CUDA-graph capture. Keep it safe:
- `--enforce-eager` (skip graph capture — the biggest transient) — costs throughput but is the difference between "serves" and "swap-wedges the whole box".
- `--kv-cache-dtype fp8` + a bounded `--max-model-len` / `--max-num-seqs` so the KV pool fits `--gpu-memory-utilization` (0.85 works; KV is sharded across TP so 64K×4-seqs fits).
- **`mem_limit: 112g`** on the container so an overcommit gets the *container* OOM-killed, not the whole box (which needs a hard reboot when it swap-wedges — pingable but SSH-dead).

## Dead-ends (do NOT waste time on these)

| Attempt | Result |
|---|---|
| Custom dspark fusion image (cu13.2) | any FP4 kernel JIT → `cudaErrorUnsupportedPtxVersion` (toolkit > driver). Dead end. |
| Stock `v0.23.0-aarch64-cu129`, default (flashinfer-cutlass) MoE | `cudaErrorNoKernelImageForDevice` — flashinfer ships **zero sm_120/121 cubins**; CUTLASS grouped FP4 GEMM is broken on sm120/121. |
| `--moe-backend flashinfer_cutedsl` | rejected: that kernel is gated to `family(100)` (datacenter sm_100), not sm_121. |
| `--moe-backend flashinfer_b12x` on `cu129` | `b12x fused MoE requires CUDA 13 or later. Current: 12.9`. |
| `--moe-backend flashinfer_b12x` on `hy3-arm64-cu130` (cutlass-dsl 4.5.2) | selects + loads, then **livelocks** in the CuTe-DSL JIT (4.5.x breaks sm121 PTX codegen). |
| Downgrade `nvidia-cutlass-dsl==4.4.2` | breaks flashinfer 0.6.13 import (`OperandMajorMode` missing). b12x is version-locked to a custom flashinfer build that isn't published. |
| `--moe-backend marlin` **on cu129/dspark** | kernel works but the marlin repack + graph capture **swap-wedged the box** (needed a reboot ×2). |
| **`--moe-backend marlin` on `hy3-arm64-cu130` + `--enforce-eager` + `mem_limit`** | ✅ **works.** Marlin repacks incrementally per-shard (stable memory), serves coherently. |

The upstream fix for a *native* fast FP4 MoE path on sm_121 is vLLM's **b12x / CuTe-DSL** backend (PR #40082) — but it currently needs a from-source flashinfer + a specific `nvidia-cutlass-dsl` built for sm_121, which no published image provides yet. When that lands, `--moe-backend flashinfer_b12x` on a CUDA-13 image is the intended (faster) replacement for marlin here.

## The checkpoint

`LibertAIDAI/Hy3-NVFP4` — weight-only NVFP4 on the 192 routed-expert FFNs (46,080 tensors, ~98% of the model); attention, shared expert, routers, dense/MTP MLP, embeddings, `lm_head`, norms kept BF16. Per-expert round-trip cos ≈ 0.995. Produced with a memory-frugal shard-streaming ModelOpt pass (no calibration data). `config.json`'s `quantization_config.ignore` must use vLLM's **fused** module names (`*.self_attn.qkv_proj`, `*.mlp.gate_up_proj`, …) or vLLM mis-quantizes BF16 layers and crashes on a shape assert.

_Quantized & documented by [LibertAI](https://libertai.io). Not affiliated with Tencent. Subject to the Tencent Hy Community License._

## Boot persistence (systemd)

`hy3-stack.service` (oneshot, `RemainAfterExit`, waits for the worker to be SSH-able, then worker-first start) + `hy3-watchdog.service` (`After=hy3-stack`; polls `/health`, restarts worker-first on death with an 11-min cooldown). Install on the head node:

```bash
sudo cp hy3-stack.service hy3-watchdog.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now hy3-stack.service hy3-watchdog.service
```

To replace an existing DeepSeek/DSpark stack on the same `:8888`, disable its units first (`sudo systemctl disable --now dspark-stack dspark-watchdog`) so both don't fight for the port. A front proxy (nginx/Caddy) pointed at `:8888` needs no change if the served-model-name advertises the old name too (`--served-model-name hy3-preview-nvfp4 <old-name>`).

## Tuning & benchmarks (2× GB10, TP=2, marlin, fp8 KV)

Production config: `--enforce-eager --max-model-len 262144 --max-num-seqs 32 --gpu-memory-utilization 0.90 --max-num-batched-tokens 16384 --kv-cache-dtype fp8 --enable-prefix-caching` → **KV cache 277,376 tokens** (a full 256K request fits), stable under load with ~2–3 GB host-free.

| Concurrency | 1 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|
| **tok/s aggregate** | 13.3 | 41.7 | 65.2 | 99.8 | **165.8** |

- **Prefix cache:** ~**92% latency cut** on a repeated ~6K-word prefix (3.95 s → 0.32 s).
- **256K long-context** works (~120K-token prefill ≈ 138 s).

**⚠️ CUDA graphs HANG with marlin on sm_121** — dropping `--enforce-eager` either fails the KV check (graph memory + 256K KV don't fit: needs 20 GiB, ~14 GiB available → vLLM refuses) or, at smaller ctx, **livelocks during graph capture** (stuck in the "No shm broadcast block" coordination loop indefinitely). So **`--enforce-eager` is mandatory with the marlin MoE backend here** — single-stream tops out ~13 tok/s; aggregate throughput scales fine with concurrency. A native b12x path (when its flashinfer/cutlass-dsl build lands for sm_121) would lift the single-stream ceiling.