Add 2x GB10 deployment recipe

deploy/.env.hy3.example

@@ -0,0 +1,30 @@
+# Hy3-preview NVFP4 — 2-node vLLM TP=2 serving (clone of dspark fabric, Hy3 args)
+WORKER_HOST=192.168.1.75
+WORKER_DIR=/home/jon/hy3-serve
+MASTER_ADDR=10.10.10.1
+MASTER_PORT=25000
+NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1
+NCCL_SOCKET_IFNAME=enp1s0f1np1
+NCCL_IB_GID_INDEX=5
+NCCL_CROSS_NIC=1
+NCCL_NET=IB
+NCCL_IB_DISABLE=0
+NCCL_CUMEM_ENABLE=0
+NCCL_IGNORE_CPU_AFFINITY=1
+NCCL_DEBUG=WARN
+NCCL_NVLS_ENABLE=0
+
+HF_CACHE=/home/jon/.cache/huggingface
+WORKER_HF_CACHE=/home/jon/.cache/huggingface
+HY3_MODEL_DIR=/home/jon/models/Hy3-preview-NVFP4
+SERVED_MODEL_NAME=hy3-preview-nvfp4
+HY3_VLLM_IMAGE=vllm/vllm-openai:hy3-arm64-cu130
+
+VLLM_HOST=0.0.0.0
+VLLM_HOST_IP=10.10.10.1
+WORKER_VLLM_HOST_IP=10.10.10.2
+
+MAX_MODEL_LEN=65536
+MAX_NUM_SEQS=4
+MAX_NUM_BATCHED_TOKENS=8192
+GPU_MEMORY_UTILIZATION=0.85

deploy/README.md

@@ -0,0 +1,81 @@
+# Hy3 NVFP4 on 2× NVIDIA GB10 (DGX Spark) — vLLM serving recipe
+
+Serve **[LibertAIDAI/Hy3-NVFP4](https://huggingface.co/LibertAIDAI/Hy3-NVFP4)** — a weight-only NVFP4 quant of Tencent's 295B `HYV3` MoE — on a **2-node GB10 / DGX-Spark cluster** (aarch64, compute capability **sm_121a**, ~120 GB unified memory/node) with vLLM tensor-parallel over RoCE.
+
+This recipe is the result of ~10 failed attempts; the notes below exist so you don't repeat them. **The GB10 (sm_121) NVFP4-MoE kernel situation is a minefield** — the exact image + backend + memory settings here are load-bearing.
+
+## TL;DR (what actually works)
+
+| Setting | Value | Why |
+|---|---|---|
+| Image | **`vllm/vllm-openai:hy3-arm64-cu130`** | CUDA-13 aarch64 vLLM 0.23.1 with `HYV3ForCausalLM`. **CUDA 13 is mandatory** for the FP4 MoE kernels; the `cu129` image fails. |
+| MoE backend | **`--moe-backend marlin`** | The only NVFP4-MoE backend that runs on sm_121 today (dequant FP4→FP16). See the dead-ends table. |
+| Stability | **`--enforce-eager`** + docker **`mem_limit`** + an OOM guard | Prevents marlin's incremental repack / graph capture from overcommitting the 120 GB unified pool and swap-wedging the box. |
+| Fabric | **`GLOO_SOCKET_IFNAME` = the RoCE iface** (=`NCCL_SOCKET_IFNAME`) | vLLM 0.23 multinode Gloo coord binds `127.0.0.1` otherwise → `Gloo connectFullMesh Connection refused`. |
+| Parallelism | `--tensor-parallel-size 2 --nnodes 2 --distributed-executor-backend mp` | 169 GB checkpoint doesn't fit one 120 GB node. |
+
+## Hardware / prerequisites
+
+- **2× GB10 / DGX Spark** (or MSI EdgeXpert GB10 etc.), aarch64, sm_121a, ~120 GB unified, NVIDIA driver **≥ 580** (CUDA-13 capable), Docker + `docker compose`.
+- A **direct RoCE link** between the nodes (QSFP DAC) with static point-to-point IPs (e.g. `10.10.10.1/.2`) and `NCCL_IB_GID_INDEX` verified (`show_gids` → the RoCEv2 IPv4 GID; often 5, sometimes 3 — **verify per firmware**).
+- The **169 GB** NVFP4 checkpoint present on **both** nodes at the same path (each TP rank reads all shards to extract its slice). `rsync` it to both; a fast cipher helps (`-e "ssh -c aes128-gcm@openssh.com"`).
+
+## Deploy
+
+1. **Pull the CUDA-13 image on both nodes:**
+   ```bash
+   docker pull vllm/vllm-openai:hy3-arm64-cu130
+   ```
+2. **Put the files** (`docker-compose.hy3.yml`, `.env.hy3`, `start-hy3.sh`, `stop-hy3.sh`, `oom-guard.sh`) in the same dir on the **head** node; edit `.env.hy3` (fabric IPs/iface, `HY3_MODEL_DIR`, model name). The start script SCPs the compose+env to the worker.
+3. **Start (worker-first):**
+   ```bash
+   ./start-hy3.sh          # brings up worker (rank 1, headless) then head (rank 0), waits for /v1/models
+   ./oom-guard.sh &        # optional but recommended: stops the container if free RAM enters the thrash zone
+   ```
+   First start takes ~11 min (weight load + marlin incremental repack + profiling).
+4. **Smoke test:**
+   ```bash
+   curl -s localhost:8888/v1/chat/completions -H 'Content-Type: application/json' -d \
+     '{"model":"hy3-preview-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
+   # -> "Paris"
+   ```
+
+## Memory math (the binding constraint)
+
+At TP=2 each node holds ~**84.5 GB** of weights (169 GB / 2). That leaves ~**35 GB** of the 120 GB unified pool for the KV cache pool, activations, marlin repack scratch, and (if not eager) CUDA-graph capture. Keep it safe:
+- `--enforce-eager` (skip graph capture — the biggest transient) — costs throughput but is the difference between "serves" and "swap-wedges the whole box".
+- `--kv-cache-dtype fp8` + a bounded `--max-model-len` / `--max-num-seqs` so the KV pool fits `--gpu-memory-utilization` (0.85 works; KV is sharded across TP so 64K×4-seqs fits).
+- **`mem_limit: 112g`** on the container so an overcommit gets the *container* OOM-killed, not the whole box (which needs a hard reboot when it swap-wedges — pingable but SSH-dead).
+
+## Dead-ends (do NOT waste time on these)
+
+| Attempt | Result |
+|---|---|
+| Custom dspark fusion image (cu13.2) | any FP4 kernel JIT → `cudaErrorUnsupportedPtxVersion` (toolkit > driver). Dead end. |
+| Stock `v0.23.0-aarch64-cu129`, default (flashinfer-cutlass) MoE | `cudaErrorNoKernelImageForDevice` — flashinfer ships **zero sm_120/121 cubins**; CUTLASS grouped FP4 GEMM is broken on sm120/121. |
+| `--moe-backend flashinfer_cutedsl` | rejected: that kernel is gated to `family(100)` (datacenter sm_100), not sm_121. |
+| `--moe-backend flashinfer_b12x` on `cu129` | `b12x fused MoE requires CUDA 13 or later. Current: 12.9`. |
+| `--moe-backend flashinfer_b12x` on `hy3-arm64-cu130` (cutlass-dsl 4.5.2) | selects + loads, then **livelocks** in the CuTe-DSL JIT (4.5.x breaks sm121 PTX codegen). |
+| Downgrade `nvidia-cutlass-dsl==4.4.2` | breaks flashinfer 0.6.13 import (`OperandMajorMode` missing). b12x is version-locked to a custom flashinfer build that isn't published. |
+| `--moe-backend marlin` **on cu129/dspark** | kernel works but the marlin repack + graph capture **swap-wedged the box** (needed a reboot ×2). |
+| **`--moe-backend marlin` on `hy3-arm64-cu130` + `--enforce-eager` + `mem_limit`** | ✅ **works.** Marlin repacks incrementally per-shard (stable memory), serves coherently. |
+
+The upstream fix for a *native* fast FP4 MoE path on sm_121 is vLLM's **b12x / CuTe-DSL** backend (PR #40082) — but it currently needs a from-source flashinfer + a specific `nvidia-cutlass-dsl` built for sm_121, which no published image provides yet. When that lands, `--moe-backend flashinfer_b12x` on a CUDA-13 image is the intended (faster) replacement for marlin here.
+
+## The checkpoint
+
+`LibertAIDAI/Hy3-NVFP4` — weight-only NVFP4 on the 192 routed-expert FFNs (46,080 tensors, ~98% of the model); attention, shared expert, routers, dense/MTP MLP, embeddings, `lm_head`, norms kept BF16. Per-expert round-trip cos ≈ 0.995. Produced with a memory-frugal shard-streaming ModelOpt pass (no calibration data). `config.json`'s `quantization_config.ignore` must use vLLM's **fused** module names (`*.self_attn.qkv_proj`, `*.mlp.gate_up_proj`, …) or vLLM mis-quantizes BF16 layers and crashes on a shape assert.
+
+_Quantized & documented by [LibertAI](https://libertai.io). Not affiliated with Tencent. Subject to the Tencent Hy Community License._
+
+## Boot persistence (systemd)
+
+`hy3-stack.service` (oneshot, `RemainAfterExit`, waits for the worker to be SSH-able, then worker-first start) + `hy3-watchdog.service` (`After=hy3-stack`; polls `/health`, restarts worker-first on death with an 11-min cooldown). Install on the head node:
+
+```bash
+sudo cp hy3-stack.service hy3-watchdog.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now hy3-stack.service hy3-watchdog.service
+```
+
+To replace an existing DeepSeek/DSpark stack on the same `:8888`, disable its units first (`sudo systemctl disable --now dspark-stack dspark-watchdog`) so both don't fight for the port. A front proxy (nginx/Caddy) pointed at `:8888` needs no change if the served-model-name advertises the old name too (`--served-model-name hy3-preview-nvfp4 <old-name>`).

deploy/docker-compose.hy3.yml

@@ -0,0 +1,72 @@
+services:
+  vllm-hy3:
+    image: ${HY3_VLLM_IMAGE:-vllm/vllm-openai:v0.23.0-aarch64-cu129}
+    network_mode: host
+    ipc: host
+    shm_size: "64gb"
+    # hard cap so a marlin repack overcommit gets the CONTAINER OOM-killed, not the box
+    mem_limit: "112g"
+    ulimits:
+      memlock: -1
+      stack: 67108864
+    gpus: all
+    devices:
+      - /dev/infiniband:/dev/infiniband
+    volumes:
+      - ${HF_CACHE:-${HOME}/.cache/huggingface}:/root/.cache/huggingface
+      - ${HY3_MODEL_DIR:-/home/jon/models/Hy3-preview-NVFP4}:/models/hy3:ro
+    environment:
+      HF_HOME: /root/.cache/huggingface
+      VLLM_CACHE_ROOT: /root/.cache/huggingface/vllm-cache
+      # proven NVFP4-on-GB10 path (ASUS box): stock cu129 image + marlin
+      # Path 1: force the purpose-built sm_121 CuTe-DSL MoE kernel (vLLM PR #40082)
+      # via --moe-backend flashinfer_cutedsl below. Do NOT set
+      # VLLM_USE_FLASHINFER_MOE_FP4=0 (that removes the flashinfer backends incl cutedsl).
+      VLLM_HOST_IP: "${VLLM_HOST_IP:-}"
+      VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
+      NCCL_NET: "${NCCL_NET:-IB}"
+      NCCL_IB_DISABLE: "${NCCL_IB_DISABLE:-0}"
+      NCCL_IB_HCA: "${NCCL_IB_HCA}"
+      NCCL_SOCKET_IFNAME: "${NCCL_SOCKET_IFNAME}"
+      # vLLM 0.23.0's multinode Gloo coord group binds localhost unless told the
+      # fabric iface (older dspark image didn't need this).
+      GLOO_SOCKET_IFNAME: "${NCCL_SOCKET_IFNAME}"
+      NCCL_IB_GID_INDEX: "${NCCL_IB_GID_INDEX:-}"
+      NCCL_CROSS_NIC: "${NCCL_CROSS_NIC:-1}"
+      NCCL_CUMEM_ENABLE: "${NCCL_CUMEM_ENABLE:-0}"
+      NCCL_IGNORE_CPU_AFFINITY: "${NCCL_IGNORE_CPU_AFFINITY:-1}"
+      NCCL_DEBUG: "${NCCL_DEBUG:-WARN}"
+      NCCL_NVLS_ENABLE: "${NCCL_NVLS_ENABLE:-0}"
+      NODE_RANK: "${NODE_RANK}"
+      HEADLESS: "${HEADLESS:-}"
+      MASTER_ADDR: "${MASTER_ADDR}"
+      MASTER_PORT: "${MASTER_PORT:-25000}"
+    entrypoint:
+      - bash
+      - -lc
+    command:
+      - >
+        exec vllm serve /models/hy3
+        --served-model-name ${SERVED_MODEL_NAME:-hy3-preview-nvfp4} deepseek-v4-flash-dspark
+        --host ${VLLM_HOST:-0.0.0.0}
+        --port 8888
+        --trust-remote-code
+        --tensor-parallel-size 2
+        --pipeline-parallel-size 1
+        --moe-backend marlin
+        --kv-cache-dtype fp8
+        --max-model-len ${MAX_MODEL_LEN:-65536}
+        --max-num-seqs ${MAX_NUM_SEQS:-4}
+        --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS:-8192}
+        --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-0.85}
+        --enforce-eager
+        --enable-prefix-caching
+        --tool-call-parser hy_v3
+        --reasoning-parser hy_v3
+        --enable-auto-tool-choice
+        --distributed-executor-backend mp
+        --nnodes 2
+        --node-rank ${NODE_RANK}
+        --master-addr ${MASTER_ADDR}
+        --master-port ${MASTER_PORT:-25000}
+        ${HEADLESS:+--headless}

deploy/hy3-stack.service

@@ -0,0 +1,23 @@
+[Unit]
+Description=Hy3-preview NVFP4 2-node vLLM stack (marlin, worker-first)
+After=network-online.target docker.service
+Wants=network-online.target
+Requires=docker.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+User=jon
+Group=jon
+SupplementaryGroups=docker
+Environment=HOME=/home/jon
+WorkingDirectory=/home/jon/hy3-serve
+# Wait (up to 10 min) for the worker box to be SSH-able before orchestrating
+ExecStartPre=/bin/bash -c 'set -a; . /home/jon/hy3-serve/.env.hy3; set +a; for i in $(seq 1 60); do ssh -o ConnectTimeout=5 -o BatchMode=yes "$WORKER_HOST" true && exit 0; sleep 10; done; echo "worker $WORKER_HOST not reachable" >&2; exit 1'
+ExecStart=/home/jon/hy3-serve/start-hy3.sh
+ExecStop=/home/jon/hy3-serve/stop-hy3.sh
+TimeoutStartSec=1800
+TimeoutStopSec=300
+
+[Install]
+WantedBy=multi-user.target

deploy/hy3-watchdog.service

@@ -0,0 +1,21 @@
+[Unit]
+Description=Hy3 vLLM health watchdog (auto-restart on death)
+# After=hy3-stack: at boot don't poll until the stack's start script returns
+# (weight load + marlin repack + profiling ~11 min), else the failure threshold
+# fires mid-init and stops the starting stack.
+After=network-online.target docker.service hy3-stack.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=jon
+Group=jon
+SupplementaryGroups=docker
+Environment=HOME=/home/jon
+WorkingDirectory=/home/jon/hy3-serve
+ExecStart=/home/jon/hy3-serve/watchdog.sh
+Restart=always
+RestartSec=15
+
+[Install]
+WantedBy=multi-user.target

deploy/oom-guard.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# Runs locally on the head. Stops Hy3 gracefully if free RAM enters the thrash
+# zone, so vLLM overcommit can't wedge the box (SSH-dead) like last time.
+# Exits 0 when API is up (guard no longer needed), 2 if it had to intervene.
+LOG=/tmp/hy3-guard.log
+THRESH_MB=6000
+echo "$(date -u) guard start (thresh ${THRESH_MB}MB)" > "$LOG"
+for i in $(seq 1 240); do
+  cid=$(sg docker -c "docker ps -q --filter name=hy3" 2>/dev/null | head -1)
+  if [ -z "$cid" ]; then echo "$(date -u) container gone, exiting" >> "$LOG"; exit 0; fi
+  if curl -fsS --max-time 3 http://127.0.0.1:8888/v1/models >/dev/null 2>&1; then
+    echo "$(date -u) API up, guard done" >> "$LOG"; exit 0
+  fi
+  avail=$(free -m | awk 'NR==2{print $7}')
+  echo "$(date -u) avail=${avail}MB" >> "$LOG"
+  if [ "${avail:-0}" -lt "$THRESH_MB" ]; then
+    echo "$(date -u) !!! OOM GUARD TRIP avail=${avail}MB — stopping hy3 to protect the box" >> "$LOG"
+    ( cd /home/jon/hy3-serve && bash stop-hy3.sh ) >> "$LOG" 2>&1
+    exit 2
+  fi
+  sleep 5
+done
+echo "$(date -u) guard timeout" >> "$LOG"; exit 3

deploy/start-hy3.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+ENV_FILE="$SCRIPT_DIR/.env.hy3"
+COMPOSE_FILE="$SCRIPT_DIR/docker-compose.hy3.yml"
+API_URL="http://127.0.0.1:8888/v1/models"
+WAIT_ATTEMPTS="${WAIT_ATTEMPTS:-120}"
+WAIT_SECONDS="${WAIT_SECONDS:-15}"
+
+set -a; source "$ENV_FILE"; set +a
+cd "$SCRIPT_DIR"
+WORKER_DIR="${WORKER_DIR:-$SCRIPT_DIR}"
+REMOTE="cd $WORKER_DIR && env -u MASTER_ADDR -u MASTER_PORT -u NODE_RANK -u HEADLESS COMPOSE_DISABLE_ENV_FILE=1"
+
+echo "Syncing Hy3 files to ${WORKER_HOST}:${WORKER_DIR}"
+ssh "$WORKER_HOST" "mkdir -p $WORKER_DIR"
+scp -q "$COMPOSE_FILE" "${WORKER_HOST}:${WORKER_DIR}/docker-compose.hy3.yml"
+scp -q "$ENV_FILE" "${WORKER_HOST}:${WORKER_DIR}/.env.hy3"
+
+echo "Starting Hy3 worker (rank 1) on ${WORKER_HOST}..."
+ssh "$WORKER_HOST" "$REMOTE NODE_RANK=1 HEADLESS=1 VLLM_HOST_IP='$WORKER_VLLM_HOST_IP' docker compose --env-file .env.hy3 -f docker-compose.hy3.yml up -d"
+
+echo "Starting Hy3 head (rank 0)..."
+COMPOSE_DISABLE_ENV_FILE=1 NODE_RANK=0 HEADLESS= docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" up -d
+
+echo "Waiting for Hy3 vLLM API (up to $((WAIT_ATTEMPTS*WAIT_SECONDS))s)..."
+for _ in $(seq 1 "$WAIT_ATTEMPTS"); do
+  if curl -fsS --max-time 5 "$API_URL" >/dev/null 2>&1; then
+    echo "Hy3 vLLM is UP: $API_URL"; exit 0
+  fi
+  sleep "$WAIT_SECONDS"
+done
+echo "TIMEOUT. head logs:" >&2
+COMPOSE_DISABLE_ENV_FILE=1 docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" logs --tail=80 vllm-hy3 2>&1 | tail -80
+exit 1

deploy/stop-hy3.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+ENV_FILE="$SCRIPT_DIR/.env.hy3"
+COMPOSE_FILE="$SCRIPT_DIR/docker-compose.hy3.yml"
+set -a; source "$ENV_FILE"; set +a
+cd "$SCRIPT_DIR"
+WORKER_DIR="${WORKER_DIR:-$SCRIPT_DIR}"
+echo "Stopping Hy3 head..."
+COMPOSE_DISABLE_ENV_FILE=1 docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" down || true
+echo "Stopping Hy3 worker on ${WORKER_HOST}..."
+ssh "$WORKER_HOST" "cd '$WORKER_DIR' && COMPOSE_DISABLE_ENV_FILE=1 docker compose --env-file .env.hy3 -f docker-compose.hy3.yml down" || true
+echo "Hy3 stopped."

deploy/watchdog.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# hy3-watchdog: poll vLLM /health and self-heal the 2-node Hy3 stack
+# (worker-first) if the server dies. Logs to /home/jon/hy3-serve/watchdog.log.
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:8888/health}"
+POLL_INTERVAL="${POLL_INTERVAL:-30}"
+FAIL_THRESHOLD="${FAIL_THRESHOLD:-3}"
+COOLDOWN="${COOLDOWN:-900}"   # min secs between restarts (init ~11 min)
+LOG="${LOG:-$SCRIPT_DIR/watchdog.log}"
+fails=0; last_restart=0
+log(){ echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') $*" | tee -a "$LOG" >&2; }
+log "watchdog start (health=$HEALTH_URL threshold=$FAIL_THRESHOLD)"
+while true; do
+  if curl -fsS --max-time 10 "$HEALTH_URL" >/dev/null 2>&1; then
+    [ "$fails" -ne 0 ] && log "health recovered after $fails fail(s)"
+    fails=0
+  else
+    fails=$((fails+1))
+    log "/health DOWN (consecutive=$fails/$FAIL_THRESHOLD)"
+    if [ "$fails" -ge "$FAIL_THRESHOLD" ]; then
+      now=$(date +%s)
+      if [ $((now - last_restart)) -lt "$COOLDOWN" ]; then
+        log "in cooldown ($((now-last_restart))s < ${COOLDOWN}s); skip"
+      else
+        log "RESTARTING Hy3 stack (worker-first)"
+        bash "$SCRIPT_DIR/stop-hy3.sh"  >>"$LOG" 2>&1 || true
+        bash "$SCRIPT_DIR/start-hy3.sh" >>"$LOG" 2>&1 || true
+        ( nohup bash "$SCRIPT_DIR/oom-guard.sh" >/dev/null 2>&1 & ) || true
+        last_restart=$(date +%s); fails=0
+      fi
+    fi
+  fi
+  sleep "$POLL_INTERVAL"
+done