Hy3-NVFP4-FP8-TP3

A tensor-parallel-3-ready variant of RedHatAI/Hy3-NVFP4-FP8 (quantized tencent/Hy3 — MoE NVFP4, attention FP8 block) plus the tooling to serve it on 3× NVIDIA DGX Spark (GB10 / sm_121) with vLLM + sparkrun.

The original is GQA with num_attention_heads=64, num_key_value_heads=8, intermediate_size=13312 — all divisible by 2 (so TP=2 works natively) but not by 3, which blocks TP=3 on a 3-Spark fleet. This repo ships a checkpoint with those dimensions padded, plus a tiny vLLM patch for the one remaining dimension (vocab).

What changed (mathematically lossless)

Zero-padded inert head/neuron slices are appended; a zero query head / zero MLP neuron contributes exactly zero to the output — the model is bit-equivalent to the original (verified numerically, max |Δ| = 0.0).

Dimension	Original	Padded	Where
num_attention_heads	64	72	checkpoint (+8 zero query heads; 8:1 GQA ratio kept)
num_key_value_heads	8	9	checkpoint (+1 zero KV head)
intermediate_size (dense layer 0)	13312	13344	checkpoint (+32 zero neurons, NVFP4-group aligned)
vocab_size	120832	120832 → 120960 at load	runtime patch (see below)
MoE experts / moe_intermediate_size	192 / 1536	unchanged	already divisible by 3

The MoE (NVFP4), embeddings, lm_head, layernorms, and the MTP/NextN layer are otherwise copied byte-for-byte from the source (the MTP layer's attention is also padded).

Why vocab needs a runtime patch: vLLM pads vocab to ×64 by default; 120832 is ×64 but not ×3. The patch (container/hy3_tp3_patch.py) bumps vLLM's pad_vocab_size to lcm(64, tp)=192, so 120832 → 120960 with correct sampler masking. It loads this checkpoint unchanged and is a no-op for TP=2/4/8.

---

Quick start

A) 3× DGX Spark via sparkrun (the target use case)

git clone https://huggingface.co/marianbusoi/Hy3-NVFP4-FP8-TP3 hy3-tp3 && cd hy3-tp3

# 1. Build the vocab-patch image (layers a dormant patch on the upstream eugr image;
#    activates only at TP=3, no-op otherwise). Public base, anonymous pull works.
docker build -f container/Containerfile -t hy3-nvfp4-tp3:latest .

# 2. Define your 3-node cluster (direct CX7 ring/mesh)
sparkrun cluster create sparks --hosts <h1>,<h2>,<h3> --topology ring

# 3. Serve (sparkrun downloads THIS checkpoint to each node automatically)
sparkrun run hy3-nvfp4-fp8-tp3.yaml --cluster sparks
sparkrun logs hy3-nvfp4-fp8-tp3

The included hy3-nvfp4-fp8-tp3.yaml uses model: marianbusoi/Hy3-NVFP4-FP8-TP3, tensor_parallel: 3, MTP speculative decode, hy_v3 tool/reasoning parsers, and the patched image. No manual weight sync or padding required — the checkpoint is already padded.

B) Any 3-GPU host with bare vLLM

docker build -f container/Containerfile -t hy3-nvfp4-tp3 .
docker run --gpus all --ipc host --privileged \
  -v $HOME/.cache/huggingface:/cache/huggingface -e HF_HOME=/cache/huggingface \
  hy3-nvfp4-tp3 \
  vllm serve marianbusoi/Hy3-NVFP4-FP8-TP3 \
    --tensor-parallel-size 3 \
    --tool-call-parser hy_v3 --reasoning-parser hy_v3 --enable-auto-tool-choice \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

C) TP=2 or TP=4 — stock vLLM, no patch needed

Because the padding is inert, this checkpoint also runs on any TP that divides the original dims with the stock eugr/vLLM image (no vocab patch):

vllm serve marianbusoi/Hy3-NVFP4-FP8-TP3 --tensor-parallel-size 2 \
    --tool-call-parser hy_v3 --reasoning-parser hy_v3 --enable-auto-tool-choice

---

Repo layout

config.json, *.safetensors, model.safetensors.index.json   # the padded checkpoint
tokenizer.json, tokenizer_config.json, chat_template.jinja  # from the source model
hy3-nvfp4-fp8-tp3.yaml          # sparkrun recipe (3-node, TP=3, MTP)
container/
  Containerfile                  # builds hy3-nvfp4-tp3 (eugr + vocab-pad patch)
  hy3_tp3_patch.py               # runtime patch (bumps pad_vocab_size to lcm(64,tp))
  hy3_tp3_patch.pth              # auto-import hook (site-packages)
scripts/
  hy3_tp3_pad.py                 # offline padder that produced this checkpoint (pure stdlib)
  pad-cluster.sh                 # pads on every Spark over SSH
  test_pad.py                    # round-trip test (real safetensors reader/writer)
  pad_math_proof.py              # proves padded heads are inert (numpy)
patches/
  hy_v3_tp3.patch                # source diff for vLLM's hy_v3.py (fork builders)
LICENSE

How the checkpoint was made

scripts/hy3_tp3_pad.py (pure stdlib, streaming safetensors rewrite — no torch/numpy needed) zero-pads the attention heads and the dense-MLP intermediate, bumps config.json, and copies everything else byte-for-byte. python scripts/test_pad.py round-trips it through the real safetensors reader; scripts/pad_math_proof.py proves losslessness at the real 64/8 → 72/9 dims.

Requirements / compatibility

• A vLLM build with hy_v3 support (the upstream dgx-vllm-eugr-nightly image ships it; stock vLLM ≥ 0.26.0 also has HYV3ForCausalLM).
• For TP=3 on DGX Spark: the hy3-nvfp4-tp3 image built from container/ (for the vocab patch).

Attribution & license

Derivative of RedHatAI/Hy3-NVFP4-FP8 (MIT) ← tencent/Hy3. Released under MIT, consistent with the source. All credit to Red Hat AI and Tencent for the model and the NVFP4/FP8 quantization; the TP=3 padding tooling here is MIT.