canada-quant/hy3-w4a16-mtp

Hy3 W4A16-MTP

Released 2026-07-16. All gates passed: artifact verification (--check-mtp), quality vs FP8 (standard + reasoning-mode), MTP acceptance, serving smoke tests, and a full benchmark matrix against the FP8 baseline and the leading 4-bit community quants (complete methodology + audit trail in the companion repo's docs/RUNLOG.md).

Tencent Hy3 quantized to W4A16 with GPTQ — routable MoE experts in INT4, all other weights (attention, dense layer 0, shared expert, NextN/MTP head) preserved in BF16. vLLM-ready (selects the INT4×FP16 Marlin/Machete kernel at load).

Why this artifact

Same box, same harness, same settings for every column (8×H100-SXM, vLLM 0.25.1 — methodology below):

	This model	FP8 (tencent)	AngelSlim GPTQ	cyankiwi AWQ
Disk size	172 GB	≈300 GB	166 GB	175 GB
AIME24 / AIME25 (thinking)	70.0 / 83.3	70.0 / 90.0	70.0 / 83.3	80.0 / 76.7
MATH-500 / GPQA-diamond	94.0 / 88.4	93.4 / 89.9	93.4 / 91.9	95.2 / 88.9
MTP speculative decoding	✅ BF16, preserved	✅	❌ layer missing — cannot speculate	✅
Single-stream tok/s (best spec cfg)	144 (k=2)	143	103 (capped, no MTP)	152
Throughput @ conc 32 (best cfg)	2,146	1,854	2,135	2,028

Quality is statistical parity with the FP8 baseline at 57% of its footprint — and among the 4-bit quants, this is the artifact that combines preserved MTP (+40% single-stream vs the same-scheme, MTP-less AngelSlim), the high-concurrency throughput crown (+21% over FP8 at conc 32), BF16 attention (fewest unvalidated surfaces at long context), and a fully reproducible, audited pipeline.

Base model	tencent/Hy3 (295B MoE, BF16)
Architecture	MoE · 80 layers (79 MoE + 1 dense) · 192 experts/MoE-layer · top-8 routing · GQA attention
Quantization	GPTQ · W4A16 · INT4 group_size=128 symmetric (routed experts only)
Precision mix	≈97% INT4 (experts) / ≈3% BF16 (attention, dense L0, shared, router, embeds, NextN/MTP)
Disk size	≈172 GB (vs ≈598 GB BF16 source — ≈3.4× compression)
Calibration data	Agentic+code+math+chat blend (512 samples × 2048 tokens; see RUNLOG)
NextN/MTP layer	model.layers.80 preserved BF16 (re-injected post-quantization), num_speculative_tokens=2
Serving	vLLM ≥ 0.25 (compressed-tensors W4A16 → Marlin/Machete kernel at load)
Hardware	8× H200 SXM5 (141 GB) or 8× RTX PRO 6000 (96 GB)

Quick Start

Stock vLLM ≥ 0.25 — no custom patches or forks. One-shot script and Docker image in the companion repo (scripts/quickstart_hopper.sh, docker/); full Hopper guide in docs/deploy-hopper.md.

# Install vLLM (hy_v3 support since v0.20.0; >=0.25 has expert_bias/token-suffix fixes)
pip install 'vllm>=0.25'

# Serve with NextN/MTP speculative decoding on 8 GPUs
vllm serve canada-quant/hy3-w4a16-mtp \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --tool-call-parser hy_v3 --reasoning-parser hy_v3 --enable-auto-tool-choice
# --enable-expert-parallel is REQUIRED at TP=8: g128 W4A16 can't split Hy3's
# 1536-wide experts 8 ways (192 % 128 != 0). Alternative: TP=4 without EP.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="hy3-w4a16-mtp",
    messages=[{"role": "user", "content": "Explain quantum computing in one paragraph."}],
    max_tokens=256,
)
print(response.choices[0].message.content)

What Is This?

This is a W4A16-MTP quantized artifact of Tencent Hy3. Approximately 97% of the model's parameters — the routed MoE expert projections (gate/up/down) of layers 1–79 — are compressed to INT4 (group_size=128, symmetric). The rest — attention, dense layer 0, shared expert, embeddings, layer norms, router, LM head, and the NextN/MTP layer — stays in BF16.

The result is a ≈3.4× reduction in on-disk footprint (≈598 GB → ≈172 GB) with minimal quality degradation, while retaining MTP speculative decoding. The artifact serves on 8×, 4×, or (short-context) 2× H200.

What Gets Quantized

Component	Precision	≈Params
Routed experts, layers 1–79 (45,504 matrices)	INT4	≈286B
Self-attention (GQA, 81 layers)	BF16	≈6.1B
Shared expert + dense layer 0 FFN	BF16	≈1.7B
Embeddings + LM head (untied)	BF16	≈1.0B
Router / gate + expert_bias	BF16	≈0.1B
NextN / MTP layer 80 (incl. its own 192 experts)	BF16 (re-injected)	≈3.7B

Why W4A16?

W4A16 (INT4 weights × FP16 activations) is the sweet spot for large MoE models:

• A16 keeps activations in FP16, matching the Marlin/Machete INT4×FP16 matmul kernel and avoiding the quality loss from activation quantization.
• W4 halves the weight footprint vs INT8 while maintaining quality for expert projections — the ensemble effect of top-8 routing mitigates per-expert quantization error.
• Kernel readiness — vLLM selects the INT4×FP16 Marlin/Machete kernel at load from the compressed-tensors metadata.

NextN / MTP Speculative Decoding

Hy3 includes 1 NextN / MTP layer (num_nextn_predict_layers=1) for speculative decoding. This ≈3.8B-parameter draft head predicts one extra token, allowing the main model to verify and accept it in the same forward pass.

The NextN head is preserved in BF16. Quantizing the draft head stacks its error on top of the hidden-state shift from main-body quantization and compounds across speculative steps, collapsing the acceptance rate.

vLLM has a native hy_v3 MTP proposer (HYV3MTPModel) since v0.20.0; the unified {"method":"mtp"} speculative config selects it automatically. The MTP layer must be present in the checkpoint in BF16 — this artifact re-injects it post-quantization and lists it in quantization_config.ignore (a quantized MTP layer does not load; vLLM #47792). Tencent's official recommendation is num_speculative_tokens=2; the single NextN layer is re-run per speculative step.

See findings/mtp-preservation-strategy.md for the full analysis.

Quality

Measured against the FP8 deployment baseline (tencent/Hy3-FP8), served on the same 8×H100 box with identical settings, using lm-eval 0.4.12 through vLLM's OpenAI-compatible endpoints. Generation tasks are greedy (temperature 0), with the chat template applied; speculative decoding is off for quality runs (it changes latency, not outputs). GSM8K is 5-shot; wikitext perplexity is loglikelihood_rolling via the completions endpoint. (BF16 is not a practical baseline — it does not fit on 8×H100, and FP8 is the deployment-relevant comparison.)

Metric	This model (W4A16-MTP)	FP8 baseline	Δ (ours − FP8)
GSM8K (5-shot, strict-match)	93.6%	93.0%	+0.7 pp
IFEval (prompt-strict / inst-strict)	89.7% / 92.9%	90.2% / 93.1%	−0.5 / −0.1 pp
HumanEval (pass@1)	94.5%	94.5%	0.0 pp
wikitext word-PPL / bits-per-byte	5.75 / 0.472	5.59 / 0.464	+0.16 / +0.008
NextN/MTP acceptance (greedy, aggregate)	62.1%	—	(BF16 draft head)

Every task is within ≈0.7 pp of FP8 — inside eval noise (GSM8K ±0.65 pp, IFEval ±≈1.3 pp, HumanEval ±1.8 pp) — so the INT4 experts are effectively quality-neutral vs FP8 at roughly half the footprint (≈172 GB vs ≈300 GB). MTP acceptance of 62.1% aggregate (position-0 76.8%) exceeds the ≈46.7% ungated-greedy figure reported for the official BF16 stack, giving ≈2.24 effective tokens per decode step.

Eval harness/config recorded in docs/RUNLOG.md. Numbers are for this calibration run (agentic+code blend, 512 samples × 2048 tokens); re-running with a different calibration set or harness settings can shift them by ≈noise.

Reasoning mode (thinking) — hard benchmarks vs 4-bit community quants

Hy3 gates reasoning via its chat template (reasoning_effort ∈ no_think/low/high, default no_think — the tables above are the conventional non-thinking protocol). The table below runs thinking=high (0-shot, 14k-token generation budget, \boxed{}/letter-grading via math_verify, identical template/harness/server settings for every model, spec-decode off):

All rows: 0-shot, thinking=high, native 256K context (max_model_len 262144), 131,072-token generation budget — uniform for every model, no truncation possible for any observed chain. (Earlier 14k/32k-budget runs are superseded: reasoning budget dominates AIME scores.) AIME24/25 are n=30 each (±7–9pp stderr) — read combined. GPQA is flexible-extract.

Bench (thinking=high, 131k budget)	Ours W4A16-MTP	FP8	cyankiwi AWQ	AngelSlim GPTQ
AIME24	70.0%	70.0%	80.0%	70.0%
AIME25	83.3%	90.0%	76.7%	83.3%
AIME combined (n=60)	76.7%	80.0%	78.3%	76.7%
MATH-500 (math_verify)	94.0%	93.4%	95.2%	93.4%
GPQA-diamond (flexible)	88.4%	89.9%	88.9%	91.9%

Every quant is statistically indistinguishable from FP8 and from each other on every bench (AIME n=30 ±7–9pp; GPQA n=198 ±2pp; MATH-500 ±1pp) — 4-bit expert quantization is quality-neutral on this model regardless of algorithm (GPTQ/AWQ) or grouping (g32/g128). The differentiators are serving traits: AngelSlim ships without the MTP layer (cannot speculate at all — capped at 103 tok/s single-stream); ours preserves it in BF16 and holds the high-concurrency throughput crown. See the throughput table.

Budget sensitivity (ours, AIME24/25): 14k → 33/23%; 32k → 53/57%; 131k → 70/83% — the thinking budget, not the quant, was the binding constraint. All models compared at the same unconstrained budget.

How the 4-bit field compares (artifact-level facts)

Artifact	Method / scheme	Size	MTP (speculative) layer
canada-quant/hy3-w4a16-mtp (this)	GPTQ INT4 · g128 sym · experts-only	172 GB	✅ BF16, preserved (62.1% greedy acceptance)
AngelSlim/Hy3-GPTQ-Int4 (Tencent toolkit)	GPTQ INT4 · g128 sym · experts-only (same scheme)	166 GB	❌ absent (layer 80 dropped; num_nextn_predict_layers: 0) — cannot speculate
kodelow/Hy3-NVFP4-W4A16	NVFP4 (FP4 g16, MSE scales) · experts-only	181 GB	⚠️ present but quantized (k=1 recommended)
cyankiwi/Hy3-AWQ-INT4	AWQ INT4 asym g32 · experts+attention	175 GB	✅ BF16, preserved
tencent/Hy3-FP8	FP8 (reference baseline)	≈300 GB	✅

Notable: AngelSlim's artifact uses the same GPTQ scheme as ours but shipped without the MTP layer — the silent transformers drop this project's inject_mtp.py step exists to fix. On H100 (SM90, no native FP4), NVFP4 weight-only also runs on the Marlin kernel, so the throughput comparison below is same-kernel-class.

Throughput

Measured with vllm bench serve on a single 8×H100-80GB SXM box (TP=8 + expert-parallel, vLLM 0.25.1, random 1024-in/512-out) — every model at every valid speculative config (k = num_speculative_tokens; AngelSlim ships no MTP layer and cannot speculate). Output tok/s (single-stream TPOT ms at concurrency 1):

Config	conc 1 (TPOT)	conc 8	conc 32
FP8 off / k1 / k2	114.6 / 138.8 / 142.8 (6.6)	634 / 665 / 677	1762 / 1854 / 1833
Ours off / k1 / k2	103.7 / 133.9 / 144.2 (6.7)	727 / 822 / 798	2127 / 2146 / 1910
Ours k3 / k5	126.3 / 84.8	791 / 607	1786 / 1450
AWQ off / k1 / k2	110.4 / 141.5 / 152.3 (6.3)	696 / 761 / 800	1989 / 2028 / 1751
AngelSlim (off only)	103.0 (9.4)	731	2135

Takeaways:

• k=2 is the single-stream sweet spot (Tencent's recommendation confirmed); k=3 already regresses and k=5 is clearly counterproductive at every concurrency.
• k=1 is the best all-rounder for this artifact — top of the entire matrix at conc 8 (822) and conc 32 (2146) while still 133.9 tok/s single-stream.
• The MTP layer is worth +40% single-stream — AngelSlim (identical GPTQ scheme and identical quality, but MTP-less) is capped at 103 tok/s vs our 144.
• INT4 beats FP8 at batch: +21% at conc 32 (2127 vs 1762 spec-off) — less weight memory per token; and matches FP8 single-stream once MTP is on.
• AWQ's g32-asym edges single-stream (152.3) but falls off at conc 32 (1751 at k2); ours holds the high-concurrency crown.

Same box, same kernel class (Marlin WNA16) for all INT4 quants. H200 scales higher. Full raw logs + methodology: docs/RUNLOG.md.

How It Was Made

1. GPTQ calibration with llm-compressor oneshot() (nightly ≥ 0.12.1a20260708 for hy_v3 MoE support), using an agentic + code + math + chat blend (scripts/build_calibration_dataset.py, 512 samples × 2048 tokens) — INT4 degrades most on code/tool-call, so the calibration mix targets those experts. moe_calibrate_all_experts (default) ensures every expert receives calibration data despite top-8 routing. Wall time: ≈15.5 h on one H100 (single-process sequential onload).

2. MTP re-injection (scripts/inject_mtp.py): transformers does not instantiate Hy3's NextN layer (model.layers.80) and silently drops it on load, so the BF16 layer is copied back from the source checkpoint after quantization and added to quantization_config.ignore.

3. Verification checks safetensors integrity, dtype distribution, group_size=128 confirmation, and NextN/MTP BF16 preservation + ignore-list coverage (--check-mtp).

There is no separate Marlin-repack step: oneshot() emits compressed-tensors format directly, and vLLM selects the Marlin/Machete kernel internally at load.

The full reproducible recipe is at recipe/recipe.yaml.

Serving on Different Hardware

8× H200 SXM5 (Recommended)

bash scripts/serve_h200.sh
# Defaults: TP=8 + --enable-expert-parallel, max_model_len=8192, port 8000,
#           --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Per-GPU VRAM: ≈25 GB / 141 GB — ample headroom for long contexts (4×H200 also fits, ≈43 GB/GPU).

8× RTX PRO 6000 Blackwell

bash scripts/serve_rtxpro6000.sh
# Defaults: TP=8, max_model_len=4096, max_num_seqs=4

Per-GPU VRAM: ≈25 GB / 96 GB. PCIe-only interconnect — expect reduced throughput vs H200. Cap max_model_len at 4096. SM120 needs cu128+ builds; Marlin INT4 kernels ship for SM120.

Limitations

• Calibration data: Calibrated on a 512-sample code + math + chat blend (canada-quant/hy3-w4a16-mtp-calibration): Magicoder-OSS-Instruct-75K (code, ~38%), gsm8k + MetaMathQA (math), and ultrachat_200k (chat). An agentic/tool-call slice was intended (xLAM / ToolACE) but its source could not be loaded at calibration time, so that budget was redistributed to ultrachat (see docs/RUNLOG.md). Primarily English; quality may vary for other languages or domains not represented in calibration.
• NextN/MTP coupling: The NextN head is preserved in BF16 but not retrained against the quantized distribution. The hidden-state shift from expert quantization causes a modest acceptance-rate drop. This is expected to be within an acceptable range but is non-zero.
• Expert parallelism required at TP=8: g128 W4A16 cannot split the 1536-wide experts 8 ways; serve with --enable-expert-parallel or TP=4 (see Quick Start).
• Acceptance-rate framing: the official BF16 stack measures ≈46.7% ungated aggregate MTP acceptance (independent audit); per-draft-token acceptance at k=1 is much higher. Published numbers will state the exact metric.
• Non-expert sensitivity: Attention, dense layer 0, shared expert, embeddings, and the LM head remain in BF16 — they are excluded for quality, not because they cannot technically be quantized.
• Context length: Hy3 supports up to 256K natively; W4A16 quality at extreme context lengths (>32K) has not been evaluated.

Reproducibility

Every step is scripted and documented:

Step	Command	Output
Bootstrap	bash scripts/bootstrap_8xh200.sh	Weights + venv + dataset
Calibrate	bash scripts/calibrate.sh	compressed-tensors W4A16 artifact (no MTP yet)
Re-inject MTP	python scripts/inject_mtp.py /scratch/weights/hy3-bf16 /scratch/weights/hy3-w4a16-mtp	+ BF16 model.layers.80 shard, updated index + ignore
Verify	python scripts/verify_artifact.py /scratch/weights/hy3-w4a16-mtp --check-mtp	Integrity report

See findings/quantization-plan.md for the full execution plan, hyperparameter rationale, and risk analysis.

Citation

@misc{canada-quant2025hy3w4a16,
  title        = {Hy3 W4A16-MTP: GPTQ-Quantized Tencent Hy3 with MTP Speculative Decoding Preservation},
  author       = {canada-quant},
  year         = 2026,
  url          = {https://huggingface.co/canada-quant/hy3-w4a16-mtp}
}