README.md

---
license: other
license_name: tencent-hy-community-license
license_link: LICENSE
base_model: tencent/Hy3
base_model_relation: quantized
quantized_by: LibertAIDAI
tags:
  - nvfp4
  - blackwell
  - vllm
  - hy3
  - hunyuan
  - moe
  - modelopt
language:
  - en
pipeline_tag: text-generation
---

# Hy3 NVFP4 (weight-only)

NVFP4 quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) — Tencent's Hy3 (final) 295B Mixture-of-Experts model (21B active, 192 routed + 1 shared expert, 8 experts/token, 256K context, unified fast/deep-reasoning) — for serving on [vLLM](https://github.com/vllm-project/vllm).

This is a **weight-only NVFP4 (NVFP4-A16)** checkpoint. The **routed-expert FFN** tensors — which are ~98% of the model — are quantized to **NVFP4** (NVIDIA's 4-bit float, E2M1 with FP8-E4M3 per-16-block scales and an FP32 per-tensor global scale). Every outlier-sensitive path is deliberately kept in **BF16**: all self-attention (Q/K/V/O), the shared expert, the MoE routers/gates, the dense-MLP layer, the MTP head, token embeddings, `lm_head`, and all norms. Activations stay BF16 (no activation quantization). Because the experts dominate the memory footprint, quantizing only them recovers almost all of the size reduction while protecting quality where it matters most.

Produced with [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.45.0 via a memory-frugal shard-streaming pass (no calibration data — weight-only NVFP4 derives its scales from the weights themselves), so the packing is exactly what vLLM's NVFP4 loader expects.

## About LibertAI

[LibertAI](https://libertai.io) is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over [Aleph Cloud](https://aleph.cloud) instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.

If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out [**LiberClaw**](https://liberclaw.ai) — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.

> **Why NVFP4?** On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200, GB10/DGX Spark) NVFP4 weights run on native FP4 tensor-core kernels, and the 4-bit expert weights cut both VRAM and the memory bandwidth that dominates MoE decode. This gives format parity with the NVFP4 path used by vLLM / SGLang / TensorRT-LLM.

## What's quantized

| Tensor group | Precision | Count |
|---|---|---|
| Routed-expert FFN (`layers.*.mlp.experts.*.{gate,up,down}_proj`) | **NVFP4** (g16, weight-only) | 46,080 |
| Self-attention, shared expert, routers, dense/MTP MLP, embeddings, `lm_head`, norms | BF16 | rest |

## Files

99 sharded `safetensors` (~169 GB total) + `config.json` carrying the `quantization_config` (ModelOpt NVFP4, weight-only). Load directly in vLLM.

## Usage (vLLM)

On **datacenter Blackwell (B200, sm_100)** or **RTX 50-series (sm_120)** the default NVFP4 MoE kernels work out of the box:

```bash
vllm serve LibertAIDAI/Hy3-NVFP4 \
  --tensor-parallel-size 2 \
  --trust-remote-code
```

On **GB10 / DGX Spark (sm_121)** the native FP4 MoE kernels currently have gaps, so force the **marlin** MoE backend (dequant-to-FP16 path, works on sm_121):

```bash
# 2× GB10, CUDA-13 image
vllm serve LibertAIDAI/Hy3-NVFP4 \
  --tensor-parallel-size 2 \
  --moe-backend marlin \
  --enforce-eager \
  --trust-remote-code
```

Requires a vLLM build with `HYV3ForCausalLM` support (vLLM ≥ 0.23) and a **CUDA-13** image on GB10. A full **2× GB10 deployment recipe** (multinode fabric, memory tuning, the exact kernel gotchas) is in [`deploy/`](./deploy). Verified serving on a 2× NVIDIA GB10 (Grace Blackwell, sm_121) cluster at tensor-parallel = 2 — coherent generation, correct arithmetic, `hy_v3` tool calls, and 256K needle retrieval.

## Benchmarks — 2× NVIDIA GB10 (DGX Spark)

Served on **two GB10 boxes** (Grace Blackwell, sm_121, ~120 GB unified each) at **tensor-parallel 2** — marlin FP4 MoE, fp8 KV cache, prefix caching, `--enforce-eager --max-model-len 262144 --gpu-memory-utilization 0.90`.

![Hy3-NVFP4 benchmarks on 2× GB10 / DGX Spark](bench-2xgb10.png)

- **KV cache: 286,640 tokens** — a full **256K**-token request fits (weights ~84.5 GB/node).
- **Throughput scales with concurrency** (128-tok completions, greedy):

| Concurrency | 1 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|
| Aggregate tok/s | 13.3 | 41.7 | 65.2 | 99.8 | **165.8** |
| Per-request tok/s | 13.3 | 10.5 | 8.2 | 6.3 | 5.2 |

- **Prefix caching:** a repeated ~6.3K-word prompt is served in **0.32 s vs 3.95 s cold — a 92% cut**.
- **Long context:** ~120K-token prefill ≈ 138 s.

> **Note on speed.** On GB10 (sm_121) CUDA graphs livelock during capture with the marlin MoE kernel, so `--enforce-eager` is required and single-stream tops out around 13 tok/s — aggregate throughput is where the two-box setup delivers. A native FP4 MoE path (vLLM's b12x / CuTe-DSL) will lift the single-stream ceiling once its sm_121 build ships. On **B200 (sm_100)** or **RTX 50-series (sm_120)** the native NVFP4 kernels run without this constraint.

## Provenance & method

- Base: `tencent/Hy3` (BF16, 99 shards, 598 GB).
- Quantizer: shard-streaming ModelOpt `NVFP4QTensor` weight-only pass; expert FFN → NVFP4, everything else copied through in BF16.
- Verification: per-expert round-trip cosine ≈ 0.995 vs the BF16 source.

_Quantized by [LibertAI](https://libertai.io). Not affiliated with Tencent. Subject to the [Tencent Hy Community License](LICENSE)._