README.md

---
license: apache-2.0
base_model: tencent/Hy3
base_model_relation: quantized
pipeline_tag: text-generation
library_name: vllm
tags:
  - hunyuan
  - hy_v3
  - mixture-of-experts
  - moe
  - 4-bit
  - int4
  - w4a16
  - compressed-tensors
  - quantized
  - vllm
  - long-context
  - yarn
---

# Hy3-1M — 4-bit (INT4) quantization of tencent/Hy3 for 1M context

A **4-bit weight-only (W4A16)** quantization of **[tencent/Hy3](https://huggingface.co/tencent/Hy3)**
(`HYV3ForCausalLM`, `hy_v3`) — a 295B-parameter / 21B-active Mixture-of-Experts model.
Packaged in the **`compressed-tensors`** `pack-quantized` format so it loads directly in **vLLM**.

## Why this model

- **Small.** **~146 GB** vs **~557 GB** for the original BF16 (~3.8× smaller). The whole
  295B MoE now **fits on a single ≥180 GB GPU** (e.g. one B200 192 GB / B300 ~288 GB)
  with KV-cache headroom — no tensor-parallel sharding required just to load it.
  (Note: it does **not** fit a 141 GB H200 without offload/TP.)
- **vLLM-native.** Loads out of the box with vLLM (recent build with `hy_v3` support) using the
  Marlin INT4 MoE + Linear kernels. Fast tensor-core prefill.
- **Long context via YaRN.** With YaRN RoPE scaling the context extends from the native
  **262,144** up to **1,048,576 (1M)** tokens (configurable). Dense needle-in-a-haystack
  retrieval is **verified past native (1M, PASS)** on a single GPU; see *Long context* below.

## Verified results (single B300, this checkpoint)

| Test | Result |
|---|---|
| **HumanEval pass@1 (greedy)** | ✅ **150/164 = 91.5%** — coding ability well-preserved at 4-bit |
| **GSM8K (0-shot CoT, greedy)** | ✅ **1265/1319 = 95.9%** — math reasoning preserved at 4-bit |
| Chat sanity  | ✅ correct |
| Needle-in-a-haystack 4K / 16K / 64K / 137K (in-range) | ✅ all PASS |
| **Needle-in-a-haystack 320K/1M dense (YaRN ×4, fp8/int4 KV)** | ✅ **PASS** — retrieval works past the native 262,144 |

## Quantization details

| | |
|---|---|
| Scheme | **W4A16** — 4-bit **int**, **symmetric**, **group_size=128**, RTN (round-to-nearest, data-free) |
| Format | `compressed-tensors` **`pack-quantized`** (`quant_method: compressed-tensors`) |
| Quantized | attention `q/k/v/o_proj`, dense-layer FFN, **all 192 routed experts** + shared expert (`gate/up/down_proj`) |
| Kept in original precision | `lm_head`, router gate (`mlp.router.gate`), `eh_proj` (MTP), all norms, `embed_tokens` |
| Base dtype | bf16 (scales stored bf16) |

Produced by a direct tensor-by-tensor RTN packer (no calibration dataset). RTN keeps the
pipeline simple and lossless-format-correct; for maximum quality at 4-bit, a calibrated
GPTQ/AWQ pass would be marginally better.

## Running with vLLM

Requires a vLLM build new enough to include the `hy_v3` architecture (vLLM `main`/nightly at time of
writing). Example on a single GPU:

```bash
vllm serve /path/to/Hy3-1M \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code
```

**NVIDIA Blackwell (sm_100/sm_103, e.g. B200/B300) note:** at the time of testing, FlashInfer's
runtime JIT could not compile for `compute_103a` with the bundled CUDA toolkit, which crashed the
default sampler/attention. Work around it with the Triton attention backend + native sampler:

```bash
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/Hy3-1M \
  --attention-backend TRITON_ATTN \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code --enforce-eager
```

`--kv-cache-dtype fp8` halves KV memory (recommended for long context). On Hopper/Ada or with a
FlashInfer build that supports your GPU, you can drop the two workaround flags.

## Inference tuning: MoE top-K (speed vs quality)

The number of routed experts per token (`num_experts_per_tok`, native **8**) can be lowered at
**inference time** (no re-quantization) to trade quality for less expert compute, via vLLM's
`--hf-overrides '{"num_experts_per_tok": K}'`. Measured on this 4-bit checkpoint (greedy):

| top-K | HumanEval | GSM8K | routed-expert FLOPs |
|---|---|---|---|
| **8** (native) | **91.5%** | **95.9%** | 100% |
| **6** | 89.6% (−1.9) | 94.8% (−1.1) | ~75% |
| **4** | 86.6% (−4.9) | 93.5% (−2.4) | ~50% |

Degradation is **graceful** — even top-4 (half the routed-expert compute) stays coherent and
usable. **top-6** is a sweet spot (~25% less expert compute for ≈1-2 pts). Coding is a bit more
sensitive to fewer experts than math. (Default = 8; only lower it if you need the speed/energy.)

## Long context (YaRN)

The base model is `rope_type: "default"` with `max_position_embeddings: 262144`. To go beyond,
enable YaRN in `config.json`:

```json
"rope_parameters": {
  "rope_theta": 11158840.0,
  "rope_type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}
```

and raise `--max-model-len` (up to `262144 * factor = 1048576`).

> **This shipped `config.json` already has YaRN factor 4 enabled** (context up to 1,048,576).
> Set `rope_type` back to `"default"` if you want the native-only 262,144 behavior.

Memory on a single ~275 GB GPU (146 GB weights):
- **fp8 KV** (`--kv-cache-dtype fp8`): comfortably fits **~500K** dense tokens; fast tensor-core prefill.
- **int4 KV** (`--kv-cache-dtype int4_per_token_head`): fits **~1M** dense tokens, but its kernel is
  compute-bound and much slower for long prefill.
- Full **dense 1M** is best served with **multi-GPU** (tensor-parallel) for both memory and speed.

## Verified results (single B300, this checkpoint)

| Test | Result |
|---|---|
| **HumanEval pass@1 (greedy)** | ✅ **150/164 = 91.5%** — coding ability well-preserved at 4-bit |
| **GSM8K (0-shot CoT, greedy)** | ✅ **1265/1319 = 95.9%** — math reasoning preserved at 4-bit |
| Chat sanity (`11+22+33` → 66; capital of France → Paris; first 5 primes) | ✅ correct |
| Needle-in-a-haystack 4K / 16K / 64K / 137K (in-range) | ✅ all PASS |
| **Needle-in-a-haystack 1M dense (YaRN ×4, fp8 KV)** | ✅ **PASS** — retrieval works past the native 262,144 |


<details>
<summary><b>How HumanEval was measured</b> (for reproducibility)</summary>

- **Engine/config:** this W4A16 checkpoint served by vLLM on a single B300, **as shipped**
  (YaRN factor 4 enabled), `--attention-backend TRITON_ATTN --kv-cache-dtype fp8
  --enforce-eager`, `VLLM_USE_FLASHINFER_SAMPLER=0`.
- **Data:** the 164 problems from `openai_humaneval` (`human-eval` package).
- **Decoding:** **greedy** (`temperature=0`), `max_tokens=768`, **pass@1** (single sample per problem).
- **Prompting:** chat template with the instruction *"Complete the following Python function.
  Return the COMPLETE function in a single ```python code block. No tests, no explanations."*
  The first ```python block is extracted; if it lacks the `entry_point` def, the original stub
  is prepended.
- **Scoring:** each candidate is run as `candidate + test + check(entry_point)` in a subprocess
  (15s timeout); exit-code 0 = pass.
- **Result:** **150/164 = 91.5%**. Note this uses a chat+extraction harness (not the canonical
  raw-completion protocol), so a few of the 14 misses may be extraction artifacts — treat 91.5%
  as a conservative figure.

**GSM8K:** full 1319-problem test set, **0-shot chain-of-thought**, greedy (`temperature=0`,
`max_tokens=512`), prompt *"Solve step by step… on the last line write 'The answer is <number>'"*;
the final number is compared to the gold answer (after `####`). Result: **1265/1319 = 95.9%**.

</details>

## Caveats & honesty

- This is a **community derivative**, not affiliated with or endorsed by Tencent.

## License

Apache-2.0, inheriting the license of the base model
[`tencent/Hy3`](https://huggingface.co/tencent/Hy3) (see `LICENSE`).