--- license: apache-2.0 base_model: tencent/Hy3 base_model_relation: quantized pipeline_tag: text-generation library_name: vllm tags: - hunyuan - hy_v3 - mixture-of-experts - moe - 4-bit - int4 - w4a16 - compressed-tensors - quantized - vllm - long-context - yarn --- # Hy3-1M — 4-bit (INT4) quantization of tencent/Hy3 for 1M context A **4-bit weight-only (W4A16)** quantization of **[tencent/Hy3](https://huggingface.co/tencent/Hy3)** (`HYV3ForCausalLM`, `hy_v3`) — a 295B-parameter / 21B-active Mixture-of-Experts model. Packaged in the **`compressed-tensors`** `pack-quantized` format so it loads directly in **vLLM**. ## Why this model - **Small.** **~146 GB** vs **~557 GB** for the original BF16 (~3.8× smaller). The whole 295B MoE now **fits on a single ≥180 GB GPU** (e.g. one B200 192 GB / B300 ~288 GB) with KV-cache headroom — no tensor-parallel sharding required just to load it. (Note: it does **not** fit a 141 GB H200 without offload/TP.) - **vLLM-native.** Loads out of the box with vLLM (recent build with `hy_v3` support) using the Marlin INT4 MoE + Linear kernels. Fast tensor-core prefill. - **Long context via YaRN.** With YaRN RoPE scaling the context extends from the native **262,144** up to **1,048,576 (1M)** tokens (configurable). Dense needle-in-a-haystack retrieval is **verified past native (1M, PASS)** on a single GPU; see *Long context* below. ## Verified results (single B300, this checkpoint) | Test | Result | |---|---| | **HumanEval pass@1 (greedy)** | ✅ **150/164 = 91.5%** — coding ability well-preserved at 4-bit | | **GSM8K (0-shot CoT, greedy)** | ✅ **1265/1319 = 95.9%** — math reasoning preserved at 4-bit | | Chat sanity | ✅ correct | | Needle-in-a-haystack 4K / 16K / 64K / 137K (in-range) | ✅ all PASS | | **Needle-in-a-haystack 320K/1M dense (YaRN ×4, fp8/int4 KV)** | ✅ **PASS** — retrieval works past the native 262,144 | ## Quantization details | | | |---|---| | Scheme | **W4A16** — 4-bit **int**, **symmetric**, **group_size=128**, RTN (round-to-nearest, data-free) | | Format | `compressed-tensors` **`pack-quantized`** (`quant_method: compressed-tensors`) | | Quantized | attention `q/k/v/o_proj`, dense-layer FFN, **all 192 routed experts** + shared expert (`gate/up/down_proj`) | | Kept in original precision | `lm_head`, router gate (`mlp.router.gate`), `eh_proj` (MTP), all norms, `embed_tokens` | | Base dtype | bf16 (scales stored bf16) | Produced by a direct tensor-by-tensor RTN packer (no calibration dataset). RTN keeps the pipeline simple and lossless-format-correct; for maximum quality at 4-bit, a calibrated GPTQ/AWQ pass would be marginally better. ## Running with vLLM Requires a vLLM build new enough to include the `hy_v3` architecture (vLLM `main`/nightly at time of writing). Example on a single GPU: ```bash vllm serve /path/to/Hy3-1M \ --max-model-len 262144 \ --gpu-memory-utilization 0.9 \ --trust-remote-code ``` **NVIDIA Blackwell (sm_100/sm_103, e.g. B200/B300) note:** at the time of testing, FlashInfer's runtime JIT could not compile for `compute_103a` with the bundled CUDA toolkit, which crashed the default sampler/attention. Work around it with the Triton attention backend + native sampler: ```bash VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/Hy3-1M \ --attention-backend TRITON_ATTN \ --kv-cache-dtype fp8 \ --max-model-len 262144 \ --gpu-memory-utilization 0.9 \ --trust-remote-code --enforce-eager ``` `--kv-cache-dtype fp8` halves KV memory (recommended for long context). On Hopper/Ada or with a FlashInfer build that supports your GPU, you can drop the two workaround flags. ## Inference tuning: MoE top-K (speed vs quality) The number of routed experts per token (`num_experts_per_tok`, native **8**) can be lowered at **inference time** (no re-quantization) to trade quality for less expert compute, via vLLM's `--hf-overrides '{"num_experts_per_tok": K}'`. Measured on this 4-bit checkpoint (greedy): | top-K | HumanEval | GSM8K | routed-expert FLOPs | |---|---|---|---| | **8** (native) | **91.5%** | **95.9%** | 100% | | **6** | 89.6% (−1.9) | 94.8% (−1.1) | ~75% | | **4** | 86.6% (−4.9) | 93.5% (−2.4) | ~50% | Degradation is **graceful** — even top-4 (half the routed-expert compute) stays coherent and usable. **top-6** is a sweet spot (~25% less expert compute for ≈1-2 pts). Coding is a bit more sensitive to fewer experts than math. (Default = 8; only lower it if you need the speed/energy.) ## Long context (YaRN) The base model is `rope_type: "default"` with `max_position_embeddings: 262144`. To go beyond, enable YaRN in `config.json`: ```json "rope_parameters": { "rope_theta": 11158840.0, "rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144 } ``` and raise `--max-model-len` (up to `262144 * factor = 1048576`). > **This shipped `config.json` already has YaRN factor 4 enabled** (context up to 1,048,576). > Set `rope_type` back to `"default"` if you want the native-only 262,144 behavior. Memory on a single ~275 GB GPU (146 GB weights): - **fp8 KV** (`--kv-cache-dtype fp8`): comfortably fits **~500K** dense tokens; fast tensor-core prefill. - **int4 KV** (`--kv-cache-dtype int4_per_token_head`): fits **~1M** dense tokens, but its kernel is compute-bound and much slower for long prefill. - Full **dense 1M** is best served with **multi-GPU** (tensor-parallel) for both memory and speed. ## Verified results (single B300, this checkpoint) | Test | Result | |---|---| | **HumanEval pass@1 (greedy)** | ✅ **150/164 = 91.5%** — coding ability well-preserved at 4-bit | | **GSM8K (0-shot CoT, greedy)** | ✅ **1265/1319 = 95.9%** — math reasoning preserved at 4-bit | | Chat sanity (`11+22+33` → 66; capital of France → Paris; first 5 primes) | ✅ correct | | Needle-in-a-haystack 4K / 16K / 64K / 137K (in-range) | ✅ all PASS | | **Needle-in-a-haystack 1M dense (YaRN ×4, fp8 KV)** | ✅ **PASS** — retrieval works past the native 262,144 |
How HumanEval was measured (for reproducibility) - **Engine/config:** this W4A16 checkpoint served by vLLM on a single B300, **as shipped** (YaRN factor 4 enabled), `--attention-backend TRITON_ATTN --kv-cache-dtype fp8 --enforce-eager`, `VLLM_USE_FLASHINFER_SAMPLER=0`. - **Data:** the 164 problems from `openai_humaneval` (`human-eval` package). - **Decoding:** **greedy** (`temperature=0`), `max_tokens=768`, **pass@1** (single sample per problem). - **Prompting:** chat template with the instruction *"Complete the following Python function. Return the COMPLETE function in a single ```python code block. No tests, no explanations."* The first ```python block is extracted; if it lacks the `entry_point` def, the original stub is prepended. - **Scoring:** each candidate is run as `candidate + test + check(entry_point)` in a subprocess (15s timeout); exit-code 0 = pass. - **Result:** **150/164 = 91.5%**. Note this uses a chat+extraction harness (not the canonical raw-completion protocol), so a few of the 14 misses may be extraction artifacts — treat 91.5% as a conservative figure. **GSM8K:** full 1319-problem test set, **0-shot chain-of-thought**, greedy (`temperature=0`, `max_tokens=512`), prompt *"Solve step by step… on the last line write 'The answer is '"*; the final number is compared to the gold answer (after `####`). Result: **1265/1319 = 95.9%**.
## Caveats & honesty - This is a **community derivative**, not affiliated with or endorsed by Tencent. ## License Apache-2.0, inheriting the license of the base model [`tencent/Hy3`](https://huggingface.co/tencent/Hy3) (see `LICENSE`).