| --- |
| license: other |
| license_name: tencent-hunyuan-community |
| base_model: tencent/Hy3-preview |
| tags: |
| - mlx |
| - speculative-decoding |
| - mtp |
| - multi-token-prediction |
| - hunyuan |
| library_name: mlx |
| --- |
| |
| # Hy3-preview-MTP-4bit |
|
|
| Native **Multi-Token-Prediction (MTP)** sidecar for |
| [`mlx-community/Hy3-preview-4bit`](https://huggingface.co/mlx-community/Hy3-preview-4bit), |
| for use as a self-speculative draft head with |
| [rapid-mlx](https://github.com/machinefi/rapid-mlx). |
|
|
| ## What this is |
|
|
| Tencent's Hunyuan 3 (`model_type=hy_v3`) ships a **DeepSeek-V3-style native MTP |
| head** as the final decoder layer (`model.layers.80.*`) of the full-precision |
| [`tencent/Hy3-preview`](https://huggingface.co/tencent/Hy3-preview) checkpoint. |
| The 4-bit MLX conversion `mlx-community/Hy3-preview-4bit` keeps only layers |
| `0..79` (the backbone) and **strips** the MTP head. |
|
|
| This repo re-supplies just that head, quantized to match the base checkpoint, |
| as a single-file sidecar (`model-mtp.safetensors`, 44 tensors). rapid-mlx loads |
| the base 4-bit backbone and grafts this head at boot to run **self-speculative |
| decoding** (one draft token per verify step, K=1 chain MTP). |
|
|
| ## Provenance |
|
|
| Extracted from `tencent/Hy3-preview` shards `model-00111-of-00112` + |
| `model-00112-of-00112` (the two shards holding layer 80). The 593 layer-80 |
| tensors are remapped to the rapid-mlx MTP param tree: |
|
|
| * `enorm` / `hnorm` — RMSNorms on the next-token embedding and previous hidden |
| state (DeepSeek-V3 convention). |
| * `eh_proj` — the `2H -> H` fused projection, applied as |
| `eh_proj(concat([enorm(embed_next), hnorm(prev_hidden)], -1))` (embedding |
| first, confirmed against vLLM `deepseek_mtp.py` and SGLang `hunyuan` nextn). |
| * one HY3 `DecoderLayer` on the MoE branch (QK-norm attention + sigmoid-router |
| SwitchGLU MoE over 192 experts + a shared expert). |
| * `norm` — the head's final RMSNorm (upstream `final_layernorm`). |
|
|
| ## Quantization |
|
|
| Matches the base checkpoint: **4-bit `group_size=64` affine** for every Linear |
| (`eh_proj`, attention projections, `switch_mlp.*`, `shared_mlp.*`); **8-bit |
| `group_size=64`** for `mlp.router.gate`; all RMSNorms and `router.expert_bias` |
| kept in full precision. |
| |
| ## Usage |
| |
| ```bash |
| rapid-mlx serve hy3-preview-4bit --speculative-config '{"method":"mtp"}' |
| ``` |
| |
| rapid-mlx auto-resolves and downloads this sidecar. The base 4-bit backbone |
| loads normally; the MTP head is grafted at boot for self-speculative decoding. |
| |
| ## Measured |
| |
| * Draft accept rate ~58% (K=1) across code / chinese / reasoning / list prompts. |
| * Greedy output is batched-consistent lossless vs the MTP-off reference. |
| |
| The projection tree is byte-identical to a quantized backbone MoE layer, so the |
| sidecar param names line up 1:1 with the runtime module. |
| |
