README.md

metadata

license: other
license_name: tencent-hunyuan-community
base_model: tencent/Hy3-preview
tags:
  - mlx
  - speculative-decoding
  - mtp
  - multi-token-prediction
  - hunyuan
library_name: mlx

Hy3-preview-MTP-4bit

Native Multi-Token-Prediction (MTP) sidecar for mlx-community/Hy3-preview-4bit, for use as a self-speculative draft head with rapid-mlx.

What this is

Tencent's Hunyuan 3 (model_type=hy_v3) ships a DeepSeek-V3-style native MTP head as the final decoder layer (model.layers.80.*) of the full-precision tencent/Hy3-preview checkpoint. The 4-bit MLX conversion mlx-community/Hy3-preview-4bit keeps only layers 0..79 (the backbone) and strips the MTP head.

This repo re-supplies just that head, quantized to match the base checkpoint, as a single-file sidecar (model-mtp.safetensors, 44 tensors). rapid-mlx loads the base 4-bit backbone and grafts this head at boot to run self-speculative decoding (one draft token per verify step, K=1 chain MTP).

Provenance

Extracted from tencent/Hy3-preview shards model-00111-of-00112 + model-00112-of-00112 (the two shards holding layer 80). The 593 layer-80 tensors are remapped to the rapid-mlx MTP param tree:

• enorm / hnorm — RMSNorms on the next-token embedding and previous hidden state (DeepSeek-V3 convention).
• eh_proj — the 2H -> H fused projection, applied as eh_proj(concat([enorm(embed_next), hnorm(prev_hidden)], -1)) (embedding first, confirmed against vLLM deepseek_mtp.py and SGLang hunyuan nextn).
• one HY3 DecoderLayer on the MoE branch (QK-norm attention + sigmoid-router SwitchGLU MoE over 192 experts + a shared expert).
• norm — the head's final RMSNorm (upstream final_layernorm).

Quantization

Matches the base checkpoint: 4-bit group_size=64 affine for every Linear (eh_proj, attention projections, switch_mlp.*, shared_mlp.*); 8-bit group_size=64 for mlp.router.gate; all RMSNorms and router.expert_bias kept in full precision.

Usage

rapid-mlx serve hy3-preview-4bit --speculative-config '{"method":"mtp"}'

rapid-mlx auto-resolves and downloads this sidecar. The base 4-bit backbone loads normally; the MTP head is grafted at boot for self-speculative decoding.

Measured

• Draft accept rate ~58% (K=1) across code / chinese / reasoning / list prompts.
• Greedy output is batched-consistent lossless vs the MTP-off reference.

The projection tree is byte-identical to a quantized backbone MoE layer, so the sidecar param names line up 1:1 with the runtime module.