README.md

---
license: apache-2.0
base_model: tencent/Hy3
base_model_relation: quantized
library_name: mlx
pipeline_tag: text-generation
tags:
- mlx
- hy_v3
- mixture-of-experts
- apple-silicon
---

# Hy3-Alis-MLX-Dynamic

Mixed-precision MLX quantizations of [tencent/Hy3](https://huggingface.co/tencent/Hy3)
(295B total / 21B active MoE) for Apple Silicon, sized for **128 / 256 / 512 GiB**
unified-memory Macs. One repo, one card — each quantization lives on its own branch.

| branch | eff. bpw | size | target Mac | decode* | KL vs 8-bit (top-1 flip) | status |
|---|---|---|---|---|---|---|
| **`main`** | 4.568 | 168.5 GB | 256 GiB | 28.7 tok/s | 0.094 (8.9%) | ✅ daily driver |
| `T512` | 6.561 | 242.0 GB | 512 GiB | 24.0 tok/s | **0.020 (4.6%)** | ✅ flagship, near-lossless |
| `T512REF` | 8.502 | 313.5 GB | 512 GiB | 22.2 tok/s | (reference) | ✅ max quality |
| `T128` | 2.375 | 87.6 GB | 128 GiB | 27 tok/s | **0.550 (19.3%)** | ⚠️ experimental, sensitivity-placed + ZH-mixed DWQ (v2) |
| `mtp-bf16` | bf16 | 7.5 GB | — | — | — | MTP layer sidecar |

\* single-stream decode on an M3 Ultra, short prompt; longer contexts decode slower.

### Final state, all tiers

Everything below is measured, not projected — KL vs the 8-bit reference on a fixed
EN/code/ZH slice, PPL on fixed corpora, and needle retrieval actually run at the stated
context. All builds keep the router in bf16 and `expert_bias` in fp32.

- **`T128` — 87.6 GB, 128 GiB Macs, experimental.** KL 0.550, PPL 5.80 / 2.44 / 5.40.
  The only build that fits a 128 GiB Mac's default wired limit; needle verified to
  16K fp16 (32K with int8 KV, needs a wired-limit bump). Received both low-bit levers —
  sensitivity-placed 3-bit `down_proj` + Chinese-mixed DWQ (v2) — halving its KL vs v1.
- **`main` (T256) — 168.5 GB, 256 GiB Macs, daily driver.** KL 0.094, PPL 3.97 / 1.98 /
  3.73. Needle verified to 64K fp16 (peak 190 GB, inside the default wired limit);
  128K via int8 KV. Uniform 4-bit — already near-optimal for its size (see above).
- **`T512` — 242 GB, 512 GiB Macs, flagship.** KL 0.020 (near-lossless), PPL 3.85 / 1.93
  / 3.54, and ~25% faster to decode than the 8-bit. Needle verified to 128K fp16; full
  256K fp16 KV fits the default wired limit.
- **`T512REF` — 313.5 GB, 512 GiB Macs, reference.** Uniform 8-bit; the KL ground truth
  and DWQ teacher. Ship only if you want the maximum-quality artifact.
- **`mtp-bf16` — 7.5 GB sidecar + harness.** Hy3's MTP head plus a self-speculative
  decode loop; measured 1.13× lossless (k=1) on top of any tier.

![Which build fits your Mac — weights and measured peak memory vs each machine tier's default wired limit](https://huggingface.co/avlp12/Hy3-Alis-MLX-Dynamic/resolve/main/images/fig1_tiers.png)

## Which build?

- **512 GiB Mac** → `T512`. Faster than the 8-bit *and* statistically near-lossless
  (KL 0.020 nats). 128K-token needle retrieval verified at 285 GB peak; 256K fp16
  KV fits in the default wired limit (242 + 80 = 322 GB < ~384 GB).
- **256 GiB Mac** → `main`. 64K-token needle verified at 190 GB peak — just inside the
  default ~192 GB wired limit. For 128K, use int8 KV (`--kv-bits 8`, generate path)
  or raise `iogpu.wired_limit_mb`.
- **128 GiB Mac** → `T128`, with eyes open: see the experimental warning below.
  16K fp16 verified at 93.8 GB peak; 32K with int8 KV peaked at 109 GB, which
  **requires** raising the wired limit on a 128 GB machine.
- **KL ground truth / archival** → `T512REF` (uniform 8-bit).

## Usage

Hy3 (`hy_v3`) support is not yet in an mlx-lm release
([ml-explore/mlx-lm#1211](https://github.com/ml-explore/mlx-lm/pull/1211) is open).
Until it merges, install from the pinned branch (PR #1211 plus an fp32-router
correctness patch, commit `14f7837`):

```bash
pip install git+https://github.com/avlp12/mlx-lm@hy3-support
```

```python
from mlx_lm import load, generate

# main = the 256 GiB build; pass revision= for other tiers
model, tokenizer = load("avlp12/Hy3-Alis-MLX-Dynamic")
model, tokenizer = load("avlp12/Hy3-Alis-MLX-Dynamic", revision="T512")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain MoE routing in two sentences."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))
```

CLI / server:

```bash
mlx_lm.generate --model avlp12/Hy3-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Hy3-Alis-MLX-Dynamic --revision T512 --local-dir Hy3-T512
mlx_lm.server --model ./Hy3-T512 --port 8080
```

Long context on the smaller tiers: quantize the KV cache in the generate path
(`--kv-bits 8 --kv-group-size 64`). Note `mlx_lm.server` currently serves fp16 KV only.

## Recipes (verified from each build's `config.json`, not from intent)

97.0% of Hy3's parameters are the 192 routed experts, so expert bit-width sets the
size; everything else is cheap to keep at high precision. Common to all tiers:

- `mlp.router.gate` — **never quantized** (bf16). The sigmoid+bias top-8 routing is a
  discrete control path; 62M params total, so full precision is free.
- `expert_bias` (e-score correction) kept **fp32**, verified in the output shards.
- Attention bits = min(expert bits + 2, 8).

![Per-tier bit allocation by component](https://huggingface.co/avlp12/Hy3-Alis-MLX-Dynamic/resolve/main/images/fig2_recipe.png)

| tier | routed experts | expert `down_proj` | attn / shared / dense | embed / lm_head |
|---|---|---|---|---|
| T128 | 2-bit g128 | 3-bit g128 on 12 sensitivity-ranked layers | 4-bit g64 | 6-bit g64 |
| main (T256) | 4-bit g64 | 4-bit g64 | 6-bit g64 | 8-bit g64 |
| T512 | 6-bit g64 | 6-bit g64 | 8-bit g64 | 8-bit g64 |
| T512REF | 8-bit g64 | 8-bit g64 | 8-bit g64 | 8-bit g64 |

**T128 (v2) — two byte-neutral quality levers.** (1) The 12 down_proj layers upgraded
to 3-bit are chosen by *measured* sensitivity (each MoE layer's KL when dropped to
2-bit, probed on the 8-bit reference), not an early/late heuristic — the most
sensitive layers turned out to be the middle band (46-65). (2) A **DWQ pass** (scales/
biases distilled against the 8-bit build's logits) on a **45% Chinese** data mix, so
the language-specialized experts actually receive gradient. Same 87.6 GB, same recipe
bits. Net vs the raw quantization: overall KL −42%, ZH −46% (1.55→0.83). The prior v1
(early/late layers + English-only DWQ) reached only ZH 1.43; it is tagged `T128-v1`.

## Quality

**KL divergence & top-1 flip rate vs `T512REF`** — measured on a fixed 3,072-token
slice (English / code / Chinese thirds). Disclosure: the reference is the 8-bit
build, *not* bf16 — the bf16 model (598 GB) does not fit in 512 GB of RAM.

![KL divergence vs the 8-bit reference by language slice, log scale](https://huggingface.co/avlp12/Hy3-Alis-MLX-Dynamic/resolve/main/images/fig3_quality.png)

| tier | overall KL | EN | code | ZH | overall flip |
|---|---|---|---|---|---|
| T512 | 0.020 | 0.017 | 0.011 | 0.033 | 4.6% |
| main | 0.094 | 0.058 | 0.054 | 0.169 | 8.9% |
| T128 (v2) | 0.550 | 0.463 | 0.355 | 0.832 | 19.3% |

**Perplexity** (strided, ctx 2048 / stride 1024, fixed corpora):

| tier | wikitext | code | Chinese |
|---|---|---|---|
| T512 | 3.85 | 1.93 | 3.54 |
| main | 3.97 | 1.98 | 3.73 |
| T128 (v2) | 5.80 | 2.44 | 5.40 |

**lm-eval** (hellaswag / piqa / winogrande, 0-shot, limit 500, same harness across
tiers — absolute numbers are base-style loglikelihood, useful only for tier-vs-tier
comparison): all three tiers are statistically indistinguishable (±2σ), e.g.
piqa acc 0.75–0.77 across every tier. Loglikelihood tasks are insensitive at this
scale; the KL table above is the discriminating metric.

**Long-context needle retrieval** (needle at 25% depth, greedy): every listed claim
was tested, not extrapolated —

| tier | context tested | result | peak memory |
|---|---|---|---|
| T128 (v2) | 16K fp16 · 32K int8-KV | PASS · PASS | 93.8 · 109.2 GB |
| main | 16K · 64K fp16 | PASS · PASS | 174.6 · 190.3 GB |
| T512 | 64K · 128K fp16 | PASS · PASS | 263.8 · 285.2 GB |

### ⚠️ T128 is experimental

2.375 bpw on a 295B MoE with 18.9M-parameter experts is aggressive. The v2 build
generates fluent English/Korean/Chinese in under 88 GB and passes 16K needle
retrieval (peak 93.8 GB). Its remaining distribution gap vs the 8-bit reference is
modest overall (KL 0.55) and no longer collapses on Chinese (ZH KL 0.83 vs v1's 1.43,
ZH PPL 5.40 vs v1's 9.01). Still the most aggressive tier — verify on your workload;
for headroom, the 256 GiB `main` build is far closer to lossless.

### Why only T128 got the extra tuning

The two levers that transformed T128 — placing the higher-bit `down_proj` layers by
*measured* per-layer sensitivity, and a Chinese-mixed DWQ pass — are **2-bit-specific**,
and we verified this rather than assumed it:

- **Sensitivity shifts with bit-width.** Probing each MoE layer's KL when dropped to a
  given bit-width, the most fragile layers at 2-bit are the middle band (46–65) but at
  4-bit they are the early layers (1–14) — so T128's layer set doesn't transfer. And a
  single layer's 4-bit damage is only ~39% of its 2-bit damage: far less to reclaim.
- **The byte-neutral trick backfires at 4-bit.** A mixed `down_proj` recipe (5-bit on
  sensitive layers, 3-bit on robust ones, same total size) *improved* T128 but measured
  **worse** on `main` (paired ΔKL −0.013): the 4-bit error surface is flat enough that
  demoting robust layers costs more than promoting sensitive ones saves.
- **DWQ has little to correct at 4-bit.** Starting a DWQ pass on `main`, the initial
  validation loss was 0.05 vs T128's ~0.5 — `main` already sits ~10× closer to the
  8-bit reference, and its Chinese KL (0.169) never collapsed the way T128's did (1.55).

So `main`, `T512`, and `T512REF` ship as uniform quantizations: at 4-bit and above the
naive recipe is already near-optimal for its size, and every intervention we measured
was neutral-to-negative. The aggressive tuning is reserved for the tier that needs it.

## How it compares to other Hy3 MLX ports

Every public Hy3(-preview) MLX quantization we could find, measured on the **same
harness, same machine** (M3 Ultra 512GB, wired-limit set, quiet disk): identical
3,072-token KL slice vs our 8-bit reference, identical PPL corpora, same prompt for
decode speed. Sorted by size:

| model | size | KL vs 8-bit† (flip) | PPL wiki / code / ZH | decode | fits default wired limit of |
|---|---|---|---|---|---|
| **ours `T128` (v2)** | **87.6 GB** | 0.550 (19.3%) | 5.80 / 2.44 / 5.40 | 27 tok/s | **128 GB Mac** (93.8 GB peak @16K) |
| [ox-ox w2q3exp](https://huggingface.co/ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx) | 112.6 GB | 0.686 (24.1%) | 5.85 / 2.45 / 7.36 | 8.9 tok/s | 256 GB Mac (112.3 GB peak @0 ctx — over a 128 GB Mac's ~96 GB) |
| [unigilby MXFP4-imatrix](https://huggingface.co/unigilby/Hy3-MLX-MXFP4-imatrix) | 161.2 GB | 0.172 (11.3%) | 4.09 / 2.00 / 3.93 | 28.1 tok/s | 256 GB Mac |
| [mlx-community 4bit](https://huggingface.co/mlx-community/Hy3-preview-4bit)‡ | 166.0 GB | 0.319 (18.0%)‡ | 3.90 / 2.10 / 3.75 | 32.5 tok/s | 256 GB Mac |
| **ours `main` (T256)** | **168.5 GB** | **0.094 (8.9%)** | 3.97 / 1.98 / 3.73 | 28.7 tok/s | 256 GB Mac (190.3 GB peak @64K) |
| **ours `T512`** | **242.0 GB** | **0.020 (4.6%)** | 3.85 / 1.93 / 3.54 | 24.0 tok/s | 512 GB Mac |
| **ours `T512REF`** | 313.5 GB | (reference) | — | 22.2 tok/s | 512 GB Mac |
| [inferencerlabs Q9](https://huggingface.co/inferencerlabs/Hy3-MLX-Q9) | 322.2 GB | — | — | — | does not load in stock mlx-lm (custom config: no `bits` in its quantization block) |

† KL(8-bit ref ‖ model) on the fixed EN/code/ZH slice — lower is better.
‡ mlx-community's build quantizes **Hy3-preview**, a different base checkpoint than
the final Hy3; its KL vs our final-Hy3 reference includes base-model drift, so compare
it on PPL. Its uniform 4-bit g64 recipe (8-bit routers) decodes faster than mixed
recipes because its attention path carries fewer bytes per token.

Reading the table:

- **4-bit class (~161–169 GB)**: at essentially the same size, `main` has the lowest
  KL of any final-Hy3 build (0.094 vs 0.172 for the imatrix MXFP4 build — ~45% lower)
  and the best code/ZH perplexity, at equal speed.
- **low-bit class**: `T128` (v2) at 87.6 GB now has the **lowest KL of any sub-160 GB
  Hy3 build measured** (0.550 vs ox-ox's 0.686 at 112.6 GB and unigilby's 0.172 at
  161 GB), and it is the only build that fits a 128 GB Mac's default wired limit.
  ox-ox spends 25 GB more (8-bit attention) yet scores worse and exceeds the 128 GB
  ceiling before the first token, decoding at 8.9 tok/s vs `T128`'s 27. If your Mac has 256 GB, use `main` instead; if it
  has 128 GB, `T128` is the build that actually fits.
- Routers: ours keep the sigmoid+bias router in **bf16** (ox-ox and mlx-community
  quantize it to 8-bit; unigilby's config applies its 4-bit default to the router).
  With 192 experts and near-tie top-8 margins, full-precision routing is the cheapest
  correctness insurance in the recipe (~124 MB total).

## `mtp-bf16` branch — MTP self-speculative decoding (measured: 1.13×, lossless)

Hy3 ships a Multi-Token-Prediction layer (`model.layers.80.*`, 3.75B params,
DeepSeek-V3-style `eh_proj`/`enorm`/`hnorm` + MoE block) that mlx-lm strips at
conversion. The `mtp-bf16` branch preserves it verbatim in bf16 (7.5 GB) **plus a
runnable self-speculative-decode harness** (`mtp_run.py` + module + step generators).

Measured on the 8-bit build (M3 Ultra, greedy, 200 tokens, output verified
token-identical to plain decode):

| config | tok/s | speedup | accept-len |
|---|---|---|---|
| plain decode | 21.4 | 1.00× | — |
| **MTP k=1 (bf16 draft)** | **24.1** | **1.13×** | 1.65 |
| MTP k=2 | 22.9 | 1.07× | 2.02 |
| MTP k=3 | 19.7 | 0.92× | 2.13 |

Notes: k=1 with the bf16 sidecar is the operating point. A 4-bit draft was
measured too (isolated draft call 2.26 → 1.74 ms, −23%): the saving is real but
amounts to ~1% of loop time (the 47 ms verify forward dominates), while
acceptance drops slightly and consistently across EN/KO/code prompts
(accept-len −0.02…−0.06) — net ≈ −1…−4% vs bf16, tie on code. Use the 4-bit
draft only if you want its 5.4 GB memory saving. The harness uses the MTP's own
KV cache and final-layernorm ("normed hidden") chaining for k>1.

```bash
hf download avlp12/Hy3-Alis-MLX-Dynamic --revision mtp-bf16 --local-dir mtp
python mtp/mtp_run.py --model <any-tier> --mtp mtp/Hy3-MTP-bf16.safetensors --k 1
```

## Correctness

- Cross-framework logit parity vs transformers 5.12.1 reference implementation on
  tiny-random weights: top-1 agreement 100%, max |Δlogit| ≤ 5e-3 (fp32).
- One correctness patch on top of PR #1211: the router matmul runs in **fp32**,
  matching the reference `F.linear(hidden.float(), weight.float())` — near-tie
  top-8 selection under a sigmoid+bias router is sensitive to bf16 matmul noise.
- Recipe verification: quantization blocks parsed from each output `config.json`;
  router dtype and `expert_bias` fp32 checked in the shards themselves.

## Credits

- **Tencent Hunyuan** for [Hy3](https://huggingface.co/tencent/Hy3) (Apache 2.0) and
  the [AngelSlim](https://github.com/tencent/AngelSlim) compression toolkit — the
  official quantization path for CUDA deployments (FP8/INT4 for vLLM/SGLang).
  These MLX builds are an independent, complementary Apple-Silicon path.
- **kernelpool** for the `hy_v3` MLX port
  ([ml-explore/mlx-lm#1211](https://github.com/ml-explore/mlx-lm/pull/1211)).
- Built with [MLX](https://github.com/ml-explore/mlx) and
  [mlx-lm](https://github.com/ml-explore/mlx-lm).

> **Citation** — if you use these builds, please cite Tencent's Hy3 release
> (tencent/Hy3, 2026) and Apple's MLX framework.