| --- |
| license: apache-2.0 |
| license_link: https://huggingface.co/tencent/Hy3/blob/main/LICENSE |
| base_model: tencent/Hy3 |
| base_model_relation: quantized |
| library_name: mlx |
| pipeline_tag: text-generation |
| tags: |
| - mlx |
| - oq |
| - quantized |
| - moe |
| - hunyuan |
| --- |
| # Hy3 (MLX, oQ2) |
|
|
| Calibrated 2-bit MLX quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Hunyuan 3.0, 295B-A21B MoE), produced with **omlx** oQ at level 2 — **2.7 bits/weight effective**, 92 GB on disk. Data-driven mixed precision: bits are allocated per-tensor from a measured sensitivity map, not a fixed rule. For Apple Silicon. |
|
|
| ## Requirements |
|
|
| mlx-lm doesn't support the `hy_v3` architecture yet — there's an open PR: [mlx-lm#1211](https://github.com/ml-explore/mlx-lm/pull/1211). Until it lands, install mlx-lm from the PR branch, otherwise the model won't load: |
|
|
| ```bash |
| uv pip install "mlx-lm @ git+https://github.com/kernelpool/mlx-lm.git@add-hy3-preview" |
| ``` |
|
|
| Once #1211 lands in a released mlx-lm, this should load as-is. If something breaks after that, open a discussion here asking for a re-upload. |
|
|
| ## How it was quantized |
|
|
| Hy3 is 556 GB in BF16 — larger than RAM on a 128 GB machine — so oQ can't hold the full model to measure layer sensitivity, and omlx's automatic 4-bit sensitivity proxy trips the Metal command-buffer watchdog (`GPU Timeout`) on this hardware: the 4-bit proxy (150 GB) overflows the default Metal working-set cap. |
|
|
| The path that worked, in two steps: |
|
|
| 1. **Intermediate `mixed_2_6`** — I first made a uniform heuristic mixed quant via `mlx_lm.convert` (2-bit base, 6-bit on a fixed structural set of layers). It's coherent but heavy (3.15 bpw, 108 GB, peaks 116 GB — needs a raised Metal cap) and uneven on harder prompts. |
| 2. **oQ2 calibrated** — then I quantized to oQ2, passing the `mixed_2_6` as `sensitivity_model_path`, so oQ skips the failing auto-proxy, measures sensitivity on it (fits in memory, runs as inference — no watchdog), then streams the final quant tensor-by-tensor. It's **more resilient than `mixed_2_6` at less memory**. |
|
|
| ## Memory |
|
|
| Peak **99 GB**, which fits under the **default** Metal working-set cap (107.5 GB on a 128 GB machine) — no `sysctl iogpu.wired_limit_mb` bump needed, unlike `mixed_2_6` (116 GB peak). Headroom for the KV cache is tight at long context, so enabling **TurboQuant KV is highly recommended** if possible — I got good performance with it at 3-bit. |
|
|
| ## Conversion check |
|
|
| Smoke-tested after conversion (`mlx_lm.generate`): coherent — solved `17 * 24 = 408` with correct step-by-step reasoning, no repetition loop (plain 2-bit and 2-bit+8-bit-router uniform quants both collapsed here; the calibrated oQ2 does not). On a **Macbook Pro M5 Max 128GB 40 GPU**: 33.6 tok/s generation, 5.05 tok/s prompt, peak 99 GB (short prompt). |
|
|
| ## Usage |
|
|
| ```bash |
| python -m mlx_lm generate --model mlx-community/Hy3-oQ2 --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300 |
| ``` |
|
|
| ```python |
| from mlx_lm import load, generate |
| model, tokenizer = load("mlx-community/Hy3-oQ2") |
| ``` |
|
|
| ## License |
|
|
| [Apache-2.0](https://huggingface.co/tencent/Hy3/blob/main/LICENSE), inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use. |
|
|
