mlx-community/Hy3-oQ2

Hy3 (MLX, oQ2)

Calibrated 2-bit MLX quantization of tencent/Hy3 (Hunyuan 3.0, 295B-A21B MoE), produced with omlx oQ at level 2 — 2.7 bits/weight effective, 92 GB on disk. Data-driven mixed precision: bits are allocated per-tensor from a measured sensitivity map, not a fixed rule. For Apple Silicon.

Requirements

mlx-lm doesn't support the hy_v3 architecture yet — there's an open PR: mlx-lm#1211. Until it lands, install mlx-lm from the PR branch, otherwise the model won't load:

uv pip install "mlx-lm @ git+https://github.com/kernelpool/mlx-lm.git@add-hy3-preview"

Once #1211 lands in a released mlx-lm, this should load as-is. If something breaks after that, open a discussion here asking for a re-upload.

How it was quantized

Hy3 is 556 GB in BF16 — larger than RAM on a 128 GB machine — so oQ can't hold the full model to measure layer sensitivity, and omlx's automatic 4-bit sensitivity proxy trips the Metal command-buffer watchdog (GPU Timeout) on this hardware: the 4-bit proxy (150 GB) overflows the default Metal working-set cap.

The path that worked, in two steps:

1. Intermediate mixed_2_6 — I first made a uniform heuristic mixed quant via mlx_lm.convert (2-bit base, 6-bit on a fixed structural set of layers). It's coherent but heavy (3.15 bpw, 108 GB, peaks 116 GB — needs a raised Metal cap) and uneven on harder prompts.
2. oQ2 calibrated — then I quantized to oQ2, passing the mixed_2_6 as sensitivity_model_path, so oQ skips the failing auto-proxy, measures sensitivity on it (fits in memory, runs as inference — no watchdog), then streams the final quant tensor-by-tensor. It's more resilient than mixed_2_6 at less memory.

Memory

Peak 99 GB, which fits under the default Metal working-set cap (107.5 GB on a 128 GB machine) — no sysctl iogpu.wired_limit_mb bump needed, unlike mixed_2_6 (116 GB peak). Headroom for the KV cache is tight at long context, so enabling TurboQuant KV is highly recommended if possible — I got good performance with it at 3-bit.

Conversion check

Smoke-tested after conversion (mlx_lm.generate): coherent — solved 17 * 24 = 408 with correct step-by-step reasoning, no repetition loop (plain 2-bit and 2-bit+8-bit-router uniform quants both collapsed here; the calibrated oQ2 does not). On a Macbook Pro M5 Max 128GB 40 GPU: 33.6 tok/s generation, 5.05 tok/s prompt, peak 99 GB (short prompt).

Usage

python -m mlx_lm generate --model mlx-community/Hy3-oQ2 --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300

from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Hy3-oQ2")

License

Apache-2.0, inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use.