---
license: apache-2.0
license_link: https://huggingface.co/tencent/Hy3/blob/main/LICENSE
base_model: tencent/Hy3
base_model_relation: quantized
library_name: mlx
pipeline_tag: text-generation
tags:
- mlx
- oq
- quantized
- moe
- hunyuan
---
# Hy3 (MLX, oQ2)

Calibrated 2-bit MLX quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Hunyuan 3.0, 295B-A21B MoE), produced with **omlx** oQ at level 2 — **2.7 bits/weight effective**, 92 GB on disk. Data-driven mixed precision: bits are allocated per-tensor from a measured sensitivity map, not a fixed rule. For Apple Silicon.

## Requirements

mlx-lm doesn't support the `hy_v3` architecture yet — there's an open PR: [mlx-lm#1211](https://github.com/ml-explore/mlx-lm/pull/1211). Until it lands, install mlx-lm from the PR branch, otherwise the model won't load:

```bash
uv pip install "mlx-lm @ git+https://github.com/kernelpool/mlx-lm.git@add-hy3-preview"
```

Once #1211 lands in a released mlx-lm, this should load as-is. If something breaks after that, open a discussion here asking for a re-upload.

## How it was quantized

Hy3 is 556 GB in BF16 — larger than RAM on a 128 GB machine — so oQ can't hold the full model to measure layer sensitivity, and omlx's automatic 4-bit sensitivity proxy trips the Metal command-buffer watchdog (`GPU Timeout`) on this hardware: the 4-bit proxy (150 GB) overflows the default Metal working-set cap.

The path that worked, in two steps:

1. **Intermediate `mixed_2_6`** — I first made a uniform heuristic mixed quant via `mlx_lm.convert` (2-bit base, 6-bit on a fixed structural set of layers). It's coherent but heavy (3.15 bpw, 108 GB, peaks 116 GB — needs a raised Metal cap) and uneven on harder prompts.
2. **oQ2 calibrated** — then I quantized to oQ2, passing the `mixed_2_6` as `sensitivity_model_path`, so oQ skips the failing auto-proxy, measures sensitivity on it (fits in memory, runs as inference — no watchdog), then streams the final quant tensor-by-tensor. It's **more resilient than `mixed_2_6` at less memory**.

## Memory

Peak **99 GB**, which fits under the **default** Metal working-set cap (107.5 GB on a 128 GB machine) — no `sysctl iogpu.wired_limit_mb` bump needed, unlike `mixed_2_6` (116 GB peak). Headroom for the KV cache is tight at long context, so enabling **TurboQuant KV is highly recommended** if possible — I got good performance with it at 3-bit.

## Conversion check

Smoke-tested after conversion (`mlx_lm.generate`): coherent — solved `17 * 24 = 408` with correct step-by-step reasoning, no repetition loop (plain 2-bit and 2-bit+8-bit-router uniform quants both collapsed here; the calibrated oQ2 does not). On a **Macbook Pro M5 Max 128GB 40 GPU**: 33.6 tok/s generation, 5.05 tok/s prompt, peak 99 GB (short prompt).

## Usage

```bash
python -m mlx_lm generate --model mlx-community/Hy3-oQ2 --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300
```

```python
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Hy3-oQ2")
```

## License

[Apache-2.0](https://huggingface.co/tencent/Hy3/blob/main/LICENSE), inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use.