---
license: apache-2.0
base_model: tencent/Hy3
base_model_relation: quantized
tags:
- mlx
- hunyuan
- hy3
- moe
pipeline_tag: text-generation
library_name: mlx
---

# Hy3-4bit (MLX)

A clean 4-bit MLX quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Hunyuan 3.0, Apache-2.0), converted with `mlx-lm` and verified running on Apple Silicon.

Hy3 is a 295B-parameter Mixture-of-Experts model with **21B active parameters**, 80 layers, 192 routed experts (top-8) plus one shared expert, a native MTP (multi-token-prediction) head, and 256K context.

## Why this quant is clean

Naive uniform 4-bit quantization degrades MoE models badly, because the router that picks which experts fire is precision-sensitive. This build follows a mixed-precision recipe:

- **4-bit** (group size 64, affine) for attention and expert weights
- **8-bit** for every MoE router gate (`*.mlp.router.gate`)
- MTP head preserved

That router protection is the difference between a coherent model and mush, and it matches the recipe used by the reference `mlx-community/Hy3-preview-4bit`.

## Footprint

- Weights: ~166 GB
- Fits: two 128GB Apple Silicon machines (or one 192GB+ Mac), does not fit a single 128GB machine at 4-bit.

## How it was made

```python
from mlx_lm.convert import convert

def hy3_predicate(path, module, config=None):
    if path.endswith("mlp.router.gate"):
        return {"group_size": 64, "bits": 8}
    return {"group_size": 64, "bits": 4}

convert(
    hf_path="tencent/Hy3",
    mlx_path="Hy3-4bit",
    quantize=True, q_bits=4, q_group_size=64, q_mode="affine",
    quant_predicate=hy3_predicate,
)
```

`hy_v3` architecture support comes from [mlx-lm PR #1211](https://github.com/ml-explore/mlx-lm/pull/1211).

## Benchmarks (2x M5 Max, Thunderbolt RDMA)

Measured on two M5 Max (128GB each), pipeline-parallel over Thunderbolt with Apple's `jaccl` RDMA backend (`MLX_METAL_FAST_SYNCH=1`). Single-stream decode.

| Metric | Value |
|---|---|
| Decode (generation) | **36.91 tok/s** |
| Prompt (prefill) | 8.6 tok/s |
| Peak memory / node | 84.4 GB |
| Backend | jaccl (RDMA), pipeline-parallel |

Notes: at 4-bit (~166GB) the model does not fit a single 128GB machine, so this is a genuine 2-node cluster run. The `ring`/TCP backend trips the macOS Metal command-buffer watchdog on the cross-node fence wait; RDMA (`jaccl`) is required for stable single-stream decode.

## Credits

- Base model: Tencent Hunyuan (`tencent/Hy3`, Apache-2.0)
- Architecture support: mlx-lm PR #1211
- Quantization + Apple Silicon cluster benchmark: bicVanYonk