---
license: apache-2.0
library_name: mlx
inference: false
pipeline_tag: text-generation
base_model: tencent/Hy3
tags:
- mlx
- optiq
- quantized
- 2bit
- mixed-precision
- moe
- ssd-streaming
- apple-silicon
- text-generation
---

# mlx-community/Hy3-OptiQ-2bit

> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)

**A 295-billion-parameter model that runs in 7.6 GB of RAM on a Mac.** This is a 2-bit mixed-precision MLX quant of [tencent/Hy3](https://huggingface.co/tencent/Hy3), produced by [mlx-optiq](https://mlx-optiq.com/). It is 88 GB on disk, and while the model generates only about 7.6 GB sits in RAM: attention, the router, the shared expert and the embeddings stay resident, and the 192 routed experts are read off the SSD as the router picks them.

## What it is

| Property | Value |
|---|---|
| Base | [tencent/Hy3](https://huggingface.co/tencent/Hy3) (sparse MoE, 192 experts, 8 active per token, 80 layers) |
| Parameters | 295 B |
| Bit-widths | 2-bit routed experts; 6-bit attention; 8-bit shared expert, embeddings and LM head |
| Achieved bits-per-weight | 2.39 |
| On disk | 88 GB |
| Resident while running | ~7.6 GB (routed experts streamed) |

No Capability Score is published for this quant. Running the six-benchmark suite against a model that decodes off SSD would take days, and at 2 bits on the routed experts the point of the artifact is different: that a 295 B MoE runs at all on consumer Apple Silicon.

## Run it

Hy3 is not an architecture stock mlx-lm knows, so `import optiq` once to register it:

```bash
pip install "mlx-optiq>=0.4.15"
```

The routed experts are far too large to sit resident, so serve it with SSD expert streaming. `optiq serve` turns this on by itself for a MoE quant that would not fit in RAM (`--stream-experts` forces it):

```bash
optiq serve --model mlx-community/Hy3-OptiQ-2bit
```

That gives you an OpenAI and Anthropic compatible endpoint with mixed-precision KV cache, tool-call healing and prompt caching. Only the routed experts stream per token, so the footprint stays near 7.6 GB regardless of how large the model on disk is.

A fast SSD matters more than RAM here. Every token reads 8 experts per layer from disk, so decode speed tracks read throughput.

## Notes

This is an **extreme quant.** Two bits on the routed experts is lossy, and anything where accuracy matters should use the bf16 weights or a higher-bit quant. What this one demonstrates is a model of this size running on a Mac, at a resident footprint that fits a 16 GB machine.

## Links

- **Project website:** [mlx-optiq.com](https://mlx-optiq.com/)
- **All OptiQ quants:** [mlx-optiq.com/models](https://mlx-optiq.com/models)
- **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/)
- **Base model:** [tencent/Hy3](https://huggingface.co/tencent/Hy3)