File size: 3,018 Bytes
e596273 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
license: apache-2.0
library_name: mlx
inference: false
pipeline_tag: text-generation
base_model: tencent/Hy3
tags:
- mlx
- optiq
- quantized
- 2bit
- mixed-precision
- moe
- ssd-streaming
- apple-silicon
- text-generation
---
# mlx-community/Hy3-OptiQ-2bit
> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)
**A 295-billion-parameter model that runs in 7.6 GB of RAM on a Mac.** This is a 2-bit mixed-precision MLX quant of [tencent/Hy3](https://huggingface.co/tencent/Hy3), produced by [mlx-optiq](https://mlx-optiq.com/). It is 88 GB on disk, and while the model generates only about 7.6 GB sits in RAM: attention, the router, the shared expert and the embeddings stay resident, and the 192 routed experts are read off the SSD as the router picks them.
## What it is
| Property | Value |
|---|---|
| Base | [tencent/Hy3](https://huggingface.co/tencent/Hy3) (sparse MoE, 192 experts, 8 active per token, 80 layers) |
| Parameters | 295 B |
| Bit-widths | 2-bit routed experts; 6-bit attention; 8-bit shared expert, embeddings and LM head |
| Achieved bits-per-weight | 2.39 |
| On disk | 88 GB |
| Resident while running | ~7.6 GB (routed experts streamed) |
No Capability Score is published for this quant. Running the six-benchmark suite against a model that decodes off SSD would take days, and at 2 bits on the routed experts the point of the artifact is different: that a 295 B MoE runs at all on consumer Apple Silicon.
## Run it
Hy3 is not an architecture stock mlx-lm knows, so `import optiq` once to register it:
```bash
pip install "mlx-optiq>=0.4.15"
```
The routed experts are far too large to sit resident, so serve it with SSD expert streaming. `optiq serve` turns this on by itself for a MoE quant that would not fit in RAM (`--stream-experts` forces it):
```bash
optiq serve --model mlx-community/Hy3-OptiQ-2bit
```
That gives you an OpenAI and Anthropic compatible endpoint with mixed-precision KV cache, tool-call healing and prompt caching. Only the routed experts stream per token, so the footprint stays near 7.6 GB regardless of how large the model on disk is.
A fast SSD matters more than RAM here. Every token reads 8 experts per layer from disk, so decode speed tracks read throughput.
## Notes
This is an **extreme quant.** Two bits on the routed experts is lossy, and anything where accuracy matters should use the bf16 weights or a higher-bit quant. What this one demonstrates is a model of this size running on a Mac, at a resident footprint that fits a 16 GB machine.
## Links
- **Project website:** [mlx-optiq.com](https://mlx-optiq.com/)
- **All OptiQ quants:** [mlx-optiq.com/models](https://mlx-optiq.com/models)
- **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/)
- **Base model:** [tencent/Hy3](https://huggingface.co/tencent/Hy3)
|