| --- |
| license: apache-2.0 |
| library_name: mlx |
| inference: false |
| pipeline_tag: text-generation |
| base_model: tencent/Hy3 |
| tags: |
| - mlx |
| - optiq |
| - quantized |
| - 2bit |
| - mixed-precision |
| - moe |
| - ssd-streaming |
| - apple-silicon |
| - text-generation |
| --- |
| |
| # mlx-community/Hy3-OptiQ-2bit |
|
|
| > **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/) |
|
|
| **A 295-billion-parameter model that runs in 7.6 GB of RAM on a Mac.** This is a 2-bit mixed-precision MLX quant of [tencent/Hy3](https://huggingface.co/tencent/Hy3), produced by [mlx-optiq](https://mlx-optiq.com/). It is 88 GB on disk, and while the model generates only about 7.6 GB sits in RAM: attention, the router, the shared expert and the embeddings stay resident, and the 192 routed experts are read off the SSD as the router picks them. |
|
|
| ## What it is |
|
|
| | Property | Value | |
| |---|---| |
| | Base | [tencent/Hy3](https://huggingface.co/tencent/Hy3) (sparse MoE, 192 experts, 8 active per token, 80 layers) | |
| | Parameters | 295 B | |
| | Bit-widths | 2-bit routed experts; 6-bit attention; 8-bit shared expert, embeddings and LM head | |
| | Achieved bits-per-weight | 2.39 | |
| | On disk | 88 GB | |
| | Resident while running | ~7.6 GB (routed experts streamed) | |
|
|
| No Capability Score is published for this quant. Running the six-benchmark suite against a model that decodes off SSD would take days, and at 2 bits on the routed experts the point of the artifact is different: that a 295 B MoE runs at all on consumer Apple Silicon. |
|
|
| ## Run it |
|
|
| Hy3 is not an architecture stock mlx-lm knows, so `import optiq` once to register it: |
|
|
| ```bash |
| pip install "mlx-optiq>=0.4.15" |
| ``` |
|
|
| The routed experts are far too large to sit resident, so serve it with SSD expert streaming. `optiq serve` turns this on by itself for a MoE quant that would not fit in RAM (`--stream-experts` forces it): |
|
|
| ```bash |
| optiq serve --model mlx-community/Hy3-OptiQ-2bit |
| ``` |
|
|
| That gives you an OpenAI and Anthropic compatible endpoint with mixed-precision KV cache, tool-call healing and prompt caching. Only the routed experts stream per token, so the footprint stays near 7.6 GB regardless of how large the model on disk is. |
|
|
| A fast SSD matters more than RAM here. Every token reads 8 experts per layer from disk, so decode speed tracks read throughput. |
|
|
| ## Notes |
|
|
| This is an **extreme quant.** Two bits on the routed experts is lossy, and anything where accuracy matters should use the bf16 weights or a higher-bit quant. What this one demonstrates is a model of this size running on a Mac, at a resident footprint that fits a 16 GB machine. |
|
|
| ## Links |
|
|
| - **Project website:** [mlx-optiq.com](https://mlx-optiq.com/) |
| - **All OptiQ quants:** [mlx-optiq.com/models](https://mlx-optiq.com/models) |
| - **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/) |
| - **Base model:** [tencent/Hy3](https://huggingface.co/tencent/Hy3) |
|
|
