| --- |
| base_model: mistralai/Mistral-Small-4-119B-2603 |
| library_name: mlx |
| license: apache-2.0 |
| tags: |
| - rotorquant |
| - kv-cache-quantization |
| - mistral |
| - moe |
| - sparse-moe |
| - multimodal |
| - quantized |
| - mlx |
| - 8bit |
| - apple-silicon |
| - 256k-context |
| - thinking |
| pipeline_tag: text-generation |
| --- |
| |
| # Mistral-Small-4-119B-RotorQuant-MLX-8bit |
|
|
| **Dual compression: 8-bit MLX weight quantization + RotorQuant KV cache quantization** for Mistral Small 4 on Apple Silicon. |
|
|
| This repository provides an 8-bit weight-quantized MLX conversion of [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) with RotorQuant KV cache quantization support. Designed for efficient inference on Apple Silicon Macs with excellent throughput. |
|
|
| Approximate model size: **~120 GB** |
|
|
| ## Overview |
|
|
| This model applies two complementary compression techniques: |
|
|
| 1. **8-bit weight quantization (MLX)** -- reduces model weights from ~238 GB to ~120 GB |
| 2. **RotorQuant KV cache quantization** -- reduces KV cache from ~32 GB to ~6.5 GB at 256K context |
|
|
| Together, these make it feasible to run a 119B-parameter MoE model on high-memory Apple Silicon machines with excellent throughput. |
|
|
| ## Model Specs |
|
|
| | Property | Value | |
| |---|---| |
| | Base Model | Mistral Small 4 (March 2026) | |
| | Total Parameters | 119B | |
| | Active Parameters | 6.5B per token (Sparse MoE) | |
| | Architecture | Sparse MoE -- 128 experts, 4 active per token | |
| | Context Length | 256K tokens | |
| | Modality | Text + Images (multimodal) | |
| | Capabilities | Thinking / reasoning, tool use, multilingual | |
| | License | Apache 2.0 | |
| | Weight Quantization | 8-bit (MLX) | |
| | KV Cache Quantization | RotorQuant 3-bit | |
|
|
| ## Memory Estimates |
|
|
| | Configuration | Weights | KV Cache (256K) | Total | |
| |---|---|---|---| |
| | FP16 baseline | ~238 GB | ~32 GB | ~270 GB | |
| | **This model (8-bit MLX + RotorQuant)** | **~120 GB** | **~6.5 GB** | **~126.5 GB** | |
|
|
| > **Note:** This is a Sparse MoE model -- only 6.5B parameters are active per token, so inference is fast despite the 119B total parameter count. |
|
|
| ## Quickstart |
|
|
| ```python |
| from mlx_lm import load, generate |
| |
| model, tokenizer = load("majentik/Mistral-Small-4-119B-RotorQuant-MLX-8bit") |
| |
| prompt = "Explain sparse mixture-of-experts architectures." |
| messages = [{"role": "user", "content": prompt}] |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| |
| response = generate(model, tokenizer, prompt=text, max_tokens=512) |
| print(response) |
| ``` |
|
|
| ## What is RotorQuant? |
|
|
| [RotorQuant](https://github.com/scrya-com/rotorquant) is a rotation-based KV cache quantization method that applies learned Clifford algebra rotations before quantizing the key-value cache. Key results: |
|
|
| - **5.3x faster prefill** compared to TurboQuant baseline |
| - **28% faster decode** throughput |
| - **Perplexity: 6.91** vs 7.07 for TurboQuant (lower is better) |
|
|
| Because it targets the KV cache rather than weights, it stacks with weight quantization for compounding memory savings. |
|
|
| ## KV-Cache Quantization Comparison |
|
|
| | Method | Prefill Speed | Decode Speed | Memory Savings | Reference | |
| |---|---|---|---|---| |
| | **TurboQuant** | Baseline | Baseline | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) | |
| | **RotorQuant** | 5.3x faster | 28% faster | High | [GitHub](https://github.com/scrya-com/rotorquant) | |
|
|
| ## Hardware Requirements |
|
|
| This model requires approximately 127 GB total memory at 256K context. Recommended hardware: |
| - Apple M2 Ultra (192 GB+) |
| - Apple M3/M4 Ultra (192 GB+) |
| - Mac Pro |
|
|
| ## See Also |
|
|
| - [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) -- Base model |
| - [majentik/Mistral-Small-4-119B-RotorQuant](https://huggingface.co/majentik/Mistral-Small-4-119B-RotorQuant) -- KV cache only (no weight quantization) |
| - [majentik/Mistral-Small-4-119B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/Mistral-Small-4-119B-RotorQuant-MLX-4bit) -- 4-bit MLX variant |
| - [majentik/Mistral-Small-4-119B-RotorQuant-MLX-2bit](https://huggingface.co/majentik/Mistral-Small-4-119B-RotorQuant-MLX-2bit) -- 2-bit MLX variant |
| - [majentik/Mistral-Small-4-119B-TurboQuant-MLX-8bit](https://huggingface.co/majentik/Mistral-Small-4-119B-TurboQuant-MLX-8bit) -- TurboQuant MLX 8-bit variant |
| - [RotorQuant GitHub](https://github.com/scrya-com/rotorquant) |
| - [MLX Framework](https://github.com/ml-explore/mlx) |
|
|
