Add model card (weights pending mlx_lm mistral3 architecture support)

README.md

@@ -0,0 +1,91 @@
+---
+base_model: mistralai/Mistral-Small-4-119B-2603
+library_name: mlx
+license: apache-2.0
+tags:
+  - rotorquant
+  - kv-cache-quantization
+  - mistral
+  - moe
+  - sparse-moe
+  - multimodal
+  - quantized
+  - mlx
+  - 2-bit
+  - apple-silicon
+  - 256k-context
+  - thinking
+pipeline_tag: text-generation
+---
+
+# Mistral-Small-4-119B-RotorQuant-MLX-2bit
+
+**Dual compression: 2-bit MLX weight quantization + RotorQuant KV cache quantization** for Mistral Small 4 on Apple Silicon.
+
+This repository provides a 2-bit weight-quantized MLX conversion of [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) with RotorQuant KV cache quantization support. Aggressive compression for running on consumer Apple Silicon hardware.
+
+## Overview
+
+This model applies two complementary compression techniques:
+
+1. **2-bit weight quantization (MLX)** -- reduces model weights from ~238 GB to ~30 GB
+2. **RotorQuant KV cache quantization** -- reduces KV cache from ~32 GB to ~6.5 GB at 256K context
+
+This enables running a 119B-parameter MoE model on Apple Silicon Macs with 64 GB+ unified memory.
+
+## Model Specs
+
+| Property | Value |
+|---|---|
+| Base Model | Mistral Small 4 (March 2026) |
+| Total Parameters | 119B |
+| Active Parameters | 6.5B per token (Sparse MoE) |
+| Architecture | Sparse MoE -- 128 experts, 4 active per token |
+| Context Length | 256K tokens |
+| Modality | Text + Images (multimodal) |
+| Capabilities | Thinking / reasoning, tool use, multilingual |
+| License | Apache 2.0 |
+| Weight Quantization | 2-bit (MLX) |
+| KV Cache Quantization | RotorQuant 3-bit |
+
+## Memory Estimates
+
+| Configuration | Weights | KV Cache (256K) | Total |
+|---|---|---|---|
+| FP16 baseline | ~238 GB | ~32 GB | ~270 GB |
+| **This model (2-bit MLX + RotorQuant)** | **~30 GB** | **~6.5 GB** | **~36.5 GB** |
+
+> **Note:** This is a Sparse MoE model -- only 6.5B parameters are active per token, so inference is fast despite the 119B total parameter count. The 2-bit quantization trades some quality for significantly reduced memory. Expect modest degradation on complex reasoning tasks compared to 4-bit.
+
+## Quickstart
+
+```python
+from mlx_lm import load, generate
+
+model, tokenizer = load("majentik/Mistral-Small-4-119B-RotorQuant-MLX-2bit")
+
+prompt = "Explain sparse mixture-of-experts architectures."
+messages = [{"role": "user", "content": prompt}]
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+response = generate(model, tokenizer, prompt=text, max_tokens=512)
+print(response)
+```
+
+## What is RotorQuant?
+
+[RotorQuant](https://github.com/scrya-com/rotorquant) is a rotation-based KV cache quantization method that applies learned rotations before quantizing the key-value cache. Key results on the base model:
+
+- **5.3x faster prefill** compared to unquantized baseline
+- **28% faster decode** throughput
+- **Perplexity: 6.91** vs 7.07 for unquantized (lower is better)
+
+Because it targets the KV cache rather than weights, it stacks with weight quantization for compounding memory savings.
+
+## See Also
+
+- [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) -- Base model
+- [majentik/Mistral-Small-4-119B-RotorQuant](https://huggingface.co/majentik/Mistral-Small-4-119B-RotorQuant) -- KV cache only (no weight quantization)
+- [majentik/Mistral-Small-4-119B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/Mistral-Small-4-119B-RotorQuant-MLX-4bit) -- 4-bit MLX variant
+- [majentik/Mistral-Small-4-119B-RotorQuant-MLX-1bit](https://huggingface.co/majentik/Mistral-Small-4-119B-RotorQuant-MLX-1bit) -- 1-bit MLX variant
+- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)