Update KV-cache card with accurate template and fork requirements

README.md

@@ -1,75 +1,123 @@
 ---
-base_model: mistralai/Voxtral-4B-TTS-2603
-library_name: transformers
 license: apache-2.0
-pipeline_tag: text-to-speech
+base_model: mistralai/Voxtral-4B-TTS-2603
 tags:
+  - rotorquant
+  - kv-cache-quantization
   - voxtral
-  - audio
-  - speech
+  - mistral
   - tts
-  - text-to-speech
   - voice-cloning
   - zero-shot
-  - kv-cache
-  - rotorquant
-  - quantization
+  - quantized
+library_name: transformers
+pipeline_tag: text-to-speech
 ---
 
 # Voxtral-4B-TTS-2603-RotorQuant
 
-RotorQuant KV-cache bundle for [`mistralai/Voxtral-4B-TTS-2603`](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603). Rotational online re-basis of the acoustic KV-cache — recommended when switching voices, languages, or styles within a batch.
+**RotorQuant KV cache compression** for [mistralai/Voxtral-4B-TTS-2603](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603).
+
+This is a **documentation repository** that explains how to combine Voxtral-4B-TTS-2603's weights with RotorQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply RotorQuant via the Python package or llama.cpp fork.
 
-This artifact ships **only the quantized KV-cache** — weights load from upstream.
+## What is this?
 
-## Overview
+KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.
 
-- **Base model:** `mistralai/Voxtral-4B-TTS-2603`
-- **Capabilities:** TTS, zero-shot voice cloning, 9 languages
-- **Quantization target:** attention KV-cache only
-- **Method:** RotorQuant — orthogonal rotation + low-bit quantization, refreshed per session
+| Technique | Where it's applied | Savings |
+|-----------|-------------------|---------|
+| Weight quantization (GGUF/MLX/AWQ) | Baked into model file | Reduces disk + weight memory |
+| **RotorQuant KV cache** | At inference time | Reduces attention memory (critical for long context) |
+
+Both can be combined for maximum efficiency.
 
 ## Quickstart
 
-```python
-from transformers import VoxtralForConditionalGeneration, AutoProcessor
-from majentik_quant import RotorQuantCache
+### Option A — Python / transformers
 
-model_id = "mistralai/Voxtral-4B-TTS-2603"
-processor = AutoProcessor.from_pretrained(model_id)
-model = VoxtralForConditionalGeneration.from_pretrained(model_id, torch_dtype="auto")
+Install the `rotorquant` package:
 
-cache = RotorQuantCache.from_pretrained("majentik/Voxtral-4B-TTS-2603-RotorQuant")
+```bash
+pip install rotorquant
+```
+
+Then use it with the base model:
 
-for line, voice in utterances:
-    inputs = processor(text=line, speaker_audio=voice, return_tensors="pt")
-    audio = model.generate(**inputs, past_key_values=cache, max_new_tokens=2048)
-    processor.save_audio(audio, f"{line[:10]}.wav")
+```python
+import torch
+from transformers import VoxtralTTSForConditionalGeneration, AutoTokenizer
+from rotorquant import IsoQuantCache
+
+tokenizer = AutoTokenizer.from_pretrained("mistralai/Voxtral-4B-TTS-2603", trust_remote_code=True)
+model = VoxtralTTSForConditionalGeneration.from_pretrained(
+    "mistralai/Voxtral-4B-TTS-2603",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+# Apply RotorQuant to the KV cache
+cache = IsoQuantCache(bits=4)  # or bits=2 for more aggressive compression
+
+inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=128,
+    past_key_values=cache,
+    use_cache=True,
+)
+print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
 ```
 
-## Model specs
 
-| Field | Value |
-|---|---|
+## Model Specifications
+
+| Property | Value |
+|----------|-------|
+| Base Model | [mistralai/Voxtral-4B-TTS-2603](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603) |
+| Architecture | Text-to-speech with zero-shot voice cloning |
 | Parameters | 4B |
-| Modality | Text-in, audio-out |
-| Languages | 9 |
-| Voice cloning | Zero-shot |
-| Cache quantization | RotorQuant (rotated int4) |
-| License | Apache 2.0 |
-
-## RotorQuant vs TurboQuant
-
-| | RotorQuant | TurboQuant |
-|---|---|---|
-| Strategy | Rotational online re-basis | Per-head static calibration |
-| Memory reduction | ~4x on KV-cache | ~3.5x on KV-cache |
-| Best for | Multi-voice / multi-language batches | Single-voice, single-language sessions |
-
-## See also
-
-- [`majentik/Voxtral-4B-TTS-2603-TurboQuant`](https://huggingface.co/majentik/Voxtral-4B-TTS-2603-TurboQuant)
-- [`majentik/Voxtral-4B-TTS-2603-RotorQuant-MLX-8bit`](https://huggingface.co/majentik/Voxtral-4B-TTS-2603-RotorQuant-MLX-8bit)
-- [`majentik/Voxtral-4B-TTS-2603-RotorQuant-MLX-4bit`](https://huggingface.co/majentik/Voxtral-4B-TTS-2603-RotorQuant-MLX-4bit)
-- [`majentik/Voxtral-4B-TTS-2603-RotorQuant-MLX-2bit`](https://huggingface.co/majentik/Voxtral-4B-TTS-2603-RotorQuant-MLX-2bit)
-- [`mistralai/Voxtral-4B-TTS-2603`](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603) — upstream base model
+| Context Length | 32K |
+| BF16 Size | ~8 GB |
+| Modalities | Text → Audio |
+| License | apache-2.0 |
+
+## What is RotorQuant?
+
+[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache compression method based on Clifford algebra (Cl(3,0)) rotors — a faster, more parameter-efficient alternative to Google's TurboQuant. Uses lightweight block-diagonal rotations (independent 2D/4D rotations per pair/quartet) achieving O(d) complexity instead of O(d log d), fully parallelisable with no inter-element dependencies.
+
+**Benchmarks** (from the RotorQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):
+
+- Prefill: 3,822 tok/s (vs TurboQuant 722 tok/s)
+- Decode: 119 tok/s (vs TurboQuant 93 tok/s)
+- Perplexity: 6.91 (vs TurboQuant 7.07)
+- Parameters: 4 per rotor (vs TurboQuant 16,384)
+
+> Benchmarks are from the RotorQuant repository using Llama 3.1 8B. Performance on Voxtral-4B-TTS-2603 will differ. Please open a discussion if you have independent results.
+
+## Current Ecosystem Support
+
+| Runtime | RotorQuant Support | Notes |
+|---------|----------------------|-------|
+| Python transformers + `rotorquant` | ✅ Full | Drop-in cache class |
+| llama.cpp upstream | ❌ Not merged | Use fork below |
+| llama-cpp-turboquant fork | ✅ `planar3`, `iso3` | [GitHub](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache) |
+| LM Studio | ❌ [Requested](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1719) | Use `q8_0` as alternative |
+| Ollama | ❌ Not supported | Use `OLLAMA_KV_CACHE_TYPE=q8_0` |
+| vLLM | ❌ Not supported | — |
+| koboldcpp | ❌ Not supported | — |
+
+## Pre-quantized weight variants
+
+If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:
+
+- [MLX (Apple Silicon)](https://huggingface.co/majentik?search=Voxtral-4B-TTS-2603+MLX)
+- [GGUF (llama.cpp / Ollama / LM Studio)](https://huggingface.co/majentik?search=Voxtral-4B-TTS-2603+GGUF)
+
+## See Also
+
+- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
+- [TurboQuant paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874)
+- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)
+- [Base model: mistralai/Voxtral-4B-TTS-2603](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603)
+- [Voxtral-4B-TTS-2603 announcement](https://mistral.ai/news/voxtral)