| --- |
| base_model: mistralai/Voxtral-Mini-3B-2507 |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - voxtral |
| - audio |
| - speech |
| - speech-recognition |
| - transcription |
| - translation |
| - kv-cache |
| - rotorquant |
| - quantization |
| --- |
| |
| # Voxtral-Mini-3B-2507-RotorQuant |
|
|
| RotorQuant KV-cache for [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507). Uses a rotational online re-basis of the attention cache that is robust to distributional drift across long, code-switched, or noisy audio streams. |
|
|
| This artifact ships **only the quantized KV-cache bundle** — model weights load from the upstream repo. |
|
|
| ## Overview |
|
|
| - **Base model:** `mistralai/Voxtral-Mini-3B-2507` |
| - **Capabilities:** transcription, speech translation, audio understanding |
| - **Quantization target:** attention KV-cache only |
| - **Method:** RotorQuant — orthogonal rotation + low-bit quantization, refreshed per session |
|
|
| RotorQuant trades a tiny per-session calibration pass for better low-bit stability on streaming audio. Preferred when audio domains shift mid-stream (multi-speaker meetings, code-switching, noise bursts). |
|
|
| ## Quickstart |
|
|
| ```python |
| from transformers import VoxtralForConditionalGeneration, AutoProcessor |
| from majentik_quant import RotorQuantCache |
| |
| model_id = "mistralai/Voxtral-Mini-3B-2507" |
| processor = AutoProcessor.from_pretrained(model_id) |
| model = VoxtralForConditionalGeneration.from_pretrained(model_id, torch_dtype="auto") |
| |
| cache = RotorQuantCache.from_pretrained("majentik/Voxtral-Mini-3B-2507-RotorQuant") |
| |
| inputs = processor(audio="meeting.wav", return_tensors="pt") |
| out = model.generate(**inputs, past_key_values=cache, max_new_tokens=512) |
| print(processor.batch_decode(out, skip_special_tokens=True)[0]) |
| ``` |
|
|
| ## Model specs |
|
|
| | Field | Value | |
| |---|---| |
| | Parameters | 3B | |
| | Modality | Audio-in, text-out | |
| | Languages | Multilingual (24+) | |
| | Cache quantization | RotorQuant (rotated int4) | |
| | License | Apache 2.0 | |
|
|
| ## RotorQuant vs TurboQuant |
|
|
| | | RotorQuant | TurboQuant | |
| |---|---|---| |
| | Strategy | Rotational online re-basis | Per-head static calibration | |
| | Memory reduction | ~4x on KV-cache | ~3.5x on KV-cache | |
| | Best for | Streaming, code-switching audio | Batch transcription, fixed domains | |
| | Calibration cost | Per-session light re-basis | One-shot, fast | |
|
|
| ## See also |
|
|
| - [`majentik/Voxtral-Mini-3B-2507-TurboQuant`](https://huggingface.co/majentik/Voxtral-Mini-3B-2507-TurboQuant) — static calibrated KV-cache variant |
| - [`majentik/Voxtral-Mini-3B-2507-RotorQuant-MLX-8bit`](https://huggingface.co/majentik/Voxtral-Mini-3B-2507-RotorQuant-MLX-8bit) — MLX weight-quantized 8-bit |
| - [`majentik/Voxtral-Mini-3B-2507-RotorQuant-MLX-4bit`](https://huggingface.co/majentik/Voxtral-Mini-3B-2507-RotorQuant-MLX-4bit) — MLX weight-quantized 4-bit |
| - [`majentik/Voxtral-Mini-3B-2507-RotorQuant-MLX-2bit`](https://huggingface.co/majentik/Voxtral-Mini-3B-2507-RotorQuant-MLX-2bit) — MLX weight-quantized 2-bit |
| - [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) — upstream base model |
|
|
