| --- |
| base_model: mistralai/Voxtral-Mini-4B-Realtime-2602 |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - voxtral |
| - audio |
| - speech |
| - speech-recognition |
| - realtime |
| - streaming |
| - asr |
| - kv-cache |
| - rotorquant |
| - quantization |
| --- |
| |
| # Voxtral-Mini-4B-Realtime-2602-RotorQuant |
|
|
| RotorQuant KV-cache bundle for [`mistralai/Voxtral-Mini-4B-Realtime-2602`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602). Rotational online re-basis of the attention cache — preferred for noisy, multi-speaker, or code-switching real-time streams. |
|
|
| This artifact ships **only the quantized KV-cache** — weights load from upstream. |
|
|
| ## Overview |
|
|
| - **Base model:** `mistralai/Voxtral-Mini-4B-Realtime-2602` |
| - **Capabilities:** real-time ASR, streaming speech-to-text |
| - **Quantization target:** attention KV-cache only |
| - **Method:** RotorQuant — orthogonal rotation + low-bit quantization, refreshed per session |
|
|
| ## Quickstart |
|
|
| ```python |
| from transformers import VoxtralForConditionalGeneration, AutoProcessor |
| from majentik_quant import RotorQuantCache |
| |
| model_id = "mistralai/Voxtral-Mini-4B-Realtime-2602" |
| processor = AutoProcessor.from_pretrained(model_id) |
| model = VoxtralForConditionalGeneration.from_pretrained(model_id, torch_dtype="auto") |
| |
| cache = RotorQuantCache.from_pretrained("majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant") |
| |
| for chunk in audio_stream(): |
| inputs = processor(audio=chunk, return_tensors="pt") |
| out = model.generate(**inputs, past_key_values=cache, max_new_tokens=32) |
| emit(processor.batch_decode(out, skip_special_tokens=True)[0]) |
| ``` |
|
|
| ## Model specs |
|
|
| | Field | Value | |
| |---|---| |
| | Parameters | 4B | |
| | Modality | Streaming audio-in, text-out | |
| | Use case | Real-time ASR | |
| | Cache quantization | RotorQuant (rotated int4) | |
| | License | Apache 2.0 | |
|
|
| ## RotorQuant vs TurboQuant |
|
|
| | | RotorQuant | TurboQuant | |
| |---|---|---| |
| | Strategy | Rotational online re-basis | Per-head static calibration | |
| | Memory reduction | ~4x on KV-cache | ~3.5x on KV-cache | |
| | Best for | Noisy/multi-speaker streams | Predictable domains, lowest p50 latency | |
|
|
| ## See also |
|
|
| - [`majentik/Voxtral-Mini-4B-Realtime-2602-TurboQuant`](https://huggingface.co/majentik/Voxtral-Mini-4B-Realtime-2602-TurboQuant) |
| - [`majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant-MLX-8bit`](https://huggingface.co/majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant-MLX-8bit) |
| - [`majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant-MLX-4bit`](https://huggingface.co/majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant-MLX-4bit) |
| - [`majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant-MLX-2bit`](https://huggingface.co/majentik/Voxtral-Mini-4B-Realtime-2602-RotorQuant-MLX-2bit) |
| - [`mistralai/Voxtral-Mini-4B-Realtime-2602`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602) — upstream base model |
|
|
