---
tags:
  - audio
  - speech-to-text
  - streaming
  - voxtral
  - mistral
language:
  - en
library_name: custom
pipeline_tag: automatic-speech-recognition
license: apache-2.0
---

# Voxtral Realtime 4B

Streaming speech-to-text model with ~4 billion parameters. Weights in BF16 safetensors format, extracted from [mistralai/Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602).

## Architecture

**Pipeline:**
```
WAV → 16kHz → Mel Spectrogram → Conv Stem → Encoder → Downsample 4x → Adapter → Decoder → Tokens
```

- **Audio Encoder**: ~0.6B params — causal transformer, 32 layers
- **Audio-Language Adapter**: 2-layer MLP with 4x downsample
- **LLM Decoder**: ~3.4B params — Ministral-3 based, 26 layers with GQA

### Audio Preprocessing

| Parameter | Value |
|-----------|-------|
| Sample rate | 16,000 Hz |
| Frame rate | 12.5 Hz |
| Mel bins | 128 |
| Hop length | 160 samples (10ms) |
| Window size | 400 samples (25ms) |
| 1 text token | 80ms of audio |

### Encoder (Causal Transformer)

| Parameter | Value |
|-----------|-------|
| dim | 1280 |
| layers | 32 |
| heads | 32 (MHA) |
| head_dim | 64 |
| hidden_dim | 5120 |
| FFN | SwiGLU |
| Norm | RMSNorm (eps=1e-5) |
| Position | RoPE (theta=1e6, interleaved) |
| Attention | causal, sliding window=750 |

Conv stem: `conv1d(128→1280, k=3, s=1)` → GELU → `conv1d(1280→1280, k=3, s=2)` → GELU

### Adapter

```
[seq/4, 5120] → Linear(5120→3072) → GELU → Linear(3072→3072) → [seq/4, 3072]
```

### Decoder (LLM)

| Parameter | Value |
|-----------|-------|
| dim | 3072 |
| layers | 26 |
| heads | 32 |
| KV heads | 8 (GQA 4:1) |
| head_dim | 128 |
| hidden_dim | 9216 |
| Norm | RMSNorm (eps=1e-5) |
| Position | RoPE (theta=1e6) |
| Attention | causal, sliding window=8192 |
| Vocab size | 131,072 |
| Tied embeddings | yes |

The decoder uses adaptive RMS normalization conditioned on transcription delay (6 delay tokens = 480ms).

## Weight Format

- **`consolidated.safetensors`** (8.3 GB) — 711 tensors, all BF16
- **`params.json`** — model config
- **`tekken.json`** (14.9 MB) — Tekken tokenizer

## Tokenizer (Tekken)

| Token | ID |
|-------|----|
| BOS | 1 |
| EOS | 2 |
| STREAMING_PAD | 32 |

Token IDs 0–999 are special tokens. IDs 1000+ index into the vocabulary (base64-encoded byte sequences in `tekken.json`).

### Audio Streaming Config

| Parameter | Value |
|-----------|-------|
| sampling_rate | 16,000 |
| frame_rate | 12.5 (80ms per token) |
| transcription_delay_ms | 480 (6 delay tokens) |
| left_pad_tokens | 32 |
| right_pad_tokens (offline) | 17 |

## Decode Schedule (Offline)

1. **Prompt**: `[BOS] + [STREAMING_PAD] × 38` (1 + 32 left-pad + 6 delay)
2. **Prefill**: Feed `audio_embed[i] + tok_embed(prompt[i])` for positions 0..L-2
3. **First token**: Greedy argmax from position L-1
4. **Autoregressive decode**: For each remaining audio position, feed `audio_embed[pos] + tok_embed(prev_token)`, greedy argmax
5. **Stop**: On EOS or end of audio span

## C Implementation

A pure C implementation of this model is available at [voxtral.c](https://github.com/tantk/mistralhack/tree/master/voxtral.c) — runs on Apple Silicon (Metal) and CPU (BLAS), with streaming microphone input.

## Credits

Original model by [Mistral AI](https://mistral.ai/): [`mistralai/Voxtral-Mini-4B-Realtime-2602`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602)

Built for the [Mistral Hackathon 2026](https://huggingface.co/mistral-hackaton-2026).