Hi,

Thanks to the Mistral team for open sourcing this incredible model!

Hoever, we are unable to reproduce benchmarks mentioned in the paper. The dataset details and inference logic is described below.

Benchmarking Details

Datasets and test split used for benchmarking ✅

📊 Reproduction Results — Voxtral Realtime (WER %)

English Short-Form (Table 5 columns)

English Long-Form (Table 6)

FLEURS (Table 7)

Mozilla Common Voice (Table 8)

Inference details

1. Text Normalization — Our Pipeline

_PUNCT_RE = re.compile(r"[^\w\s]", re.UNICODE)
_WS_RE = re.compile(r"\s+")

def normalise_text(text: str) -> str:
    text = text.lower()
    text = _PUNCT_RE.sub("", text)    # strip everything NOT \w or whitespace
    text = _WS_RE.sub(" ", text).strip()
    return text

Applied to both reference and hypothesis before WER/CER computation.

What this does:

Important: Hindi/Devanagari issue ⚠️

The regex [^\w\s] strips all non-word, non-space characters. In Devanagari, dependent vowel signs (matras) like े, ु, ि, ा are Unicode category Mark (Mn/Mc), NOT matched by \w. So our normalization strips them:

Input:  "नमस्ते, दुनिया!" 
Output: "नमसत दनय"  
         ↑ matras े, ु, ि, ा stripped!

This corrupts Hindi text. However, since the same normalization is applied to both reference and hypothesis, the WER impact depends on whether both sides lose the same characters. If the model faithfully reproduces the matras and the reference has them too, both get stripped symmetrically and WER may not be heavily affected. But any asymmetric stripping (e.g., a word boundary shifts due to padding) could create artificial mismatches.

2. Inference & Padding Logic — Detailed Walkthrough

Overview of the data flow

Raw audio (16 kHz) 
     │
     ▼
  _right_pad_audio()      ← add silence at the END
     │
     ▼
  processor() with is_streaming=False
     │  ├── left-pads the raw waveform with silence (32 adapter frames ≈ 2.56 s)
     │  ├── converts waveform → log-Mel spectrogram (128 bins, hop=160)
     │  └── tokenizes the audio into [STREAMING_PAD]*32 + audio tokens
     │
     ▼
  model.generate() with max_new_tokens
     │
     ▼
  processor.batch_decode(skip_special_tokens=True)
     │
     ▼
  normalise_text()        ← lower-case, strip punctuation, collapse whitespace
     │
     ▼
  evaluate_pair()         ← HuggingFace evaluate WER/CER

Right-padding: why τ + 10 frames?

The decoder emits one token per 80 ms adapter frame. For the last word in the utterance, its word-boundary token [W] fires τ frames after the acoustic evidence ends. Then the subword tokens of that word follow at τ+1, τ+2, … without right-padding, the encoder runs out of states and the decoder must hallucinate text without acoustic grounding → inflated WER. The 10-frame headroom covers subword tokens after [W] (~800 ms worth).

Left-padding: 32 [STREAMING_PAD] tokens

The model's AudioConfig has streaming_n_left_pad_tokens=32. When we call the processor with is_streaming=False, it:

1. Prepends 32 adapter frames of silence (2.56 s) to the raw waveform
2. Injects 32 [STREAMING_PAD] tokens at the start of input_ids

Per the paper's §6.3, this acts as attention sinks — the initial attention mass is absorbed by these dummy tokens, stabilizing the decoding. The paper tested 0, 16, and 32 frames and found 32 works best (Table 4).

Generation is greedy, no sampling

model.generate(**model_kwargs, max_new_tokens=batch_max_tokens)

No temperature, do_sample, top_k, or top_p — default greedy decoding (equivalent to temperature=0). Standard evaluation practice.

Thanks again and please reply for any missing details!