README.md

---
license: apache-2.0
base_model: CohereLabs/cohere-transcribe-03-2026
pipeline_tag: automatic-speech-recognition
library_name: transformers
language: [en, fr, de, es, it, pt, nl, pl]
tags: [asr, streaming, speech-recognition, cache-aware, fastconformer, distillation]
datasets: [openslr/librispeech_asr, facebook/multilingual_librispeech, facebook/voxpopuli, fixie-ai/common_voice_17_0,
           google/fleurs, speechcolab/gigaspeech, edinburghcstr/ami, MLCommons/peoples_speech]
---

# Cohere Streaming (2B)

A **cache-aware streaming** version of [Cohere Transcribe](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026)
(2B, offline). Words appear while the person is speaking. When they stop, a second pass produces the final
transcript, close to offline accuracy.

- **One model, five latencies:** live-caption lookahead of 80, 160, 320, 560 or 1,120 ms, chosen at load time.
- **Final transcript doesn't depend on the latency setting:** at the end of speech the buffered utterance is
  re-encoded at 1.12 s lookahead and decoded with a 4-beam search, about 0.3 s after the speaker stops.
- **8 languages:** English, French, German, Spanish, Italian, Portuguese, Dutch, Polish.

## Results

All Open ASR Leaderboard numbers below use the leaderboard's own data loading, normaliser and scoring
([huggingface/open_asr_leaderboard](https://github.com/huggingface/open_asr_leaderboard) @ `67e8bd6`). The other
systems' numbers are the leaderboard's published ones. Our final transcript is the same at every latency setting,
within ±0.05 average WER.

**English (8 public datasets, WER %)**

| | AMI | Earnings22 | GigaSpeech | LS clean | LS other | SPGISpeech | Monsoon | VoxPopuli | **Avg** |
|---|---|---|---|---|---|---|---|---|---|
| **Cohere Streaming (2B)** | **9.80** | 14.38 | 10.30 | 2.03 | **4.72** | 3.91 | 7.76 | 5.19 | **7.26** |
| Nemotron 3.5 ASR streaming (0.6B) | 13.43 | 15.30 | 9.86 | 2.83 | 6.79 | 3.27 | 7.30 | 4.24 | 7.88 |
| Voxtral Mini 4B Realtime | 13.34 | 9.31 | 8.80 | 1.62 | 4.94 | 2.23 | 8.86 | 2.60 | 6.46 |
| *Cohere Transcribe, offline (not streaming)* | *7.02* | *7.88* | *7.88* | *0.97* | *2.05* | *2.74* | *3.44* | *5.38* | *4.67* |

**Multilingual (average of FLEURS / Common Voice / MLS per language, WER %)**

| | fr | de | es | it | pt | nl | **Avg** |
|---|---|---|---|---|---|---|---|
| **Cohere Streaming (2B)** | **6.91** | 6.41 | 4.63 | **5.48** | 10.19 | 9.78 | **7.23** |
| Nemotron 3.5 ASR streaming (0.6B) | 9.40 | 8.96 | 5.43 | 8.66 | 8.25 | 14.50 | 9.20 |
| Voxtral Mini 4B Realtime | 7.78 | 6.34 | 4.18 | 6.41 | 6.22 | 9.19 | 6.69 |
| *Cohere Transcribe, offline* | *4.02* | *3.10* | *2.81* | *2.97* | *5.99* | *4.17* | *3.84* |

**Latency (LibriSpeech, real-time streaming harness, one A100, one stream)**

| Setting | WER clean / other | Median word delay (clean / other) | Final text after end of speech | Real-time factor, 1 stream |
|---|---|---|---|---|
| 80 ms | 1.36 / 4.85 | 303 / 356 ms | 364 ms | 0.68× |
| 320 ms | 1.36 / 4.85 | 263 / 296 ms | 302 ms | 1.69× |
| 560 ms | 1.36 / 4.85 | 387 / 429 ms | 296 ms | 2.46× |
| 1120 ms | 1.36 / 4.85 | 680 / 725 ms | 321 ms | 2.99× |

Word delay is measured against forced-aligned word ends, and includes compute time. At 80 ms the per-stream Python
loop is slower than real time on one A100. Use 320 ms or more, or a faster GPU, for live use.

## Usage

The checkpoint includes its streaming model code, processor, and generation settings. Load it directly with Transformers:

```bash
pip install "transformers==5.14.1" torch librosa soundfile
hf auth login
```

```python
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Steveeeeeeen/cohere-streaming",
    trust_remote_code=True,
    lookahead=6,
)
model.reset(language="en")
for chunk in audio_chunks_80ms:  # Mono float32 NumPy arrays: 16 kHz, 1,280 samples each.
    print(model.step(chunk))
print(model.finish())
```

The loader chooses CUDA, Apple Metal, or CPU. Pass `device="cuda"` or `device="cpu"` to choose explicitly, and `dtype=torch.bfloat16` or `dtype=torch.float32` to choose precision. Streaming supports one device. `reset()` starts an utterance, `step()` updates its partial transcript, and `finish()` flushes the remaining audio and decodes the endpoint.

Inference loads the weights and processor from this public repository without an authentication token; it does not download the original Cohere weights. Access to the original gated Cohere model is still needed for teacher labeling and distillation.

`trust_remote_code=True` loads the streaming implementation shipped here. This adapter exposes the streaming methods above; standard ASR pipeline streaming and `AutoModel.from_config()` are not implemented.

To save a copy with its code and processor:

```python
model.save_pretrained("cohere-streaming-local")
```

The directory contains `student.safetensors`, `config.json`, `mel_stats.json`, `model.py`, processor/tokenizer files, and `generation_config.json`. The published repository also retains `log.json` from training. The local `CohereStreaming` adapter in the [smol-audio example](https://github.com/Deep-unlearning/smol-audio/tree/master/cohere_transcribe_streaming) remains compatible with this checkpoint.

## How it was made

1. **Exact conversion** of Cohere's 48-layer FastConformer encoder into transformers' cache-aware streaming encoder:
   - BatchNorm folded into a per-channel affine;
   - subsampling made causal in time;
   - parity checked to 1e-4 in non-causal mode.
2. **Distillation from offline Cohere.** About 12.5k hours were transcribed by the offline model, and the streaming
   student was trained on those transcripts. Losses: decoder cross-entropy, CTC, and cosine feature distillation at 4
   encoder layers. Each batch samples one lookahead from {0, 1, 3, 6, 13}.
   - **Read speech:** LibriSpeech, MLS, VoxPopuli, Common Voice, FLEURS-train.
   - **Conversational speech:** about 1,460 h of AMI, GigaSpeech and People's Speech.
3. **Robustness:**
   - causal running feature normalisation with a randomised warm-up prior;
   - ±10 dB gain augmentation.
4. **Endpoint decoding, tuned on dev sets:**
   - re-encode the utterance at 1.12 s lookahead;
   - 4-beam search;
   - bracket tokens suppressed.

## Limitations

- **The final text takes about 0.3 s after end of speech.** Nemotron takes about 50 ms.
- **Live partials can be rewritten** before they settle, especially at 80–320 ms on noisy audio.
- **Transcription style follows Cohere's:** lightly edited, not verbatim. This costs WER on verbatim references
  (VoxPopuli-cleaned) and on fully edited ones (SPGISpeech, where it keeps "you know").
- **Weaker spots:** financial vocabulary (Earnings22), Portuguese, and Common Voice Dutch.
- **Long inputs:** utterances over 30 s fall back to the streamed states for the final pass, and the final decode
  is capped at 256 tokens. Segment long audio with a VAD.

## License and attribution

Apache-2.0, like the base model [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026).
The training audio comes from the datasets listed above, under their own licenses.