Model card

README.md

@@ -0,0 +1,113 @@
+---
+license: apache-2.0
+base_model: CohereLabs/cohere-transcribe-03-2026
+pipeline_tag: automatic-speech-recognition
+library_name: transformers
+language: [en, fr, de, es, it, pt, nl, pl]
+tags: [asr, streaming, speech-recognition, cache-aware, fastconformer, distillation]
+datasets: [openslr/librispeech_asr, facebook/multilingual_librispeech, facebook/voxpopuli, fixie-ai/common_voice_17_0,
+           google/fleurs, speechcolab/gigaspeech, edinburghcstr/ami, MLCommons/peoples_speech]
+---
+
+# Cohere Streaming (2B)
+
+A **cache-aware streaming** version of [Cohere Transcribe](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026)
+(2B, offline). Words appear while the person is speaking. When they stop, a second pass produces the final
+transcript, close to offline accuracy.
+
+- **One model, five latencies:** live-caption lookahead of 80, 160, 320, 560 or 1,120 ms, chosen at load time.
+- **Final transcript doesn't depend on the latency setting:** at the end of speech the buffered utterance is
+  re-encoded at 1.12 s lookahead and decoded with a 4-beam search, about 0.3 s after the speaker stops.
+- **8 languages:** English, French, German, Spanish, Italian, Portuguese, Dutch, Polish.
+
+## Results
+
+All Open ASR Leaderboard numbers below use the leaderboard's own data loading, normaliser and scoring
+([huggingface/open_asr_leaderboard](https://github.com/huggingface/open_asr_leaderboard) @ `67e8bd6`). The other
+systems' numbers are the leaderboard's published ones. Our final transcript is the same at every latency setting,
+within ±0.05 average WER.
+
+**English (8 public datasets, WER %)**
+
+| | AMI | Earnings22 | GigaSpeech | LS clean | LS other | SPGISpeech | Monsoon | VoxPopuli | **Avg** |
+|---|---|---|---|---|---|---|---|---|---|
+| **Cohere Streaming (2B)** | **9.80** | 14.38 | 10.30 | 2.03 | **4.72** | 3.91 | 7.76 | 5.19 | **7.26** |
+| Nemotron 3.5 ASR streaming (0.6B) | 13.43 | 15.30 | 9.86 | 2.83 | 6.79 | 3.27 | 7.30 | 4.24 | 7.88 |
+| Voxtral Mini 4B Realtime | 13.34 | 9.31 | 8.80 | 1.62 | 4.94 | 2.23 | 8.86 | 2.60 | 6.46 |
+| *Cohere Transcribe, offline (not streaming)* | *7.02* | *7.88* | *7.88* | *0.97* | *2.05* | *2.74* | *3.44* | *5.38* | *4.67* |
+
+**Multilingual (average of FLEURS / Common Voice / MLS per language, WER %)**
+
+| | fr | de | es | it | pt | nl | **Avg** |
+|---|---|---|---|---|---|---|---|
+| **Cohere Streaming (2B)** | **6.91** | 6.41 | 4.63 | **5.48** | 10.19 | 9.78 | **7.23** |
+| Nemotron 3.5 ASR streaming (0.6B) | 9.40 | 8.96 | 5.43 | 8.66 | 8.25 | 14.50 | 9.20 |
+| Voxtral Mini 4B Realtime | 7.78 | 6.34 | 4.18 | 6.41 | 6.22 | 9.19 | 6.69 |
+| *Cohere Transcribe, offline* | *4.02* | *3.10* | *2.81* | *2.97* | *5.99* | *4.17* | *3.84* |
+
+**Latency (LibriSpeech, real-time streaming harness, one A100, one stream)**
+
+| Setting | WER clean / other | Median word delay (clean / other) | Final text after end of speech | Real-time factor, 1 stream |
+|---|---|---|---|---|
+| 80 ms | 1.36 / 4.85 | 303 / 356 ms | 364 ms | 0.68× |
+| 320 ms | 1.36 / 4.85 | 263 / 296 ms | 302 ms | 1.69× |
+| 560 ms | 1.36 / 4.85 | 387 / 429 ms | 296 ms | 2.46× |
+| 1120 ms | 1.36 / 4.85 | 680 / 725 ms | 321 ms | 2.99× |
+
+Word delay is measured against forced-aligned word ends, and includes compute time. At 80 ms the per-stream Python
+loop is slower than real time on one A100. Use 320 ms or more, or a faster GPU, for live use.
+
+## Usage
+
+The streaming loop lives in the companion code repository (it needs the cache-aware encoder and the endpoint logic,
+which are not part of 🤗 Transformers):
+
+```python
+import numpy as np
+from baselines.cohere_streaming import CohereStreaming
+
+model = CohereStreaming("Steveeeeeeen/cohere-streaming", lookahead=6)   # 560 ms live captions
+model.reset(language="en")
+for chunk in audio_chunks_80ms:               # float32, 16 kHz, 1280 samples each
+    live_text = model.step(chunk)             # current partial transcript
+final_text = model.finish()                   # end of speech: final transcript
+```
+
+Or stream a file from the command line: `python examples/stream_file.py speech.wav --lookahead 6 --realtime`.
+
+Files: `student.safetensors` (streaming encoder, Cohere decoder, CTC head), `mel_stats.json` (statistics for the
+causal feature normalisation), `config.json` (architecture and training arguments), `log.json` (training log).
+
+## How it was made
+
+1. **Exact conversion** of Cohere's 48-layer FastConformer encoder into transformers' cache-aware streaming encoder:
+   - BatchNorm folded into a per-channel affine;
+   - subsampling made causal in time;
+   - parity checked to 1e-4 in non-causal mode.
+2. **Distillation from offline Cohere.** About 12.5k hours were transcribed by the offline model, and the streaming
+   student was trained on those transcripts. Losses: decoder cross-entropy, CTC, and cosine feature distillation at 4
+   encoder layers. Each batch samples one lookahead from {0, 1, 3, 6, 13}.
+   - **Read speech:** LibriSpeech, MLS, VoxPopuli, Common Voice, FLEURS-train.
+   - **Conversational speech:** about 1,460 h of AMI, GigaSpeech and People's Speech.
+3. **Robustness:**
+   - causal running feature normalisation with a randomised warm-up prior;
+   - ±10 dB gain augmentation.
+4. **Endpoint decoding, tuned on dev sets:**
+   - re-encode the utterance at 1.12 s lookahead;
+   - 4-beam search;
+   - bracket tokens suppressed.
+
+## Limitations
+
+- **The final text takes about 0.3 s after end of speech.** Nemotron takes about 50 ms.
+- **Live partials can be rewritten** before they settle, especially at 80–320 ms on noisy audio.
+- **Transcription style follows Cohere's:** lightly edited, not verbatim. This costs WER on verbatim references
+  (VoxPopuli-cleaned) and on fully edited ones (SPGISpeech, where it keeps "you know").
+- **Weaker spots:** financial vocabulary (Earnings22), Portuguese, and Common Voice Dutch.
+- **Long inputs:** utterances over 30 s fall back to the streamed states for the final pass, and the final decode
+  is capped at 256 tokens. Segment long audio with a VAD.
+
+## License and attribution
+
+Apache-2.0, like the base model [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026).
+The training audio comes from the datasets listed above, under their own licenses.