README.md

---
license: apache-2.0
language:
- en
- de
- fr
- es
- nl
- it
- pl
- pt
library_name: pytorch
tags:
- automatic-speech-recognition
- asr
- audio
- speech-recognition
- multilingual
- wren
- mimi
- qwen2.5
- neural-codec
pipeline_tag: automatic-speech-recognition
datasets:
- shangeth/mls-mimi-codes
- shangeth/libritts-r-mimi-codes
- shangeth/vctk-mimi-codes
- shangeth/jenny-mimi-codes
- shangeth/ljspeech-mimi-codes
- shangeth/expresso-mimi-codes-tagged
- facebook/multilingual_librispeech
- mythicinfinity/libritts_r
- keithito/lj_speech
- CSTR-Edinburgh/vctk
- reach-vb/jenny_tts_dataset
- ylacombe/expresso
---

# Wren-ASR-0.5B-multi

**Multilingual** automatic speech recognition model in the Wren series. Encodes
audio with the [Kyutai Mimi](https://huggingface.co/kyutai/mimi) neural codec,
then transcribes with a [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)
backbone — no acoustic encoder, no CTC, just a small LLM consuming Mimi codes as
input embeddings.

Supports **8 languages**: English, German, French, Spanish, Dutch, Italian, Polish, Portuguese.

## Links

- **Training & inference code:** [github.com/shangeth/wren-asr](https://github.com/shangeth/wren-asr)
- **Wren research project:** [github.com/shangeth/wren](https://github.com/shangeth/wren)
- **TTS counterpart:** [shangeth/Wren-TTS-0.5B-multi](https://huggingface.co/shangeth/Wren-TTS-0.5B-multi)
- **Dataset extraction (Mimi codes):** [github.com/shangeth/wren-datasets](https://github.com/shangeth/wren-datasets)
- **Demo Space:** [huggingface.co/spaces/shangeth/Wren-ASR-0.5B-multi-demo](https://huggingface.co/spaces/shangeth/Wren-ASR-0.5B-multi-demo)

## Architecture

```
audio ──► Mimi encoder (k=3) ──► Qwen2.5-0.5B (audio prefix → text) ──► transcript
```

Mimi codes serve as a discrete audio prefix in the LLM's input embedding space.
At each audio frame the k=3 codebook codes go through k separate input embedding
tables; their sum (scaled by 1/√k) is the input embedding for that step. The
audio prefix is wrapped in `<|audio_start|>` / `<|audio_end|>` tokens, after
which the LLM autoregressively emits text using its native vocabulary and
`lm_head` — no new output heads were added.

- **Backbone:** Qwen2.5-0.5B (causal LM; transformer body ~358M params, 151k-token multilingual vocab)
- **Audio tokenizer:** Mimi (`kyutai/mimi`), 12.5 fps, 2048-entry codebooks
- **Codebooks used:** first 3 (semantic-content-rich); reduces input embedding size 8/3× vs 8-codebook variants
- **Audio prefix:** `<|audio_start|>` + summed-codebook embeds × T_frames + `<|audio_end|>`
- **Output:** standard text autoregression via `model.llm.generate(inputs_embeds=...)`

## Training data

Trained on the **union of every dataset used to train Wren-TTS** — the same
6 corpora that power the en/multi/expressive TTS recipes, with text used as the
ASR target:

| Dataset | Rows | Language(s) |
|---|---|---|
| VCTK              | ~44k    | en (109 speakers, multiple accents) |
| Jenny             | ~21k    | en (single speaker) |
| LibriTTS-R        | ~360k   | en (clean_100 + clean_360 + other_500) |
| LJSpeech          | ~13k    | en (single speaker) |
| MLS               | ~6.0M   | de · fr · es · it · nl · pl · pt |
| Expresso (tagged) | ~26k    | en (style tags stripped at load time) |
| **Total**         | **~6.46M** rows / epoch | |

Mimi codes are pre-extracted and published as the per-corpus mimi-codes datasets
(see Datasets above) — no online encoding during training. Single-pass
from-scratch training, ~k=3 codebooks. Held-out validation combines LibriTTS-R
`dev_clean` + MLS `dev` (all 7 langs) + Expresso `dev` (tags stripped) + 5%
per single-speaker English source. All weights set to 1.0 (every row, every
epoch, no subsampling). Trained on a single A100-40GB.

Text casing and punctuation are preserved in the ground-truth transcripts.

## Usage

```bash
pip install torch torchaudio transformers
```

```python
import torch
import torchaudio
from transformers import AutoModel, AutoProcessor

model_id  = "shangeth/Wren-ASR-0.5B-multi"
device    = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model     = AutoModel.from_pretrained(model_id, trust_remote_code=True).to(device).eval()

# Load any short clip (one of the 8 supported languages, ≤ 30 s)
wav, sr = torchaudio.load("input.wav")

inputs = processor(audio=wav, sampling_rate=sr)
inputs = {k: v.to(device) for k, v in inputs.items()}

ids  = model.generate(**inputs, max_new_tokens=200)
text = processor.batch_decode(ids, skip_special_tokens=True)[0]
print(text)
```

## Sampling tips

Defaults: greedy decoding (`do_sample=False`). For longer / harder utterances:
- Pass `do_sample=True, temperature=0.7, top_p=0.9` for diverse beams
- Raise `max_new_tokens` if transcripts are getting cut off
- Audio is hard-capped at 30 s (375 frames @ 12.5 fps) by the training recipe;
  for longer audio, segment first

## Limitations & known issues

- **Language coverage:** only the 8 trained languages. Out-of-distribution
  audio produces noise / hallucinated text in the closest matching language.
- **Per-language quality varies with data volume:** German / Dutch / French
  are strongest (largest training shares); Polish / Portuguese / Italian have
  less training data and may be less accurate.
- **Audiobook-style audio dominates training:** MLS / LibriTTS-R / LJSpeech /
  Jenny are all studio-style read speech. Performance on conversational audio,
  noisy environments, or accented far-field input may degrade.
- **0.5B backbone** — quality is below frontier ASR systems (Whisper-large-v3,
  USM, etc.). The pitch is "small enough to run anywhere" + "shares architecture
  with Wren-TTS-0.5B-multi for unified speech-text experimentation".
- **30s audio cap.** Hard-cap at training time; longer audio needs to be
  segmented externally.
- **No speaker diarization.** Single-stream transcription only.

## The Wren series

Wren is a family of compact (<3B parameter) multimodal speech LLMs — small
enough to run on a single consumer GPU, designed for open research on unified
speech understanding and synthesis.

- **Wren-TTS** — text → speech (English + multilingual + expressive variants)
- **Wren-ASR** — speech → text (this release)
- **Wren-LM** — speech-language modelling / dialog (planned)
- **Wren-Omni** — unified ASR + TTS + LM in one checkpoint (planned)

All Wren models share the same design principles: small backbone LLM + neural
audio codec, open weights, simple PyTorch checkpoints, reproducible training
recipes. Wren-ASR uses the same Qwen2.5-0.5B backbone as Wren-TTS-0.5B-multi
and is trained on the same corpora — making the pair a natural starting point
for unified speech-text modelling research.

## Repository contents

| File | Purpose |
|---|---|
| `model.safetensors` | Model weights |
| `config.json` | `WrenASRConfig` (with `auto_map` for `trust_remote_code`) |
| `tokenizer.json` + friends | Qwen2.5 tokenizer with Wren-ASR's 2 special tokens added |
| `processor_config.json` | `WrenASRProcessor` auto_map |
| `configuration_wren_asr.py` | `WrenASRConfig(PretrainedConfig)` |
| `modeling_wren_asr.py` | `WrenForASR(PreTrainedModel)` — loads Mimi codec lazily on first call |
| `processing_wren_asr.py` | `WrenASRProcessor(ProcessorMixin)` — audio → Mimi codes + text decode |
| `README.md` | This model card |

## Citation

```bibtex
@misc{wren2026,
  title  = {Wren: A Family of Small Open-Weight Models for Unified Speech-Text Modelling},
  author = {Shangeth Rajaa},
  year   = {2026},
  url    = {https://github.com/shangeth/wren}
}
```

## License

Apache-2.0 for the checkpoint weights and code in this repo.
Upstream components carry their own licenses — review before redistribution.
The Expresso dataset (used for English style robustness) is CC-BY-NC-4.0; if
you build derived models on this checkpoint and want to release them
commercially, retrain with Expresso excluded.