README.md

---
license: apache-2.0
language: [en, de, fr, es, hi, zh]
library_name: transformers
pipeline_tag: automatic-speech-recognition
tags:
  - asr
  - speech-to-text
  - diffusion
  - diffusiongemma
  - gemma
base_model:
  - google/diffusiongemma-26B-A4B-it
  - openai/whisper-small
---

# diffusion-gemma-asr-small

📝 **Links:** [Blog](https://interfaze.ai/blog/the-first-open-source-diffusion-audio-asr-model) · [Demo Space](https://huggingface.co/spaces/interfaze-ai/diffusion-gemma-asr-demo) · [Code](https://github.com/JigsawStack/diffusion-gemma-asr)

**Audio-native, multilingual speech recognition that transcribes through DiffusionGemma's own
discrete-diffusion decoder** — not autoregressive, not an external ASR decoder. Audio is projected
directly into the Gemma embedding space, and the transcript is produced by parallel diffusion
denoising (~8–16 steps), giving real-time-plus throughput where cost is set by the number of
denoising steps, not the length of the transcript.

This repo ships the **trained adapter only** (projector + LoRA, ~42M params — 0.16% of the model).
The frozen 26B DiffusionGemma backbone and the frozen whisper-small encoder load from their own repos.

## How it works

```
raw audio ─► whisper-small encoder (frozen) ─► projector (trained, ~19M)
          ─► scatter into <audio> token slots of DiffusionGemma's encoder
          ─► DiffusionGemma decoder denoises a 192-token canvas (bidirectional, cross-attends audio)
          ─► transcript
```

- **Backbone:** `google/diffusiongemma-26B-A4B-it` — frozen, small LoRA adapters on encoder/decoder attention.
- **Audio frontend:** `openai/whisper-small` encoder — frozen feature extractor (NOT a decoder).
- **Grounding:** trained with three losses — uniform-diffusion (the generator), an AR auxiliary,
  and a **CTC loss on the projector via the frozen `lm_head`** (the key unlock that makes the
  audio embeddings transcript-predictive).

## Usage

**Install**

```bash
pip install torch peft soundfile librosa huggingface_hub \
  "transformers @ git+https://github.com/huggingface/transformers.git"   # DiffusionGemma support
```

**Transcribe in Python**

```python
import sys, soundfile as sf
from huggingface_hub import snapshot_download

repo = snapshot_download("interfaze-ai/diffusion-gemma-asr-small")   # this adapter (~170 MB)
sys.path.insert(0, repo)
from inference import load, transcribe                       # bundled in this repo

# Loads frozen DiffusionGemma-26B + whisper-small + this adapter (downloads bases on first run).
model, tok, fe = load(f"{repo}/diffusion_asr_small.pt", device="cuda")

wav, sr = sf.read("audio.wav")        # 16 kHz mono float32 (inference.py resamples if needed)
print(transcribe(wav, model, tok, fe, max_steps=16))
```

**Or from the command line**

```bash
python inference.py audio.wav        # run inside the downloaded repo dir
```

Long audio is split at silence (the encoder has a 30 s window, like Whisper). `max_steps` trades
speed for accuracy — 8 is near-best and fastest, 16 is the default.

## Languages & accuracy

Trained on FLEURS (6 languages) + LibriSpeech (en) + VoxPopuli (en/de/fr/es). WER/CER are
**Whisper-normalized** (Open-ASR / Artificial-Analysis convention), 16 diffusion steps:

| benchmark | metric | score |
|---|---|---|
| LibriSpeech test-clean (en) | WER | 6.6% |
| FLEURS English | WER | 15.7% |
| VoxPopuli English | WER | 18.5% |
| FLEURS Hindi | CER | 15.8% |
| FLEURS Mandarin | CER | 29.6% |

Among diffusion / non-autoregressive ASR it leads (6.6% on LibriSpeech vs Whisfusion's 8.3%, with a
smaller encoder). It trails autoregressive Whisper — a training-data gap (~219 h seen), not architecture.

## Files
- `diffusion_asr_small.pt` — trained adapter (`{"projector": ..., "lora": ...}`)
- `model.py`, `audio.py` — model definition (self-contained)
- `inference.py` — runnable example (load + segment + transcribe)
- `requirements.txt`

## Requirements / licensing
- Needs `transformers` from **main** (DiffusionGemma support) + `torch`, `peft`.
- Base models load from their own repos under their licenses:
  `google/diffusiongemma-26B-A4B-it` (Gemma terms) and `openai/whisper-small` (MIT).
- This adapter: Apache-2.0.

## Limitations
- Per-segment window is ≤30 s (encoder limit) — long audio is chunked at silence, same as Whisper.
- Mandarin is the weakest language; more data is the lever.

## Prior work
Diffusion ASR is not new — [TransFusion](https://arxiv.org/abs/2210.07677) (2022) and
[Whisfusion](https://arxiv.org/abs/2508.07048) (2025) came before. diffusion-gemma-asr-small is, as far as
we know, the first *multilingual* one, the first built on DiffusionGemma, and the first done by
training only a ~42M-param adapter on a frozen, off-the-shelf diffusion LLM.