README.md

---
language: en
license: other
base_model:
  - JetLM/SDAR-1.7B-Chat
  - Qwen/Qwen3-ASR-0.6B
datasets:
  - librispeech_asr
tags:
  - automatic-speech-recognition
  - speech
  - diffusion
  - lora
  - block-diffusion
pipeline_tag: automatic-speech-recognition
---

# Diffusion ASR

Speech recognition with a **block-diffusion language model as the decoder**. A frozen audio
encoder and a frozen text diffusion LM are joined by a small trained adapter plus LoRA — about
1 % of the stack is trained.

**4.60 % WER on LibriSpeech test-clean** (all 2,620 clips).

> **These are trained deltas, not a standalone model.** This repo holds the 5.25 M-parameter
> adapter and the 17.4 M-parameter LoRA. Running them also needs the two frozen base models
> below and the inference code from
> [ipritamdash/diffusion-asr-research](https://github.com/ipritamdash/diffusion-asr-research).

## Why this exists

The audio encoder came from a working ASR system. That system's own decoder is thrown away here
and a text-only diffusion model put in its place, connected through roughly 5 M trained
parameters.

The question is whether a language model that has never seen audio can be conditioned on
acoustic evidence through a thin learned interface — and whether writing several tokens per
forward pass buys anything.

Compare it against other frozen-encoder + frozen-LM + adapter systems. Do not compare it against
end-to-end ASR trained on thousands of hours; it saw 30.

## Results

Full LibriSpeech test-clean, 2,620 clips, corpus WER after Whisper English normalisation.

| Decoder | Config | WER | vs real time |
|---|---|---:|---:|
| **Reference loop** | block 2 | **4.60** | — |
| Reference loop | block 4 | 5.23 | — |
| **LMDeploy engine, swept config** | **block 4 · 4 steps · `sequential`** | **4.78** | 7.0× |
| LMDeploy engine, old default | block 2 · dynamic | 5.00 | 9.1× |
| LMDeploy engine, old default | block 4 · dynamic | 5.63 | 13.5× |

Speed is single-stream real-time multiple on an A100; batched serving measured far higher
(~800× on one L40S in an earlier benchmark). Under the engine's original confidence-ordered
commit rule it read a constant 0.40 worse than the reference loop; the swept `sequential`
commit order closes half of that (engine best 4.78 vs reference 4.60). The remaining +0.18 is
unexplained and stated rather than left out.

### Error analysis

Computed over every clip of the 4.60 decode.

**Output health** — zero empty transcripts, zero repeat loops, zero length blow-ups, and no clip
hit the token budget.

**Composition** — of 53,029 reference words: 3.75 % substituted, 0.40 % deleted, 0.45 %
inserted. Four errors in five are a *wrong* word rather than a missing one; the model commits to
a plausible word from unclear evidence instead of failing to produce one.

**Substitutions** — 40 % are within a small character edit of the reference (*jailer* →
*jailor*, *holmes* → *homes*), which reads as blurred acoustic detail rather than a language
failure.

**Multi-piece words** — words the tokeniser splits into several subword pieces are 7.5 % of
words but carry 52 % of errors, at 13× the failure rate of single-piece words. This is the
dominant failure mode.

**Length and position** — the shortest quarter of clips is hardest (5.85 % against 4.37 % on the
longest), and quality is flat across an utterance. A fixed token budget and a block decoder could
have degraded on long input; neither did.

## Files

| File | Contents | Params |
|---|---|---:|
| `adapter.pt` | audio adapter + `gain`, CTC logit scale, CTC blank projection | 5,250,049 (+2,050) |
| `lora.pt` | LoRA on the decoder, PEFT state dict, 392 tensors | 17,432,576 |

`adapter.pt` keys: `adapter`, `log_s`, `blank_w`, `blank_b`, `epoch`, `adapter_kind` (`conv`),
`adapter_stride` (`1`), `attn_mode` (`stair`), plus dev metrics.
`lora.pt` keys: `lora`, `epoch`, `dev_ce`, `objective`.

`log_s`, `blank_w` and `blank_b` belong to the CTC alignment loss used during training. They are
not used at decode time, and are kept so training can be resumed or audited.

## Base models — frozen, download separately

| Role | Repo | Pinned revision |
|---|---|---|
| Decoder | `JetLM/SDAR-1.7B-Chat` | `97cf0dc9a7e85433f3f71a6bbac6fec9d381294f` |
| Encoder | `Qwen/Qwen3-ASR-0.6B` | `5eb144179a02acc5e5ba31e748d22b0cf3e303b0` |

Use these exact revisions — the weights here were trained against them.

> The decoder must be loaded through the project repo's patched loader
> (`qwen_sdar/sdar.py::load_patched_sdar`), which swaps CUDA-only flash-attn for SDPA and adds
> the embeddings-input path. Plain `from_pretrained` will not reproduce these numbers.

## Architecture

```
audio 16 kHz → Qwen3-ASR-0.6B encoder → adapter → SDAR-1.7B decoder + LoRA → text
               frozen, 1024-d @ 12.5 Hz  trained   frozen                     trained
```

**Adapter** — `Conv1d(1024→1024, k=3, stride=1)` → GELU → `Linear(1024→2048)` →
`LayerNorm(2048)` → ×`gain`. Stride 1 keeps the encoder's native 12.5 Hz; pooling to 6.25 Hz
measured worse across three seeds per arm, at roughly half the training cost.

`gain` initialises to the RMS of the decoder's token embeddings. A random projection emits
vectors 10–50× the size of real word vectors, and a decoder fed a prefix that loud learns to
ignore it; matching the embedding scale is what makes the audio legible at all.

**LoRA** — `r=16`, `alpha=32`, dropout 0, on `q,k,v,o,gate,up,down` in all 28 layers. LoRA scales
updates by `alpha/r`, so the effective learning rate on those weights is 2× nominal.

**Attention is block-causal at every stage**, training and inference alike. Each block sees the
audio, all earlier blocks, and itself. Training and serving therefore use the same convention,
and the model runs on a standard diffusion inference engine without retraining.

## Training

30 hours of LibriSpeech `train-clean-100`, speaker-disjoint from the held-out split. Three
stages, never joint:

| Stage | Trains | Loss |
|---|---|---|
| Grounding | adapter | CTC only, decoder detached |
| SFT | adapter | CTC + diffusion cross-entropy |
| LoRA | LoRA | diffusion cross-entropy only, adapter frozen |

Roughly 32 GPU-hours in total.

## Decoding

Block-causal, blocks written left to right, earlier blocks frozen once committed.

| | |
|---|---|
| Block width | 4 (engine, swept) · 2 (reference loop) |
| Denoising steps per block | up to 4 |
| Commit rule | reference loop: confidence ≥ 0.95 with forced fallback; engine: **`sequential` unmasking** (position order), found best in a 50-arm sweep |
| Token budget | 160 |

A 50-arm sweep of every engine dial (widths 1–64, steps, thresholds, all three unmasking
strategies), confirmed on the full dev set, chose the engine config above; decoding wider than
the trained block-4 width collapses (width 8 already mass-produces empty output), so the
frontier honestly ends at width 4.

With the original confidence-ordered commits, width 2 (below the paper's swept range) measured
best; with `sequential` commits the trained width 4 wins — commit order, not width alone, was
carrying that result.

## Usage

The encoder and decoder have conflicting dependencies, so inference runs in two steps in two
environments. Both need a CUDA GPU.

```bash
git clone https://github.com/ipritamdash/diffusion-asr-research
cd diffusion-asr-research

huggingface-cli download Tachyeon/diffusion-asr adapter.pt lora.pt --local-dir .

# 1. encoder environment: audio -> features
pip install -r requirements-encoder.txt
python featurize.py --audio clip.wav --out features.pt

# 2. decoder environment: features -> transcript
pip install -r requirements-decode.txt
python decode.py --features features.pt --adapter adapter.pt --lora lora.pt --block_length 2
```

`decode.py` reads `adapter_stride` from the checkpoint and refuses to run if it is not recorded.
A stride-1 and a stride-2 adapter have identical tensor shapes, so guessing wrong loads cleanly
and silently emits a prefix of the wrong length.

For serving, merge the LoRA and run on LMDeploy's PyTorch diffusion engine; see `engine/` in the
project repo.

## Scope

A research model for **English read speech**. LibriSpeech is clean, read audiobook material and
these numbers describe that domain.

## Licence

The adapter and LoRA here are trained deltas. Any use depends on the two frozen base models,
which carry **their own licences** — check `JetLM/SDAR-1.7B-Chat` and `Qwen/Qwen3-ASR-0.6B`
before redistributing anything, and do not redistribute a merged checkpoint (which embeds the
base weights) without confirming their terms.