README.md

---
language: en
license: other
base_model:
  - JetLM/SDAR-1.7B-Chat
  - Qwen/Qwen3-ASR-0.6B
datasets:
  - librispeech_asr
tags:
  - automatic-speech-recognition
  - speech
  - diffusion
  - lora
  - block-diffusion
pipeline_tag: automatic-speech-recognition
---

# Diffusion ASR — trained adapter + LoRA (30h)

Trained weights for a speech-recognition system that pairs a **frozen** audio encoder with a
**frozen** block-diffusion language model, bridged by a small trained adapter and LoRA.

> **These are deltas, not a standalone model.** This repo contains only the ~5M-parameter
> adapter and the ~17M-parameter LoRA. To run anything you also need the two frozen base
> models (below) and the code from the project repo:
> **https://github.com/ipritamdash/diffusion-asr-research**

## What's in this repo

| File | Contents | Params | sha256 (first 16) |
|---|---|---|---|
| `ctc_sft30h_stair_best.pt` | audio adapter (incl. `gain`) + logit-scale `log_s` | 5,250,050 | `fe982b73182c5516` |
| `lora30h_stair_best.pt` | LoRA on the decoder (PEFT state dict, 392 tensors) | 17,432,576 | `d80455b2c66e3f1e` |

`ctc_sft30h_stair_best.pt` keys: `adapter`, `log_s`, `epoch` (40), plus dev metrics.
`lora30h_stair_best.pt` keys: `lora`, `epoch` (24), `dev_ce` (0.5147).

## Base models (frozen — download separately)

| Role | Repo | Pinned revision |
|---|---|---|
| Decoder (block-diffusion LM) | `JetLM/SDAR-1.7B-Chat` | `97cf0dc9a7e85433f3f71a6bbac6fec9d381294f` |
| Encoder (audio tower) | `Qwen/Qwen3-ASR-0.6B` | `5eb144179a02acc5e5ba31e748d22b0cf3e303b0` |

Use these exact revisions — the weights here were trained against them.

> **The base decoder must be loaded through the project repo's patched loader**
> (`qwen_sdar/sdar.py::load_patched_sdar`), which swaps the CUDA-only flash-attn for SDPA and
> adds the embeddings-input path. Loading stock `JetLM/SDAR-1.7B-Chat` with plain
> `from_pretrained` will **not** reproduce these results.

## Architecture

```
audio → Qwen3-ASR-0.6B encoder → adapter → SDAR-1.7B decoder + LoRA → text
        frozen                   trained    frozen          trained
```

- **Adapter** (`Adapter` in `qwen_sdar/adapter.py`): `Conv1d(1024→1024, k=3, stride=2, pad=1)`
  → GELU → `Linear(1024→2048)` → `LayerNorm(2048)` → ×`gain`. Per-frame; the stride-2 conv
  pools the encoder's 12.5 Hz features to 6.25 Hz. `gain` is initialised to the RMS of the
  decoder's token embeddings so the prefix matches their scale.
- **LoRA** (PEFT): `r=16`, `alpha=32`, `dropout=0`, on `q_proj, k_proj, v_proj, o_proj,
  gate_proj, up_proj, down_proj`.
- Trained with **block-causal ("staircase") attention** at every stage, so the model can be
  served on fast inference engines (see the project repo, `engine/`).

## Training

- **Data:** a 30-hour subset of LibriSpeech `train-clean-100`, with a 300-clip held-out dev split.
- **Staged, never joint:** grounding (adapter via CTC, decoder detached) → SFT (adapter
  through the frozen decoder) → LoRA (adapter frozen).
- Only ~1% of the ~2.5B-parameter system is trained (encoder 0.78B + decoder 1.7B; ~22M trained).

## Results (held-out dev-300)

WER %, lower is better. `raw` = macro WER; `norm` = normalized corpus WER (Whisper normalizer).

| Decode | raw / norm |
|---|---|
| block-2 (best) | **6.96 / 6.48** |
| block-4 | 8.24 / 7.69 |

Also served on **LMDeploy's PyTorch diffusion engine** at parity (block-2: 7.33 / 6.71, within
the decoder's kernel noise), at RTF 0.0013 (~800× real time, **block-4 decode**) under batching
on one L40S.

*Note: dev-300 is one in-domain held-out subset — not the standard `test-clean`/`test-other`
benchmark, and not evaluated on multiple datasets.*

## How to use

Clone the project repo (`github.com/ipritamdash/diffusion-asr-research`) and run inference in
**two steps, in two environments** — the encoder (`qwen-asr`) and the decoder (`transformers`)
have conflicting dependencies, so they don't share one env:

```bash
# 1. encoder env — audio -> features
pip install -r requirements-encoder.txt
python featurize.py --audio clip.wav --out features.pt

# 2. decoder env — features -> transcript (loads the adapter + LoRA from this repo)
pip install -r requirements-decode.txt
python decode.py --features features.pt \
  --adapter ctc_sft30h_stair_best.pt --lora lora30h_stair_best.pt --block_length 2
```

Both steps need a CUDA GPU. For fast serving instead, see `engine/` (merge the LoRA with
`engine/merge_lora.py`, then run on LMDeploy).

Decode config: `block_length=2` (best) or 4, `denoising_steps=4`, `confidence_threshold=0.95`,
greedy, with top-1 fallback and EOS truncation.

## Intended use & limitations

- **Scope:** a research model for **English read speech** (LibriSpeech domain). Not evaluated
  for streaming, other languages, or noisy/conversational audio.
- **In-domain, tuned set:** the 6.96 / 6.48 dev-300 numbers are on a held-out split of the same
  `train-clean-100` pool the model was tuned on — read them as an *in-domain* result, not a
  public benchmark.
- **Out-of-domain degrades sharply:** the earlier 10h version scored ~16% on `test-other` and
  ~40% on VoxPopuli; expect large degradation off clean read speech.
- **No standard-benchmark number for this 30h model:** `test-clean` / `test-other` were run
  only for a separate 10h proof-of-concept, not this model — do **not** read 6.48 as a
  `test-clean` score.

## Licensing

The adapter and LoRA weights here are trained deltas. Any use depends on the two frozen base
models, which are governed by **their own licenses** — check `JetLM/SDAR-1.7B-Chat` and
`Qwen/Qwen3-ASR-0.6B` before any redistribution, and do not redistribute a merged checkpoint
(which would embed the base weights) without confirming their terms.