---
license: mit
tags:
  - speech-recognition
  - speech-translation
  - diffusion
  - flow-matching
  - asr
  - s2tt
language:
  - en
  - de
library_name: pytorch
---

# ELF-S2T pretrained weights

Pretrained checkpoints for **"Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation"**.

Code: https://github.com/Sslnon/ELF-S2T

ELF-S2T performs speech-to-text as audio-conditioned generation in a continuous
text-embedding space (a frozen Whisper-large-v3 encoder + a single linear
projector conditioning a pretrained ELF flow-matching backbone). See the GitHub
repo for the full method, training, and inference code.

## Checkpoints

| File | Task | Backbone | Test metric |
|---|---|---|---|
| `asr_elf_b.pt` | ASR (LibriSpeech) | ELF-B (105.9 M) | 10.50% WER |
| `asr_elf_l.pt` | ASR (LibriSpeech) | ELF-L (653.4 M) | **5.69% WER** |
| `st_deen_elf_b.pt` | S2TT (CoVoST2 de->en) | ELF-B (105.9 M) | 25.35 BLEU |
| `st_deen_elf_l.pt` | S2TT (CoVoST2 de->en) | ELF-L (653.4 M) | **28.55 BLEU / 54.91 chrF** |

WER on LibriSpeech test-clean, BLEU/chrF on CoVoST2 de->en test. Inference
recipe: SDE sampler, K=128 steps, audio guidance w=2.0.

Each `.pt` is a training checkpoint dict with keys `elf` (backbone state_dict),
`audio_proj` (the linear projector), `step`, and `config` (the full training
config, so inference auto-detects backbone size, EOS-route, etc.).

## Usage

```bash
git clone https://github.com/Sslnon/ELF-S2T
cd ELF-S2T
pip install -r requirements.txt

# download these weights
huggingface-cli download ssinon/ELF-S2T asr_elf_l.pt --local-dir outputs/asr_elf_l_dl

# ASR inference on LibriSpeech test-clean
CKPT=outputs/asr_elf_l_dl/asr_elf_l.pt ACFG=2.0 STEPS=128 \
    GPUS=0 NPROC=1 BS=8 ./scripts/infer_ls_test_clean.sh
```

The ELF-S2T inference script reads the backbone size and recipe from each
checkpoint's embedded `config`, so the same command works for every file.

## Citation

```bibtex
@misc{li2026speechmeetselfaudio,
      title={Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation}, 
      author={Xuanchen Li and Tianrui Wang and Yuheng Lu and Zikang Huang and Yu Jiang and Chenghan Lin and Chenrui Cui and Ziyang Ma and Xingyu Ma and Chunyu Qiang and Guochen Yu and Xie Chen and Longbiao Wang and Jianwu Dang},
      year={2026},
      eprint={2606.10368},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.10368}, 
}
```