MSP-ASR

MSP-ASR is the audio model in Multimodal Speech Perception (MSP). It fine-tunes a Wav2Vec2 encoder with a CTC head for English speech recognition and supports greedy decoding or the companion 3-gram language model.

Property	Value
Input	Mono, 16 kHz audio
Architecture	Wav2Vec2 encoder + CTC head
Base checkpoint	facebook/wav2vec2-large-robust-ft-libri-960h
Demo	MSP-ASR

MSP-ASR also supplies the audio encoder to MSP-AVSR. There, its representations participate in both directions of the bidirectional cross-attention fusion block; the standalone checkpoint remains audio-only.

Usage

import torch
from transformers import AutoModelForCTC, AutoProcessor

model_id = "MahmoodAnaam/MSP-ASR"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCTC.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = processor(audio="sample.wav", return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits

text = processor.tokenizer.batch_decode(logits.argmax(dim=-1))[0]
print(text)

For beam search, install pyctcdecode and kenlm, then load MahmoodAnaam/MSP-Processor-With-LM and call batch_decode(logits.cpu().numpy()). Pin Hub revisions when loading custom code in controlled environments.

Training

Setting	Value
Maximum steps	40,000
Learning rate	1e-4
Train / eval batch size per device	16 / 16
Gradient accumulation	2
Scheduler	Cosine; 1,000 warmup steps
Precision	bfloat16

The selected validation checkpoint recorded loss 0.3481 and WER 20.40%. The recorded stack was Transformers 5.10.2, PyTorch 2.10.0 with ROCm 7.2.4, Datasets 4.0.0, and Tokenizers 0.22.2.

Evaluation

The following tables report MSP-ASR only. Results use MSP-Processor-With-LM; WER is a percentage and lower is better.

LRS2

Evaluation	WER
Clean test	6.30
Clean + eight noisy conditions, macro average	46.34

Interferer	-5 dB	0 dB	5 dB	10 dB
1	90.22	42.76	47.40	22.52
2	92.99	42.86	48.91	23.09

AVCocktail

Segmentation	WER
Active-speaker-detection chunks	77.62
Fixed 10-second chunks	127.36
Gold chunks	74.37

Resources

• Source and training configuration
• MSP-ASR demo
• 3-gram LM processor