Interspeech tutorial — DeSTA-style SpeechLLM checkpoints
Whisper-large-v3 encoder (frozen) → concat+MLP adapter → Qwen3-4B-Instruct-2507 + LoRA r32. Only the adapter and the LoRA weights are trained, so each checkpoint is ~147 MB; the base models are downloaded from their own repos at load time.
| folder | training data | test-clean WER | test-clean gender |
|---|---|---|---|
selfgen_only |
281k self-generated conversational responses (no task labels) | 3.93 (best prompt + post-processing) | 98.24 |
selfgen_mix |
the same + 20% ASR + 5% gender SFT | 2.45 | 98.82 |
The self-generation targets were written by Qwen3-4B given only the transcript and the
speaker's gender: <audio>{transcription} (Gender: {gender})</audio> + "The audio is a
passage read aloud from a book. Respond directly as a natural conversation partner. Do not
mention the audio, the transcription, or the speaker attributes."
selfgen_only has never seen a transcription instruction: it can transcribe, but needs an
explicit prompt ("Transcribe the speech word for word. Output only the transcription …") and
sometimes answers in the training-time metadata format. selfgen_mix follows the plain
prompts directly.
Usage
See the Colab notebook in the tutorial repo, or:
from huggingface_hub import hf_hub_download, snapshot_download
import sys, torch
code = snapshot_download("kehanlu/interspeech-tutorial", allow_patterns="code/*")
sys.path.insert(0, f"{code}/code")
from inference import SpeechLLMForInference
ckpt = hf_hub_download("kehanlu/interspeech-tutorial", "selfgen_mix/model.ckpt")
pipe = SpeechLLMForInference.from_checkpoint(ckpt, dtype=torch.float16) # float16 for a Colab T4
print(pipe.generate([{"role": "user",
"content": "<audio><|AUDIO|></audio>\n\nTranscribe the speech into text",
"audios": [{"audio": "sample.flac"}]}]))