| ---
|
| language: de
|
| license: apache-2.0
|
| tags:
|
| - whisper
|
| - lora
|
| - peft
|
| - automatic-speech-recognition
|
| - singing-voice
|
| - lyrics-transcription
|
| base_model: openai/whisper-small
|
| library_name: peft
|
| pipeline_tag: automatic-speech-recognition
|
| ---
|
|
|
| # AUTOLYRICS — Whisper-small + LoRA for Singing Lyrics Transcription
|
|
|
| LoRA adapter for `openai/whisper-small`, fine-tuned for **singing voice → lyrics**
|
| transcription. Built as a 4-day end-to-end ML project; see the full repo at
|
| [GitHub](https://github.com/ramduvvuri/autolyrics) and live demo at
|
| [HF Space](https://huggingface.co/spaces/Petercoder/autolyrics).
|
|
|
| ## Why this exists
|
|
|
| Off-the-shelf ASR fails on singing because of pitch variation, sustained
|
| phonemes, rhythm irregularities, and (often) backing music. This adapter
|
| recovers a substantial fraction of that loss with ~0.5% extra trainable
|
| parameters.
|
|
|
| ## Results on held-out singing test set
|
|
|
| | Metric | Whisper-small (baseline) | + LoRA (this adapter) | Δ |
|
| |---|---|---|---|
|
| | WER | 37.5% | **34.5%** | **-3.0 pts** |
|
| | CER | 27.1% | **17.8%** | -9.3 pts |
|
| | RTF on T4 | 0.03 | 0.03 | ~same |
|
|
|
| Test set: 13 clips, song-disjoint from train.
|
|
|
| ## How to use
|
|
|
| ```python
|
| from peft import PeftModel
|
| from transformers import WhisperForConditionalGeneration, WhisperProcessor
|
| import torchaudio
|
|
|
| base = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
|
| model = PeftModel.from_pretrained(base, "Petercoder/autolyrics-whisper-small-lora")
|
| proc = WhisperProcessor.from_pretrained("Petercoder/autolyrics-whisper-small-lora")
|
| model.generation_config.language = "de"
|
| model.generation_config.task = "transcribe"
|
| model.generation_config.forced_decoder_ids = None
|
|
|
| wav, sr = torchaudio.load("song_clip.wav")
|
| if wav.shape[0] > 1: wav = wav.mean(0, keepdim=True)
|
| if sr != 16000: wav = torchaudio.functional.resample(wav, sr, 16000)
|
|
|
| feats = proc(wav.squeeze(0).numpy(), sampling_rate=16000,
|
| return_tensors="pt").input_features
|
| ids = model.generate(feats, num_beams=5, max_new_tokens=225)
|
| print(proc.batch_decode(ids, skip_special_tokens=True)[0])
|
| ```
|
|
|
| For best results, isolate vocals first with [Demucs](https://github.com/facebookresearch/demucs)
|
| (`htdemucs_ft`), then pass the `vocals.wav` to this model.
|
|
|
| ## Training details
|
|
|
| - Base model: `openai/whisper-small` (244M params)
|
| - PEFT: LoRA, r=32, alpha=64, dropout=0.05, target=`q_proj,v_proj`
|
| - Trainable params: ~1.2M (~0.5% of total)
|
| - Optimizer: AdamW, lr=1e-3, linear warmup 50 steps
|
| - Batch: 8 × grad_accum 2 = effective 16; fp16
|
| - Epochs: 5 with early stopping (patience=2) on eval WER
|
| - Hardware: single NVIDIA T4 (Colab Pro)
|
|
|
| ## Dataset
|
|
|
| DSing30 + curated Jamendo Lyrics subset, vocal-isolated via Demucs htdemucs_ft, song-disjoint train/val/test splits.
|
|
|
| ## Limitations
|
|
|
| - German only (training data was German).
|
| - Heavy distortion / extreme growl vocals are still hard.
|
| - Best results require vocal isolation as a preprocessing step.
|
|
|
| ## Citation
|
|
|
| ```
|
| @misc{autolyrics2026,
|
| author = { ramduvvuri },
|
| title = {AUTOLYRICS: LoRA Fine-tuning of Whisper for Singing Lyrics},
|
| year = {2026},
|
| howpublished = {\url{https://github.com/ramduvvuri/autolyrics}}
|
| }
|
| ```
|
|
|
