aahanlulla18/whisper-medium-sindhi-lora

Whisper Medium fine-tuned for Sindhi (LoRA)

This is a LoRA fine-tune of openai/whisper-medium, adapted for automatic speech recognition (ASR) in Sindhi — a language spoken by a minority community across India and Pakistan, and severely under-represented in existing speech AI training data.

Motivation

Sindhi is one of the most under-resourced languages in Whisper's training data. In testing, stock whisper-medium almost never produces a correct Sindhi transcription — it frequently understands the sounds of Sindhi speech correctly, but writes the output in the wrong script entirely (Devanagari or Romanized text instead of Sindhi's actual script), or collapses into repeating meaningless characters.

This project quantified that failure and tested whether lightweight fine-tuning (LoRA) on a small amount of real Sindhi speech could correct it.

Results

Evaluated on a held-out set of 324 human-validated Sindhi clips from Mozilla Common Voice 26.0, fine-tuned on 200 separate training clips using LoRA (~1.2% of the model's parameters trained):

Metric	Before fine-tuning	After fine-tuning
Word Error Rate (WER)	148.35%	72.97%
Exact word matches	7	819
Insertions (garbled/looping output)	1,250	124
Substitutions	1,791	1,624
Deletions	773	128

Key finding: fine-tuning almost entirely fixed the wrong-script and gibberish-loop failure modes (insertions dropped ~90%), and the model now consistently outputs real Sindhi script. Word-level accuracy (substitutions) improved more modestly — with only 200 training clips, the model learned what Sindhi script looks like far better than it learned precise vocabulary. This is an honest, expected limitation of fine-tuning on a small dataset, not a fully "solved" result.

Training details

• Base model: openai/whisper-medium
• Method: LoRA (r=32, alpha=64, target modules: q_proj, v_proj), ~9.4M trainable parameters out of ~773M total
• Training data: 200 clips from Mozilla Common Voice 26.0 Sindhi (unvalidated split)
• Evaluation data: 324 clips from Mozilla Common Voice 26.0 Sindhi (validated split, human-confirmed)
• Evaluation metric: Word Error Rate (jiwer)

Limitations

• Trained on a small amount of data (200 clips) due to compute constraints; a larger training set would likely improve word-level accuracy further.
• The evaluation set, while human-validated, is relatively small (324 clips / Sindhi's officially validated Common Voice data is limited).
• Not evaluated on speech outside the Common Voice domain (e.g. spontaneous conversational speech, regional dialect variation).

Intended use

Research and experimentation with Sindhi ASR. Not intended for production or safety-critical transcription use without further evaluation.