Blgn94/whisper-small-mn-v3

whisper-small-mn-v25

Fine-tune of openai/whisper-small for Mongolian (Cyrillic Khalkha) speech-to-text. This is the v25 run — the winning configuration of a controlled A/B against a plain fine-tuning baseline, selected on a text-disjoint evaluation set (transcripts that never appear in training) rather than the optimistic in-domain split.

• Base model: openai/whisper-small (244M params)
• Language: Mongolian, Cyrillic script (mn)
• Dataset: Blgn94/mongolian-stt-dataset-v24 (~148 h of audio)
• Honest clean WER: 21.33 (raw) / 19.67 (normalized)

Results

All numbers are Word/Character Error Rate (lower is better). Normalized = lowercased, punctuation stripped, NFC-normalized Cyrillic (ё/й) — this is what published benchmarks use, so compare against the _norm columns.

Three evaluation sets are reported:

eval set	what it is	trust
clean	dev clips whose transcripts never appear in training (241 clips)	the honest metric — model was selected on this
indomain	old random split; ~89% of its transcripts also appear in training (300 clips)	optimistic (memorization), kept for continuity
mic	10 clips recorded on a laptop mic — real, noisy, out-of-distribution	reported only, too small to trust

v25 final metrics

eval set	WER	CER	WER (norm)	CER (norm)
clean	21.33	9.78	19.67	9.13
indomain	23.01	8.40	21.04	7.92
mic	84.36	48.66	72.51	44.87

Per-source normalized WER on the clean set: mbspeech 6.03, fleurs 21.67, youtube 57.65 (spontaneous speech is the weak spot — see Limitations).

A/B vs baseline (the reason this recipe was chosen)

Both runs share the identical text-disjoint clean eval set, so the comparison is fair.

metric (normalized)	baseline	v25	Δ
clean WER (selection metric)	22.85	19.67	−3.18
clean CER	9.82	9.13	−0.69
clean — mbspeech	8.18	6.03	−2.14
clean — youtube (spontaneous)	66.15	57.65	−8.50
mic (real-world)	87.68	72.51	−15.17
indomain (optimistic)	17.07	21.04	+3.97

The baseline is a plain fine-tune on the full corpus with no de-duplication or augmentation. v25 improves the honest clean WER by 3.2 points and generalizes markedly better on spontaneous YouTube speech and on real microphone recordings. The in-domain score rises — this is expected and desirable: capping duplicate transcripts removes the memorization crutch (the corpus repeats each Common Voice sentence ~14.5×), so the optimistic in-domain number falls while genuine transcription of unseen speech improves.

Training data

Blgn94/mongolian-stt-dataset-v24 — ~148 h of 16 kHz Mongolian Cyrillic speech across four sources (measured by decoding all clips):

source	~hours	notes
Common Voice	~126	read speech; only 6,000 distinct sentences (14.5× text reuse)
FLEURS	~14	read speech
MBSpeech	~7	read speech
YouTube	~1	manual-subtitle clips — the only spontaneous speech

Known weakness: the corpus is 85% Common Voice read speech and very light on spontaneous speech (1 h). Read-speech sources score in the single-to-low-twenties WER; spontaneous YouTube sits near 58%. Expect the model to be strongest on clear, read-style Mongolian and weaker on conversational/noisy audio.

v25 corpus shaping (what makes this run different)

• Per-sentence cap = 4: at most 4 recordings per distinct transcript, dropping 66.5% of rows (95,946 → 32,134) and cutting Common Voice text reuse from 14.5× to 4×. Breaks the transcript-memorization shortcut.
• Source-balanced sampling (α = 0.5): re-weights so Common Voice does not drown out FLEURS, MBSpeech, and YouTube.
• Audio augmentation (train only; eval always clean):
  • SpecAugment (time masking, p = 0.05)
  • Waveform augmentation (p = 0.5, up to 2 ops): speed 0.9–1.1×, gain, additive noise at 10–30 dB SNR, light reverb.

Training procedure

Base model	openai/whisper-small
Precision	bf16
Effective batch	16 (batch 4 × grad-accum 4)
Learning rate	1e-5, linear schedule, warmup ratio 0.05
Weight decay	0.01
Label smoothing	0.0
Epochs	4 (8,036 steps)
Max label tokens	440
Seed	42
Hardware	1× NVIDIA RTX 5060 (8 GB, Blackwell)
Wall-clock	~2 h 53 m
transformers / torch	5.14.1 / 2.13.0+cu130

Model selection: best checkpoint by eval_clean_wer_norm (load_best_model_at_end). Labels drop the leading <|startoftranscript|> token (the model re-prepends decoder_start_token_id).

Reproduce

MAX_PER_SENTENCE=4 SOURCE_ALPHA=0.5 AUG_PROB=0.5 EPOCHS=4 OUT=out_v25 \
  python scripts/train.py

Full config and final metrics are in run_manifest.json in this repo.

Usage

import torch
from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="Blgn94/whisper-small-mn-v25",
    device=0 if torch.cuda.is_available() else -1,
)
result = pipe(
    "audio.mp3",
    generate_kwargs={"language": "mn", "task": "transcribe"},
)
print(result["text"])

For long audio, add chunk_length_s=30. For raw model + processor access:

from transformers import WhisperForConditionalGeneration, WhisperProcessor

processor = WhisperProcessor.from_pretrained("Blgn94/whisper-small-mn-v25")
model = WhisperForConditionalGeneration.from_pretrained("Blgn94/whisper-small-mn-v25")

Limitations

• Spontaneous / conversational speech: weak (~58% WER on YouTube), because the corpus has only ~1 h of it. Real-world microphone input is also harder (mic set ~72% WER, but only 10 clips — treat as directional, not a benchmark).
• Not evaluated on the official Common Voice test split. The clean set is a custom text-disjoint split of v24; it is not speaker-disjoint from training, so real-world WER on entirely new speakers may be somewhat higher. For a speaker-disjoint reference point, bayartsogt/whisper-small-mn-8 reports ~26.5% WER on the official Common Voice mn test.
• Cyrillic Khalkha only. No traditional Mongolian script; not tuned for other dialects.

License

Apache-2.0, inheriting from the openai/whisper-small base model. Note the underlying training data combines sources under their own licenses (Common Voice, FLEURS, MBSpeech, YouTube manual subtitles); review those before commercial use.