Noobbbbb/whisper-small-bn-v2

Whisper Small — Bengali ASR (fine-tuned)

This model is a fine-tuned version of Noobbbbb/whisper-small-bn on the Noobbbbb/bengali-ai-asr-80k dataset, for automatic speech recognition (ASR) in Bengali.

Model Details

• Base model: Noobbbbb/whisper-small-bn (Whisper-small architecture, previously fine-tuned for Bengali)
• Architecture: Whisper small (12 encoder layers, 12 decoder layers, 768 hidden size)
• Language: Bengali (bn)
• Task: transcribe
• Fine-tuning framework: 🤗 transformers Seq2SeqTrainer

Training Data

• Dataset: Noobbbbb/bengali-ai-asr-80k
• Train subset used: first 40,000 examples of the train split
• Validation subset used: first 4,000 examples of the val split
• Audio was processed via WhisperProcessor (log-mel spectrogram input features); transcripts were tokenized with the Whisper tokenizer.

Training Procedure

Hyperparameter	Value
Epochs	3
Per-device train batch size	64
Per-device eval batch size	32
Gradient accumulation steps	1
Learning rate	1e-5
Warmup steps	50
Precision	fp16 (mixed precision)
Gradient checkpointing	enabled
Optimizer	adamw_torch_fused
Generation max length	225
Eval strategy	every 400 steps

Training Results — Stage 1 (initial fine-tune)

• Final training loss: 0.2969
• Total steps: 1,875 (3 epochs)
• Train runtime: 6,500s (108 minutes)
• Throughput: ~18.5 samples/sec

Evaluation used Word Error Rate (WER) and Character Error Rate (CER) via the evaluate library (wer_metric / cer_metric), computed every 400 steps on the 4,000-example validation subset.

Step	Training Loss	Validation Loss	WER	CER
400	0.3408	0.1606	0.5021	0.1601
800	0.2654	0.1340	0.4438	0.1360
1200	0.2472	0.1204	0.4193	0.1273
1600	0.2179	0.1159	0.4101	0.1261
1875 (final)	0.2159	0.1144	0.4075	0.1262

Stage 1 final validation WER: 40.75% · CER: 12.62%

Training Results — Stage 2 (extended fine-tune, +5 epochs)

Training was continued for 5 additional epochs from the Stage 1 checkpoint.

• Final training loss: 0.1609
• Total steps: 6,645 (5 epochs this stage)
• Train runtime: 17,215s (4.78 hours)
• Throughput: ~24.7 samples/sec

Stage 2 out-of-domain evaluation:

Evaluated on google/fleurs (bn_in config)

Metric	Value
WER	38.90%
CER	13.98%

This was run on an out-of-domain evaluation set (i.e. not the in-domain validation subset used for Stage 1 tracking). Compared to Stage 1's in-domain validation numbers, WER improved (40.75% → 38.90%) while CER is slightly higher (12.62% → 13.98%) — consistent with evaluating on different, harder data rather than a straightforward regression.

Sample Predictions (Stage 2)

Prediction	Reference
একটি দল কৃষি যন্ত্রপাত্ত্তী তৈরী করে এবং অন্য দলটি, আলংকারিক এবং ট্রুটেমিও স্তম্ভ তৈরী করে।	একটি দল কৃষি যন্ত্রপাতি তৈরি করে এবং অন্য দলটি আলংকারিক এবং টোটেমীয় স্তম্ভ তৈরি করে।
জবরদস্থক খান বিদ্রোহীকে পরাভূত করেন।	জবরদস্ত খান বিদ্রোহীকে পরাভূত করেন।
তিনিই বামলা চলচ্চিত্র অবশেষে তার কণ্ঠস্বর প্রদানের জন্য জাতীয় চলচ্চিত্র পুরস্কার জিতে নেন।	তিনি বাংলা চলচ্চিত্র "অবশেষে" তার কণ্ঠস্বর প্রদানের জন্য জাতীয় চলচ্চিত্র পুরস্কার জিতে নেন।

Qualitatively, Stage 2 predictions track the reference more closely than Stage 1 (fewer inserted/garbled syllables), though small substitution and diacritic errors remain.

How to Use

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch, librosa

repo_id = "Noobbbbb/bengali-ai-asr-80k"  # update if you rename the repo

processor = WhisperProcessor.from_pretrained(repo_id)
model = WhisperForConditionalGeneration.from_pretrained(repo_id)

model.generation_config.language = "bengali"
model.generation_config.task = "transcribe"

waveform, sr = librosa.load("audio.wav", sr=16000, mono=True)
inputs = processor(waveform, sampling_rate=16000, return_tensors="pt")

with torch.no_grad():
    predicted_ids = model.generate(inputs["input_features"])

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

For longer audio

from transformers import pipeline, WhisperForConditionalGeneration, WhisperProcessor, AutoModelForSpeechSeq2Seq

model_id = "Noobbbbb/whisper-small-bn-v2"

model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)

if isinstance(model.generation_config.eos_token_id, list):
    model.generation_config.eos_token_id = model.generation_config.eos_token_id[0]

if isinstance(model.config.eos_token_id, list):
    model.config.eos_token_id = model.config.eos_token_id[0]

processor = WhisperProcessor.from_pretrained(model_id)

waveform, _ = librosa.load("/content/yt.mp3")

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    device=0 if torch.cuda.is_available() else -1,
)

result = pipe(
    waveform,
    return_timestamps=True,   # enables long-form transcription
    generate_kwargs={"language": "bengali"}
)

Note: This will return timestamp-wise transcription

Limitations

• Stage 1 was fine-tuned only on a 40k-example subset of the full dataset; Stage 2's exact train subset size is unconfirmed (see note above).
• Stage 1 metrics (WER 40.75%, CER 12.62%) are in-domain validation numbers; Stage 2 metrics (WER 38.90%, CER 13.98%) are from an out-of-domain evaluation set, so the two aren't directly comparable apples-to-apples.
• Performance on noisy/telephony-style audio (e.g. call center recordings) has not been separately validated for this checkpoint.

Citation / Acknowledgements

• Base model: Noobbbbb/whisper-small-bn
• Dataset: Noobbbbb/bengali-ai-asr-80k
• Built on OpenAI's Whisper architecture via 🤗 Transformers.