File size: 3,068 Bytes
3a897f3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | ---
language:
- hi
license: apache-2.0
base_model: openai/whisper-small
tags:
- automatic-speech-recognition
- hindi
- whisper
- fine-tuned
- conversational-speech
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---
# 🎙️ VaaniAI — Whisper-small Fine-tuned on Hindi Conversational Speech
Fine-tuned version of `openai/whisper-small` on real-world Hindi conversational audio
collected across **102 speakers** from India, as part of an AI Researcher Intern
assignment at **Josh Talks**.
---
## 📊 Model Performance
| Metric | Value |
|--------|-------|
| Baseline WER (Whisper-small) | 1.2537 |
| Fine-tuned WER | **0.4028** |
| WER Improvement | ↓ **67.8%** |
| Post-processing WER gain | ↓ additional **27.7%** |
---
## 🗂️ Training Data
| Property | Value |
|----------|-------|
| Total audio | 11.44 hours |
| Speakers | 102 unique speakers across India |
| Segments (after cleaning) | 4,442 |
| Raw segments | 5,941 |
| Train / Val split | 4,093 / 349 |
**Cleaning steps applied:**
- Removed 209 REDACTED-label segments
- Removed 1,012 sub-1-second clips
- Removed 878 segments with fewer than 5 characters
- Resampled all audio from 44,100 Hz → 16,000 Hz
---
## ⚙️ Training Configuration
| Hyperparameter | Value |
|----------------|-------|
| Base model | openai/whisper-small (241.7M params) |
| Learning rate | 1e-5 |
| Effective batch size | 32 (batch 4 × grad accum 8) |
| Epochs | 3 |
| Precision | FP16 |
| Hardware | Kaggle T4 GPU (14.6 GB) |
**Training loss progression:**
| Epoch | Train Loss | Val Loss | WER |
|-------|-----------|----------|-----|
| 1 | 13.22 | 0.657 | 0.546 |
| 2 | 6.98 | 0.471 | 0.435 |
| 3 | 5.07 | 0.414 | **0.403** |
---
## 🧹 Post-processing Pipeline
**1. Repetition Loop Detection**
Collapses tokens repeated 4+ times — targets hallucination on noisy audio.
Example: `आ आ आ... (100x)` → `आ`
**2. Spelling Normalization Dictionary**
Maps common dialectal Hindi variants to standard spellings.
Example: `वगैरा` → `वगैरह`, `इदर` → `इधर`
---
## 🔍 Error Analysis (25 sampled validation errors)
| Error Type | Count | % |
|------------|-------|---|
| Phonetic Confusion | 10 | 40% |
| Spelling Variation | 7 | 28% |
| English Loanword Error | 4 | 16% |
| Filler Word Confusion | 3 | 12% |
| Hallucination / Repetition | 1 | 4% |
---
## ⚖️ Evaluation: Lattice-Based WER
Implemented a **multi-alternative bin-based lattice** where each position accepts all valid alternatives (numeric, synonymous, dialectal) for fairer evaluation.
---
## ⚠️ Limitations
- Optimized for conversational Hindi; may underperform on formal/broadcast speech
- English loanword transcription remains a known weak point
- Model weights not released due to proprietary training data (Josh Talks internal dataset)
---
## 🔗 Links
- 💻 GitHub: [Daksh159/VaaniAI](https://github.com/Daksh159/VaaniAI)
- 📓 Kaggle Notebook: [josh-talks-q1-preprocessing](https://www.kaggle.com/code/daksh159/josh-talks-q1-preprocessing) |