Use markdown PNG images in model card

deb594e verified 1 day ago

18 kB

	---
	library_name: nemo
	pipeline_tag: automatic-speech-recognition
	license: apache-2.0
	language:
	- hi
	- bn
	- ta
	- te
	- kn
	- ml
	- mr
	- gu
	- pa
	- or
	- en
	tags:
	- automatic-speech-recognition
	- speech-recognition
	- fastconformer
	- nemo
	- indic
	- multilingual
	- telephony
	- call-center
	- zero-shot-language-identification
	- rnn-t
	datasets:
	- ai4bharat/IndicVoices
	- ARTPARK-IISc/Vaani
	metrics:
	- wer
	- cer
	model-index:
	- name: VaaniCall
	results:
	- task:
	type: automatic-speech-recognition
	name: Automatic Speech Recognition
	dataset:
	name: AI4Bharat IndicVoices (telephonic)
	type: ai4bharat/IndicVoices
	split: test
	metrics:
	- type: wer
	value: 24.06
	name: WER
	- type: cer
	value: 8.26
	name: CER
	- task:
	type: automatic-speech-recognition
	name: Automatic Speech Recognition
	dataset:
	name: AI4Bharat IndicVoices (clean)
	type: ai4bharat/IndicVoices
	split: test
	metrics:
	- type: wer
	value: 24.20
	name: WER
	- type: cer
	value: 8.31
	name: CER
	- task:
	type: automatic-speech-recognition
	name: Automatic Speech Recognition
	dataset:
	name: Google FLEURS (clean)
	type: google/fleurs
	split: test
	metrics:
	- type: wer
	value: 23.52
	name: WER
	- type: cer
	value: 8.34
	name: CER
	---

	![VaaniCall — multilingual telephony ASR for India](assets/banner.png)

	<br>

	# VaaniCall

	Call-grade speech recognition for 11 Indian languages.
	By [Tausif Iqbal](https://huggingface.co/TieIncred). Built on NVIDIA NeMo FastConformer, fine-tuned from VAANI for PSTN, VoIP, and contact-center audio.

	VaaniCall transcribes a phone call without a language code. It infers language and script, and it is trained through a 21-stage telephony pipeline so AMR, packet loss, and handset noise look like training data — not a domain shift.

	![Supported languages: Hindi, Bengali, Tamil, Telugu, Kannada, Malayalam, Marathi, Gujarati, Punjabi, Odia, English](assets/languages.png)

	---

	## Why VaaniCall

	\| \| \|
	\|---\|---\|
	\| Zero-shot language ID \| No `lang=` argument. One model, eleven languages, native scripts. \|
	\| Telephony-hardened \| 21-stage GPU augmentation: codecs, burst loss, VAD bite, handset EQ, room IR. \|
	\| Holds on clean audio \| Phone fine-tune does not tax clean WER (24.20% → 24.06% on IndicVoices). \|
	\| Beats Whisper on Indic \| 3.5× lower overall WER than Whisper large-v3 on IndicVoices. \|
	\| FastConformer + RNNT \| Streaming-friendly NVIDIA NeMo `EncDecRNNTBPE` checkpoint (`.nemo`). \|

	---

	## Performance at a glance

	All numbers are Word Error Rate (lower is better). Evaluations use 500 clips per language.

	![Overall WER: VaaniCall vs Whisper large-v3 on IndicVoices and FLEURS](assets/wer_overall_vs_whisper.png)

	\| Benchmark \| Condition \| VaaniCall \| Whisper large-v3 \| Relative WER cut \|
	\|---\|---\|---:\|---:\|---:\|
	\| IndicVoices \| Clean \| 24.20% \| 84.07% \| 71% \|
	\| IndicVoices \| Telephonic \| 24.06% \| 84.50% \| 72% \|
	\| Google FLEURS \| Clean \| 23.52% \| 65.23% \| 64% \|

	Whisper large-v3 still leads on English. VaaniCall is the Indic-language model.

	![VaaniCall WER stays flat from clean to telephonic audio](assets/telephony_robustness.png)

	---

	## Benchmarks

	### vs Whisper large-v3 — IndicVoices, telephonic

	![Per-language telephonic WER vs Whisper large-v3](assets/wer_indicvoices_telephonic.png)

	<details>
	<summary><b>IndicVoices · telephonic · full table</b></summary>

	\| Language \| N \| VaaniCall WER ↓ \| VaaniCall CER ↓ \| Whisper WER \| Whisper CER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 15.41% \| 4.93% \| 73.87% \| 38.04% \|
	\| English \| 500 \| 13.26% \| 4.56% \| 7.80% \| 3.04% \|
	\| Gujarati \| 500 \| 21.46% \| 7.56% \| 61.48% \| 29.72% \|
	\| Hindi \| 500 \| 16.04% \| 6.87% \| 30.31% \| 15.61% \|
	\| Kannada \| 500 \| 38.01% \| 10.96% \| 98.03% \| 53.05% \|
	\| Malayalam \| 500 \| 40.70% \| 11.53% \| 138.83% \| 108.41% \|
	\| Marathi \| 500 \| 17.10% \| 6.30% \| 93.50% \| 47.21% \|
	\| Odia \| 500 \| 31.52% \| 10.40% \| 110.63% \| 97.36% \|
	\| Punjabi \| 500 \| 14.46% \| 6.03% \| 86.10% \| 54.26% \|
	\| Tamil \| 500 \| 33.82% \| 9.84% \| 80.72% \| 40.71% \|
	\| Telugu \| 500 \| 28.45% \| 8.63% \| 146.64% \| 97.12% \|
	\| Overall \| 5500 \| 24.06% \| 8.26% \| 84.50% \| 56.30% \|

	</details>

	<details>
	<summary><b>IndicVoices · clean · full table</b></summary>

	\| Language \| N \| VaaniCall WER ↓ \| VaaniCall CER ↓ \| Whisper WER \| Whisper CER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 15.16% \| 4.96% \| 73.36% \| 36.81% \|
	\| English \| 500 \| 14.14% \| 4.67% \| 8.37% \| 3.12% \|
	\| Gujarati \| 500 \| 20.02% \| 6.68% \| 56.27% \| 25.96% \|
	\| Hindi \| 500 \| 16.45% \| 6.77% \| 30.71% \| 15.20% \|
	\| Kannada \| 500 \| 38.26% \| 11.02% \| 99.44% \| 52.16% \|
	\| Malayalam \| 500 \| 41.79% \| 12.11% \| 145.50% \| 114.81% \|
	\| Marathi \| 500 \| 16.76% \| 5.80% \| 92.38% \| 46.12% \|
	\| Odia \| 500 \| 31.85% \| 10.63% \| 107.96% \| 95.15% \|
	\| Punjabi \| 500 \| 15.71% \| 6.59% \| 89.11% \| 58.95% \|
	\| Tamil \| 500 \| 34.50% \| 10.22% \| 77.39% \| 38.11% \|
	\| Telugu \| 500 \| 28.39% \| 8.73% \| 141.52% \| 96.52% \|
	\| Overall \| 5500 \| 24.20% \| 8.31% \| 84.07% \| 56.30% \|

	</details>

	### vs Whisper large-v3 — Google FLEURS (out of domain)

	Held-out read speech. Odia is not in FLEURS.

	![FLEURS out-of-domain WER vs Whisper large-v3](assets/wer_fleurs.png)

	<details>
	<summary><b>FLEURS · clean · full table</b></summary>

	\| Language \| N \| VaaniCall WER ↓ \| VaaniCall CER ↓ \| Whisper WER \| Whisper CER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 26.00% \| 9.22% \| 77.60% \| 32.63% \|
	\| English \| 500 \| 15.16% \| 7.88% \| 4.47% \| 1.78% \|
	\| Gujarati \| 500 \| 24.94% \| 8.22% \| 54.99% \| 22.19% \|
	\| Hindi \| 500 \| 15.70% \| 6.18% \| 27.61% \| 9.63% \|
	\| Kannada \| 500 \| 25.80% \| 7.95% \| 72.85% \| 22.87% \|
	\| Malayalam \| 500 \| 37.55% \| 10.36% \| 130.76% \| 102.00% \|
	\| Marathi \| 500 \| 22.32% \| 7.18% \| 76.52% \| 22.61% \|
	\| Punjabi \| 500 \| 16.53% \| 6.06% \| 74.63% \| 35.88% \|
	\| Tamil \| 500 \| 32.78% \| 9.84% \| 47.73% \| 12.11% \|
	\| Telugu \| 500 \| 29.19% \| 9.99% \| 126.29% \| 74.64% \|
	\| Overall \| 5000 \| 23.52% \| 8.34% \| 65.23% \| 34.04% \|

	> Tamil CER is lower for Whisper (12.11%) despite a higher WER — typically an artifact of shorter predicted strings. VaaniCall still wins WER.

	</details>

	### Per-language profile

	![VaaniCall WER by language on telephonic IndicVoices](assets/wer_by_language.png)

	Strongest Indic languages on phone audio: Punjabi 14.5% · Bengali 15.4% · Hindi 16.0% · Marathi 17.1%.

	### vs VAANI base — the telephony fine-tune

	VaaniCall is a telephony specialist on top of VAANI. On phone-channel audio it improves 9 of 10 Indic languages. On clean audio the two models are statistically tied (24.37% vs 24.38% WER).

	![WER reduction versus VAANI base under telephonic conditions](assets/delta_vs_vaani_base.png)

	<details>
	<summary><b>vs VAANI base · telephonic</b></summary>

	\| Language \| N \| VaaniCall WER ↓ \| VAANI WER \| ΔWER \| VaaniCall CER ↓ \| VAANI CER \| ΔCER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 19.85% \| 21.70% \| +1.85 \| 7.74% \| 9.36% \| +1.62 \|
	\| Gujarati \| 500 \| 26.27% \| 28.97% \| +2.70 \| 10.72% \| 11.50% \| +0.78 \|
	\| Hindi \| 500 \| 19.65% \| 20.28% \| +0.63 \| 8.92% \| 9.22% \| +0.30 \|
	\| Kannada \| 500 \| 44.44% \| 47.03% \| +2.59 \| 14.93% \| 15.73% \| +0.80 \|
	\| Malayalam \| 500 \| 47.67% \| 48.51% \| +0.84 \| 16.46% \| 16.79% \| +0.33 \|
	\| Marathi \| 500 \| 23.65% \| 25.12% \| +1.46 \| 9.29% \| 10.02% \| +0.73 \|
	\| Odia \| 500 \| 35.93% \| 34.56% \| −1.37 \| 13.65% \| 13.28% \| −0.37 \|
	\| Punjabi \| 500 \| 20.26% \| 20.78% \| +0.51 \| 9.50% \| 9.71% \| +0.21 \|
	\| Tamil \| 500 \| 40.88% \| 41.84% \| +0.96 \| 14.69% \| 15.10% \| +0.41 \|
	\| Telugu \| 500 \| 32.17% \| 33.04% \| +0.88 \| 11.41% \| 11.75% \| +0.34 \|
	\| Overall \| 5000 \| 29.96% \| 31.04% \| +1.08 \| 11.96% \| 12.46% \| +0.51 \|

	Δ = VAANI − VaaniCall. Positive Δ → VaaniCall is better. English excluded (not a VAANI fine-tune target).

	</details>

	<details>
	<summary><b>vs VAANI base · clean</b></summary>

	\| Language \| N \| VaaniCall WER \| VAANI WER \| ΔWER \| VaaniCall CER \| VAANI CER \| ΔCER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 14.94% \| 15.07% \| +0.13 \| 5.01% \| 4.97% \| −0.04 \|
	\| Gujarati \| 500 \| 21.11% \| 21.93% \| +0.82 \| 7.38% \| 7.61% \| +0.24 \|
	\| Hindi \| 500 \| 16.47% \| 16.39% \| −0.08 \| 6.78% \| 6.75% \| −0.03 \|
	\| Kannada \| 500 \| 37.91% \| 38.78% \| +0.87 \| 10.99% \| 11.16% \| +0.17 \|
	\| Malayalam \| 500 \| 40.37% \| 40.63% \| +0.26 \| 11.46% \| 11.60% \| +0.14 \|
	\| Marathi \| 500 \| 17.05% \| 17.01% \| −0.04 \| 5.38% \| 5.33% \| −0.05 \|
	\| Odia \| 500 \| 31.18% \| 28.49% \| −2.69 \| 9.68% \| 9.15% \| −0.53 \|
	\| Punjabi \| 500 \| 14.17% \| 14.38% \| +0.21 \| 5.93% \| 5.86% \| −0.07 \|
	\| Tamil \| 500 \| 34.26% \| 34.49% \| +0.24 \| 9.82% \| 9.87% \| +0.05 \|
	\| Telugu \| 500 \| 27.12% \| 27.55% \| +0.43 \| 7.95% \| 8.07% \| +0.12 \|
	\| Overall \| 5000 \| 24.37% \| 24.38% \| +0.01 \| 8.17% \| 8.18% \| +0.01 \|

	Results are statistically equivalent on clean audio — the fine-tune does not spend general accuracy to buy telephony robustness.

	</details>

	### vs AI4Bharat IndicConformer

	IndicConformer is a strong general-purpose Indic ASR and still leads this comparison. VaaniCall is optimized for call-channel robustness and zero-shot language ID, not for matching a dedicated per-language conformer on clean read speech.

	\| Condition \| VaaniCall \| IndicConformer \| Gap \|
	\|---\|---:\|---:\|---:\|
	\| Telephonic \| 29.38% \| 26.11% \| −3.26 pp \|
	\| Clean \| 24.56% \| 20.63% \| −3.93 pp \|

	Largest telephonic gaps: Malayalam −5.61 pp, Telugu −5.49 pp. English excluded.

	<details>
	<summary><b>vs IndicConformer · telephonic</b></summary>

	\| Language \| N \| VaaniCall WER \| IndicConformer WER ↓ \| ΔWER \| VaaniCall CER \| IndicConformer CER ↓ \| ΔCER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 18.99% \| 16.81% \| −2.18 \| 7.29% \| 6.65% \| −0.63 \|
	\| Gujarati \| 500 \| 25.31% \| 22.48% \| −2.83 \| 9.89% \| 9.46% \| −0.43 \|
	\| Hindi \| 500 \| 19.61% \| 17.27% \| −2.34 \| 9.25% \| 7.86% \| −1.40 \|
	\| Kannada \| 500 \| 44.50% \| 43.22% \| −1.28 \| 15.24% \| 13.92% \| −1.33 \|
	\| Malayalam \| 500 \| 45.91% \| 40.30% \| −5.61 \| 15.74% \| 12.75% \| −2.99 \|
	\| Marathi \| 500 \| 22.31% \| 18.78% \| −3.53 \| 8.27% \| 7.09% \| −1.18 \|
	\| Odia \| 500 \| 36.38% \| 32.34% \| −4.04 \| 13.86% \| 12.42% \| −1.44 \|
	\| Punjabi \| 500 \| 19.16% \| 16.48% \| −2.68 \| 8.75% \| 8.74% \| −0.01 \|
	\| Tamil \| 500 \| 38.82% \| 35.30% \| −3.52 \| 13.53% \| 11.34% \| −2.18 \|
	\| Telugu \| 500 \| 34.36% \| 28.87% \| −5.49 \| 12.52% \| 10.10% \| −2.41 \|
	\| Overall \| 5000 \| 29.38% \| 26.11% \| −3.26 \| 11.62% \| 10.15% \| −1.47 \|

	</details>

	<details>
	<summary><b>vs IndicConformer · clean</b></summary>

	\| Language \| N \| VaaniCall WER \| IndicConformer WER ↓ \| ΔWER \| VaaniCall CER \| IndicConformer CER ↓ \| ΔCER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Bengali \| 500 \| 15.42% \| 12.79% \| −2.64 \| 4.96% \| 4.00% \| −0.95 \|
	\| Gujarati \| 500 \| 20.37% \| 15.24% \| −5.13 \| 7.09% \| 5.11% \| −1.98 \|
	\| Hindi \| 500 \| 15.97% \| 14.23% \| −1.74 \| 6.58% \| 5.68% \| −0.89 \|
	\| Kannada \| 500 \| 37.67% \| 33.94% \| −3.73 \| 11.10% \| 9.40% \| −1.70 \|
	\| Malayalam \| 500 \| 41.67% \| 35.23% \| −6.43 \| 12.06% \| 9.67% \| −2.39 \|
	\| Marathi \| 500 \| 16.08% \| 13.02% \| −3.05 \| 5.26% \| 4.28% \| −0.99 \|
	\| Odia \| 500 \| 30.82% \| 25.86% \| −4.96 \| 9.86% \| 8.01% \| −1.85 \|
	\| Punjabi \| 500 \| 15.00% \| 10.65% \| −4.35 \| 6.48% \| 4.21% \| −2.27 \|
	\| Tamil \| 500 \| 34.76% \| 30.35% \| −4.41 \| 10.31% \| 8.52% \| −1.79 \|
	\| Telugu \| 500 \| 27.90% \| 24.43% \| −3.47 \| 8.49% \| 6.92% \| −1.57 \|
	\| Overall \| 5000 \| 24.56% \| 20.63% \| −3.93 \| 8.40% \| 6.74% \| −1.66 \|

	</details>

	---

	## How it works

	![Phone audio into FastConformer encoder, RNNT decoder, transcript with no language ID](assets/architecture.png)

	### Training

	Two-phase fine-tune from the VAANI FastConformer checkpoint.

	![Phase 1 language alignment, phase 2 acoustic refinement](assets/training.png)

	1. Language alignment — encoder frozen; decoder learns script and language mapping.
	2. Acoustic refinement — last five encoder layers plus the full decoder unfrozen for telephonic acoustics.

	### Data

	Trained on [AI4Bharat IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices) and [VAANI](https://huggingface.co/datasets/ARTPARK-IISc/Vaani): 50,000 train / 5,000 val / 5,000 test clips per language (mean duration ~18 s).

	### Telephony augmentation

	Every training batch can pass through a 21-stage, GPU-accelerated call-channel simulator.

	![Talker, handset, codec, network, and line stages of the augmentation pipeline](assets/pipeline.png)

	<details>
	<summary><b>All 21 stages</b></summary>

	\| # \| Stage \| Prob \| What it does \|
	\|---\|---\|---\|---\|
	\| 0 \| Identity setup \| 100% \| Tier: Premium / Standard / Legacy. Locks mic profile and codec behavior. \|
	\| 1 \| Formant shift \| 25% \| Vocal-tract warp (0.95–1.05×) so the model does not memorize speakers. \|
	\| 2 \| Speed perturbation \| 30% \| Tempo + pitch together (0.9–1.1×). \|
	\| 3 \| Room IR \| 50% \| GPU conv1d reverb, partial RMS match, up to 20 ms pre-delay. \|
	\| 4 \| Background noise \| 90% \| WHAM! floor (15–25 dB SNR) + MUSAN chatter (40%, 10–20 dB). \|
	\| 5 \| Mic profile \| 100% \| Flat / mid / low / muffled EQ + tanh saturation. \|
	\| 6 \| Mic plosives \| 35% \| Breath thumps on transients, 20–50 Hz. \|
	\| 7 \| Handling noise \| 45% \| Handset friction, lowpass @ 2.5 kHz, 25–35 dB SNR. \|
	\| 8 \| Vocal EQ \| 75% \| Presence peak 900–1600 Hz; optional nasal / low-mid dip. \|
	\| 9 \| Volume drift \| 20% \| Interpolated gain envelope, ±3 dB. \|
	\| 10 \| Pre-codec AGC \| 85% \| Noise-aware rolling-RMS AGC, gain capped at 3.5×. \|
	\| 11 \| Codec roundtrip \| 100% \| pcm_alaw 35% / amrnb 35% / amrwb 15% / opus 10% / pcm_mulaw 5%. \|
	\| 12 \| Loudness recovery \| 100% \| Peak targets: Premium −8 / Standard −4 / Legacy −1 dBFS. \|
	\| 13 \| Gateway VAD clipping \| 35% \| IVR VAD bite: 15–45 ms off 40% of onsets. \|
	\| 14 \| Network loss \| 40% \| Gilbert–Elliott burst loss, 20 ms packets, optional PLC smear. \|
	\| 15 \| Stutter / jitter \| 25% \| 5–40 ms micro-dropouts to 0–10% amplitude. \|
	\| 16 \| Clock drift \| 15% \| Per-100 ms resample warp 0.998–1.002×. \|
	\| 17 \| IMD distortion \| 35% \| Quadratic + cubic harmonics. \|
	\| 18 \| Soft clipping \| 70% \| Handset tanh above 0.5–0.85. \|
	\| 19 \| Receiver hiss \| 40–85% \| Hardware floor from −85 to −54 dBFS by tier. \|
	\| 20 \| Filter mode \| 100% \| Static 75% / dynamic 25%; NB or WB band limits. \|
	\| 21 \| Mastering \| 100% \| Gain, smart nudge, tanh, brick-wall at ±0.98. \|

	</details>

	---

	## Quick start

	```python
	from nemo.collections.asr.models import EncDecRNNTBPEModel

	model = EncDecRNNTBPEModel.from_pretrained(
	model_name="TieIncred/VaaniCall"
	)

	hypotheses = model.transcribe(["sample.wav"], return_hypotheses=True)
	print(hypotheses[0].text)
	```

	No language flag. 16 kHz mono WAV is the happy path; typical telephony (8 kHz, μ-law / AMR) is the training domain.

	---

	## Intended use

	- Inbound / outbound contact-center transcription in Indic languages
	- VoIP and PSTN call analytics
	- Multilingual IVR and voice-bot post-processing
	- Offline batch transcription of call recordings

	Not intended for: medical or legal dictation, real-time emergency dispatch, or English-only broadcast ASR (use Whisper or a dedicated English model).

	---

	## Limitations

	- English trails Whisper large-v3 (expected: this checkpoint is Indic-first).
	- Malayalam, Kannada, Tamil remain the hardest languages (WER 34–41% on IndicVoices).
	- Odia slightly regresses vs the VAANI base under telephony (−1.37 pp).
	- IndicConformer is still stronger as a general Indic ASR.
	- Code-mixed utterances and heavy music-on-hold are not separately benchmarked.
	- Evaluations are 500 clips / language; treat per-language gaps of <1 pp as noise.

	---

	## Citation

	If you use VaaniCall, please cite this model and VAANI, the base the fine-tune starts from:

	```bibtex
	@misc{iqbal2026vaanicall,
	title={VaaniCall: Multilingual Telephony ASR for Indic Languages},
	author={Iqbal, Tausif},
	year={2026},
	howpublished={Hugging Face},
	url={https://huggingface.co/TieIncred/VaaniCall}
	}
	```

	```bibtex
	@misc{pulikodan2026vaanicapturinglanguagelandscape,
	title={VAANI: Capturing the language landscape for an inclusive digital India},
	author={Sujith Pulikodan and Abhayjeet Singh and Agneedh Basu and Nihar Desai and Pavan Kumar J and Pranav D Bhat and Raghu Dharmaraju and Ritika Gupta and Sathvik Udupa and Saurabh Kumar and Sumit Sharma and Vaibhav Vishwakarma and Visruth Sanka and Dinesh Tewari and Harsh Dhand and Amrita Kamat and Sukhwinder Singh and Shikhar Vashishth and Partha Talukdar and Raj Acharya and Prasanta Kumar Ghosh},
	year={2026},
	eprint={2603.28714},
	archivePrefix={arXiv},
	primaryClass={eess.AS},
	url={https://arxiv.org/abs/2603.28714}
	}
	```

	Also cite [IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices) when reporting numbers on that test set.

	---

	<p align="center">
	<em>VaaniCall · Tausif Iqbal · Apache 2.0</em>
	</p>