Language Identification for 42 Indian Languages: Opening New Frontiers in Indic Speech AI

Most speech pipelines start with a routing decision. Audio comes in, something has to decide which language it is, and everything downstream depends on that call being right.

For Indian languages that decision has been hard to make well, because the models that could make it did not cover enough of the map. We are releasing Vaani-LID_v0, an open spoken language classifier that identifies 42 Indian languages across four language families, including many that no other speech model supports.

It runs in four lines:

from transformers import pipeline

pipe = pipeline("audio-classification", model="ARTPARK-IISc/Vaani-LID_v0", trust_remote_code=True)
print(pipe("path/to/16kHz/mono.wav"))

What the dataset had to contain for this study to happen

Figure 1: The experimental setup. We held the architecture fixed so that differences in results come from the encoder or the training objective, not from modeling tricks.

Language identification is the front door of any multilingual speech pipeline. Audio arrives, LID decides which language it is, and the system routes it to the right speech recognition or translation model. Get it wrong and everything downstream is wrong, however good your ASR is.

Doing this for India is hard for reasons that are specifically about data:

• Closely related varieties sound alike. Hindi, Bhojpuri, Chhattisgarhi and Sadri share a lot of phonology and vocabulary. Hindi and Urdu are near-identical when spoken. Separating them needs many examples of each, not a token sample.
• The long tail is thin everywhere else. Most Indian languages have very little labeled audio in public corpora, so any method that needs heavy supervision per language simply cannot be evaluated across 42 of them.
• Corpora do not transfer. Recording conditions and speaker demographics differ between datasets, so in-domain scores flatter models badly.

From Vaani we drew 10 hours of speech for each of the 42 languages, selected to maximize district and speaker diversity, and split it 8:1:1 into train, validation and test with no speaker appearing in more than one split. That last constraint is only possible because Vaani carries speaker and district metadata. Without it, a model can score well by recognizing voices rather than languages, and you would never know.

The 42 languages span four families: Indo-Aryan, Dravidian, Sino-Tibetan and English. To our knowledge, LID had not previously been attempted at this scale of Indic diversity.

Result 1: data built for breadth produces representations that travel

Figure 2: Frozen vs fine-tuned, both encoders, all three test sets. FLEURS and Kathbath are the out-of-domain tests. Source: paper Table I.

We compared two starting points. Whisper's encoder (350M parameters, trained on 90+ languages, mostly not Indian) and a FastConformer encoder pretrained on 60+ Indic languages from Vaani (430M parameters). Each was tested frozen, where only a small classifier on top is trained, and fine-tuned end to end.

The Vaani-pretrained encoder, left completely frozen, beat a fully fine-tuned Whisper by more than 20 points on both out-of-domain benchmarks. It did that while training far fewer parameters.

The interesting part is why, and it is a point about the data rather than the architecture. FastConformer had never seen FLEURS or Kathbath either. What it had seen was Vaani: spontaneous speech, recorded across 165 districts, from a very large and varied speaker pool. Exposure to that spread appears to produce representations that already generalize, which is exactly what you would hope a geographically and demographically diverse corpus would buy you.

There is a corollary worth stating honestly. Fine-tuning that encoder on Vaani made it worse out of domain, dropping FLEURS from 94.2 to 89.9. Pretraining on broad Vaani data helps; over-fitting to one training slice of it does not. And in-domain, on Vaani-Test, fine-tuned Whisper is genuinely better (74.2 vs 67.7 with the best objective). We are not claiming our encoder wins everywhere. It wins where generalization is the thing being measured.

Result 2: the training objective is not a detail

Figure 3: The same models under three training objectives. Source: paper Table II.

We compared plain cross-entropy, cross-entropy plus a supervised contrastive loss, and hierarchical softmax (HSM), which predicts down a four-level tree of Root, Family, Sub-family, Language instead of choosing from 42 flat options.

HSM was best for both encoders on all three benchmarks, with the largest gains out of domain. For Whisper it lifted Kathbath by 7.5 points over cross-entropy. The intuition is that it lets a model be partially right: confusing Sadri with Chhattisgarhi still gets the family correct, and that structure survives a change of corpus better than a flat 42-way decision.

Result 3: the dataset doubles as a map of what is still hard

Figure 4: Accuracy by language family and sub-family. Source: paper Table III.

Because every language contributed the same 10 hours, differences between families cannot be explained by training data volume. What they reflect is linguistic proximity, and the difficulty turns out to be very unevenly distributed. Sino-Tibetan languages reach 97.1%, English 92.3%, Dravidian 85.9%. Indo-Aryan as a whole sits at 67.6%, and within it Central Indo-Aryan drops to 58.7%.

Two confusion patterns account for most of that. Hindi and Urdu are heavily mixed up, which is linguistically expected since they share a phonological inventory and differ mainly in script and formal vocabulary. Separately, Sadri, Chhattisgarhi and Surgujia blur into one another.

Note what it takes to produce a finding like that at all. Sadri, Surgujia, Bajjika and Halbi have to be present in sufficient quantity, correctly labeled, from enough speakers, before anyone can measure how confusable they are. Vaani is not only training fuel here; it is the instrument that shows where the remaining problem sits.

What we would tell a practitioner

• If your deployment audio will differ from your training audio, start from an encoder pretrained on diverse in-region data and freeze it. Fine-tuning cost us up to 4 points out of domain.
• If your audio closely matches your training data, fine-tune a broad multilingual model instead. Whisper wins that setting.
• Use a linguistically structured objective. HSM was free accuracy across every encoder and benchmark we tried.

Download the model, the dataset it was trained on, and the related models

Everything below is open and free to use.

→ Vaani-LID_v0 — the language classifier described in this post. 42 Indian languages, MIT licensed, no gating.

from transformers import pipeline

pipe = pipeline("audio-classification", model="ARTPARK-IISc/Vaani-LID_v0", trust_remote_code=True)
print(pipe("path/to/16kHz/mono.wav"))

→ The Vaani dataset — the corpus it was trained on. About 31,255 hours from 156,000 speakers across 165 districts, covering 106 languages, CC-BY-4.0.

→ SraVaani — our multilingual ASR model for 63 Indian languages and dialects. The natural next step once you know which language you are dealing with.

If you work on Central Indo-Aryan varieties, that 58.7% is an open invitation. It is the clearest gap we found, and both the model and the data to attack it are free to download.

Questions, feedback or collaborations

We would like to hear how the model performs on your audio, especially for the low-resource varieties. Reach us at agneedh@artpark.in, vaani.connect@artpark.in, or open a discussion on the model repository.

Paper: arXiv:2606.09317 · Case study: vaani.iisc.ac.in · Earlier posts: Inside the Vaani dataset, how Vaani is being used