Can a Speech Model Hear Where You're From? Vaani Inside PRiSM's Dialect Geolocation Benchmark

Play a short clip of spontaneous Hindi to a modern audio AI and ask a simple question: where is this speaker from?

Qwen3-Omni-Instruct gives the same answer for nearly every voice: New Delhi. It says this whether the speaker is from Rajasthan, Bihar, or anywhere in between. Gemini 2.5 Flash does only slightly better. About 65% of its guesses cluster in a few tight spots near the capital.

Here is the surprising part: the answer is right there in the audio. A small sequence model, reading nothing but the speech sounds in each clip, can place these speakers within 146 km of home on average, across a region 1,478 km wide.

Both facts come from PRiSM, a new benchmark for speech models accepted at ACL 2026. The task that revealed them was built entirely on the Vaani dataset. This post walks through how, and what it found.

At a glance, this is what each part of PRiSM contributed to the story:

What is PRiSM?

A "phone" is a single speech sound, like the difference between t and d. Phone recognition models listen to audio and write down the exact sounds they hear, using the symbols linguists use, rather than words. This matters when how something was said is the point: documenting rare languages, analyzing speech disorders, or teaching pronunciation.

PRiSM (Benchmarking Phone Realization in Speech Models) is the first open-source benchmark designed to expose blind spots in this kind of phonetic perception. It was built by 16 researchers from Carnegie Mellon, GIST, UT Austin, LMU Munich, UC Berkeley, NVIDIA, and UBC.

It puts 9 systems through 15 tasks covering clinical, educational, and multilingual speech. Seven of those systems are specialized phone recognition models. The other two are Large Audio Language Models (LALMs), the chat-style AIs that answer questions about audio: Gemini 2.5 Flash and Qwen3-Omni-Instruct.

The Question No Benchmark Had Tested

Existing speech benchmarks already test many things. Can a model transcribe? Can it identify a language? Can it tell similar sounds apart? Suites like SUPERB, Dynamic-SUPERB, BabySLM, and ZeroSpeech cover phoneme recognition, phonological analysis, pronunciation scoring, and minimal-pair contrasts.

None of them test one basic ability: given how a person pronounces things, can a model tell where they are from within one language?

Dialect is geographic. Hindi in Haryana does not sound like Hindi in Jharkhand. If a model truly hears fine phonetic detail, some of that geography should be recoverable from its output.

Testing this places exacting demands on the data. You need speech recorded across a wide, connected region, a precise location label on every clip, enough clips per location for the statistics to hold, and a license that allows reuse.

Dialect corpora with geography exist for a few languages, and the PRiSM team actually experimented with a Swiss German dataset for this task. But the geographically fine-grained nature of Vaani made it a much better fit: ~31,255 hours of spontaneous, image-prompted speech from 156,000+ speakers, collected district by district across 165 districts of India, with pincode-level location metadata, released under CC-BY-4.0.

So PRiSM's speech geolocation task, called GEO-v, was built on Vaani and nothing else. It is one of the benchmark's 15 tasks, and the only one built on Indian dialectal speech.

How the Test Works

The PRiSM team first carved a clean benchmark subset out of Vaani, called Vaani-Hi, and released it publicly under CC-BY-4.0. The recipe is simple and fully reproducible:

The result: 26,433 clips covering 1,478 km east to west and 1,703 km north to south, the entire Hindi-speaking region.

Vaani powers the entire experiment. Every clip the models listen to is a Vaani clip, and the classifier that learns to place speakers is trained end to end on Vaani. What the models themselves contribute is only their phonetic hearing: each of the 9 is used exactly as released, with no Vaani-specific tuning, so that the test measures what they already know how to hear.

Each model listens to a Vaani clip and produces what it would produce anyway: a phonetic transcript (the sounds it heard) and, for the seven specialized models, its internal features (the numbers inside the model). The two LALMs' representations are not easily accessible or poolable, so LALMs are probed only through their transcripts.

The one component that is trained on Vaani is a small classifier head, called a probe, that sits on top of each model's output. Its only job is to turn that output into a location guess, and it learns to do so entirely from the Vaani-Hi subset: 19,780 training clips teach it the sound-to-place mapping, 2,668 validation clips tune it, and every number reported below comes from 3,985 held-out test clips the probe has never heard.

The split is done pincode by pincode. Each pincode contributes some of its clips to training, some to validation, and some to test. So every location shows up in all three sets, but the specific recordings the probe is scored on are ones it has never listened to before.

The probe itself is deliberately small: a two-layer bi-directional GRU for transcripts, and attention pooling followed by a 2-layer MLP for internal features. Both are trained for geolocation with an angular error loss to predict a point on the globe. One probe is trained per model, with the same architecture, the same recipe, and fresh weights.

This division of labor is the whole point. The frozen speech model supplies the phonetic evidence. The Vaani-trained probe supplies the map. If the probe can find the speaker, the geographic signal was present in the model's output. If it cannot, the model lost that information along the way.

Figure: How a single Vaani clip flows through a frozen speech model and a Vaani-trained probe to produce a location prediction.

Finding 1: The Sounds Alone Can Locate a Speaker

Probes reading phonetic transcripts missed the speaker's true location by 146 km on average. The best combination, the W2V2P-LV60 model plus its probe, picked exactly the right location 45.7% of the time. Always guessing the most common location gets you 3.3%. That is nearly a 14x lift over blind guessing, from sound patterns alone.

The surprise is that transcripts beat the models' internal features, which missed by 253 km on average. On paper, a transcript should be the weaker signal, because writing speech down as discrete symbols throws away melody and rhythm, which are strong dialect cues.

The team hypothesizes that sound order does the work. The transcript probe reads the sequence of phones, so it can exploit how often certain sounds follow each other, which differs between dialects even when the overall sound inventories look similar.

Finding 2: At Least Some of the Signal Is Real Dialect

Could the probe be keying on something spurious, like clip length rather than dialect? The paper flags that risk in its limitations, and offers one illustrative example in the other direction using attribution maps, a technique that highlights which parts of the input a model relied on for its answer.

The Bangru dialect of Haryanvi is documented to double certain consonants. The English loanword "cooler," [kuːlar] in standard pronunciation, becomes [kullar]. The Hindi word "rakha" (meaning kept) gains a doubled k sound in the same way.

In one test clip, W2V2P-LV60 wrote the doubled sounds into its transcript, and the probe's attribution landed on one of the doubled phones. In that clip, at least, the system was reading a real, previously documented feature of the dialect straight out of the audio. The authors leave a fuller interpretability analysis to future work.

Finding 3: Ask an Audio LLM Directly, and It Says New Delhi

When Gemini 2.5 Flash and Qwen3-Omni-Instruct were prompted directly to return a lat/lon in JSON, both landed near chance level. Qwen returned a coordinate at New Delhi for nearly every clip. Gemini scored 6.5%, with about 65% of its guesses concentrated in three or four spots near the capital.

The paper calls this geographic mode collapse. With little sensitivity to dialectal variation, the models default toward the highest-resourced dialect and its home coordinates.

The story has a second half. When their transcripts were fed through the same probe setup used for the specialized models, Gemini reached 33.2% and Qwen 34.7%. That is far above chance, though still below every specialized phone recognition model (even the weakest scored 42.9%). So geographic information partly survives inside their transcriptions. It is the direct question that triggers the collapse.

What This Episode Says About Vaani

Vaani carried the task on both sides. The probes learned the dialect-to-geography mapping from its training split, and the mapping held up on strictly held-out clips. The link between dialect and geography survived Vaani's entire collection pipeline, from field recording to metadata. That is not automatic for any speech corpus.

Density mattered. Because Vaani holds many clips per pincode, PRiSM could keep only locations with 450+ clips and still cover the whole Hindi belt evenly. Sparse location labels would have made the task statistically impossible.

The license mattered. CC-BY-4.0 is what allowed an outside team to derive, rehost, and maintain Vaani-Hi as a public benchmark that anyone can adopt.

Here is how the team behind the benchmark put it:

The Vaani dataset helped us build the dialectal speech geolocation task in our paper, PRiSM: Benchmarking Phone Realization in Speech Models — the first open-source benchmark for evaluating the phonetic abilities of speech models across clinical, educational, and multilingual settings. Vaani's rich coverage of Hindi dialects and detailed regional metadata let us curate the Vaani-Hi subset and probe how well phone recognition systems capture fine-grained phonetic variation across India. PRiSM's code is openly available on GitHub and all of its datasets are on Hugging Face, and we hope they help advance research on inclusive, phonetically robust speech technology.

— Shikhar Bharadwaj, co-lead author of PRiSM, PhD student at Carnegie Mellon University

GEO-v covers one language belt. Vaani spans 105 languages across 165 districts, and the same recipe applies to any of them: sample by pincode, filter by density, split within location.

Get the Data

Building or evaluating speech models for Indian languages? Want to check whether your model actually hears dialects, or just words? The full corpus is open.

→ Download the Vaani dataset on Hugging Face

Paper: arXiv:2601.14046 · Code: github.com/changelinglab/prism · Earlier posts: Inside the Vaani dataset · How Vaani is being used