01| When Is Not One Language: Foundation ASR Models Fail on Regional Bangla Dialects

A zero-shot benchmark of seven open speech-recognition models on spontaneous regional Bengali speech from 12 districts, using the test split of RegSpeech12.

---

Current speech foundation models treat "Bengali" as a single language: the standard, mostly read variety heard in news broadcasts, audiobooks and crowd-sourced recordings. Bengali speakers do not live in that variety. Someone from Sylhet, Chittagong, Noakhali or Sandwip speaks a regional dialect that differs from standard Bangla in pronunciation, vocabulary and morphology. Several of these dialects are spoken by millions of people, and the models fail on them almost completely.

This is my own research history. At Bengali.AI I benchmarked and fine-tuned Wav2Vec2 and Whisper on more than 100 hours of regional Bangla speech, and I co-built the RegSpeech12 and Ben-10 dialect corpora. The pattern has been the same every time: a model that looks excellent on standard Bangla becomes unusable once the speaker comes from a different district.

Table 1: clips and hours per district

Table 2: WER and CER per district

Findings

Table 2 shows that recognition performance degrades consistently along a dialectal gradient for every model. WER is lowest for Tangail and Comilla (0.31–0.32), the varieties closest to Standard Bangla. The best WER rises to 0.82–0.92 for Sylhet, Chittagong, Kishoreganj, Sandwip and Noakhali. For four of these districts, the best CER remains at 0.52–0.58, so errors extend beyond orthographic variation. In 7 of 12 districts, all models exceed a WER of 0.70.

Scale does not close this gap. Whisper-large-v3 (2B) exceeds a WER of 1.0 in 9 districts and SeamlessM4T v2 in 8, while the much smaller BanglaASR (0.2B) outperforms Whisper-large-v3 in all 12. Indic-Transcribe-core achieves the lowest error in 23 of 24 district–metric WER-CER cases.

Indic-specific fine-tuning also does not guarantee robustness. IndicSeamless improves on its base model for Tangail (WER 0.47 vs. 0.57) but degrades sharply for Noakhali (2.34 vs. 1.36), Kishoreganj (2.11 vs. 1.44) and Sandwip (1.97 vs. 1.21). WER above 1.0 requires substantial insertions, which suggests fluent but unfaithful transcriptions.

---

Proposal

Cross-dialect transfer: how much dialect data is needed, and does it carry over to unseen dialects?

Collecting transcribed speech for every dialect is expensive. Ben-10 showed that dialect-specific training narrows the gap. Work on Arabic speech recognition and Quebec French shows that multi-dialect training helps low-resource dialects, and that fluency in the standard variety does not carry over to regional ones.

I propose to answer two questions for both ASR models and speech-aware LLMs:

• RQ1 (data efficiency): How many minutes or hours of dialect speech does a strong base model need before its error rate on that dialect approaches its error rate on standard Bangla?
• RQ2 (transfer): If a model is trained on some dialects, how much does it improve on dialects it has never heard, and does closeness between dialects (geographic, phonological, lexical) predict that transfer?

1. Data curation: a dialect-stratified, transfer-ready corpus

• Group districts into dialect clusters using the dialect groupings documented in RegSpeech12, plus the dialect distances measured in this benchmark. One possible grouping is: south-eastern coastal (Chittagong, Noakhali, Sandwip), north-eastern (Sylhet, Habiganj), central (Tangail, Narsingdi, Kishoreganj, Comilla), southern (Barishal), western (Narail) and northern (Rangpur).
• Leave-one-cluster-out splits: every cluster is held out once and never seen in training, which directly measures transfer to an unseen dialect.
• Data-budget: subsets of 0, 15 and 30 minutes, and 1, 2 and 5 hours per dialect, with fixed speakers per subset.
• Two-layer transcripts: Each clip gets a dialect-faithful transcript of what was said, plus a standard-Bangla gloss of what it means. This lets us train and evaluate a verbatim transcriber and a normalising transcriber separately, and it measures how often a model "standardises" a speaker instead of transcribing them.
• Failure-driven collection: The most frequent substitutions from this benchmark (dialect word to standard word) form a seed lexicon of dialect vocabulary. New recording prompts and active-learning selection then target the words and phonetic patterns models get wrong most often.
• Pseudo-labelling unlabelled regional audio (regional radio, local video and podcasts), kept only where two architecturally different models agree. For example, Indic-Transcribe-core and the Bengali.AI Whisper-medium make different errors, so agreement between them is a useful confidence filter.

2. Fine-tuning paradigm: shared dialect knowledge without erasing variation

• Start from the strongest open base models found here: Indic-Transcribe-core (Canary, 1B) for encoder-decoder ASR, and Shrutam-2 as a speech-aware LLM.
• Parameter-efficient dialect adapters: one shared LoRA adapter for "regional Bengali" plus small per-cluster adapters. For an unseen dialect, mix the cluster adapters with weights based on how acoustically similar the new speech is to each cluster. This turns RQ2 into a measurable question: does interpolating between known dialects beat the shared adapter alone?
• Dialect tokens in the decoder prompt (for example <|bn-syl|> and <|bn-ctg|>), in the same way Whisper uses language tokens, with a generic <|bn-regional|> fallback. The model can then condition on the dialect when it is known, and predict it when it is not.
• Replay plus a consistency penalty against forgetting: mix standard Bangla data into every batch and penalise drift from the base model's outputs on standard speech. IndicSeamless shows that specialising without these safeguards can make the far dialects worse, so standard-Bangla WER is tracked as a guard-rail metric throughout.
• Temperature-based sampling across dialects, so the low-resource dialects (Noakhali, Comilla, Sandwip) receive meaningful gradient signal.

3. Architectural adjustments (speculative)

• An auxiliary phoneme/IPA head: a CTC head on the encoder that predicts IPA alongside text. Dialects differ mainly in pronunciation, so a shared phonetic layer should transfer between dialects more easily than spelling does. This builds on the IPA-transcription competitions I helped organise at Bengali.AI.
• Dialect-aware experts in speech-LLMs: Shrutam-2 already routes each utterance through a Mixture-of-Experts projector. Adapting only the projector and its experts on dialect data, while keeping the LLM frozen, lets dialect variation live in the acoustic interface without disturbing the model's knowledge of written Bangla. The router weights also show which dialects the model treats as similar.

4. Evaluation and deliverables

• **Per-district WER and CER **, Data-efficiency curve for each dialect, and a train-on-A / test-on-B transfer matrix between clusters.
• Standardisation rate: how often the output matches the standard-Bangla gloss instead of the dialect-faithful transcript. This measures erasure directly, not only recognition errors.
• Standard-Bangla WER as a guard-rail, to show that the dialect gains do not come at the cost of the standard variety.
• Open release of the splits, data-budget subsets, adapters, normalisation code and evaluation scripts, so that other low-resource language communities with strong dialect variation can reuse the pipeline.

---

References

Dataset

• Md. Rezuwan Hassan, Azmol Hossain, Kanij Fatema, Rubayet Sabbir Faruque, Tanmoy Shome, Ruwad Naswan, Trina Chakraborty, Md. Foriduzzaman Zihad, Tawsif Tashwar Dipto, Nazia Tasnim, Nazmuddoha Ansary, Md. Mehedi Hasan Shawon, Ahmed Imtiaz Humayun, Md. Golam Rabiul Alam, Farig Sadeque and Asif Sushmit. RegSpeech12: A Regional Corpus of Bengali Spontaneous Speech Across Dialects. arXiv:2510.24096, 2025. https://arxiv.org/abs/2510.24096

02| UnmadBench: Can AI models explain cultural grounded satire, or only recognise it?

UnmadBench is a small, annotated multimodal benchmark built from Unmad (উন্মাদ), Bangladesh's longest‑running satirical magazine (Dhaka, 1978 to present). It asks whether vision‑language and language models can explain why a Bangla cartoon is funny, not just label it as satire or pick an answer from a list, and whether giving the model real cultural precedent (the news, idiom or earlier cartoon the joke depends on) closes the gap.

---

Source

Twenty cartoons from the March and March–April 2025 (Eid) issues of Unmad, with permission (rights_status: permission_granted; editor Ahsan Habib, publisher Sagar Khandaker). Formats: 10 single‑panel cartoons, 9 multi‑panel strips (up to 5 panels), 1 annotated‑diagram parody. 76 dialogue turns in total.

Subsets

Annotation schema

Tasks

---

3. Evaluation Setup

Four input modes isolate reading from understanding

Findings

Multiple-choice evaluation substantially overstated model understanding. All instruction-tuned models achieved 100% accuracy on the multiple-choice task, even when given only the image. However, the same models performed only 34% of the required key points. This deficit was primarily cultural. Under closed-book conditions, recall was 0.28 on culture-bound items compared with 0.49 on culture-light items. The most frequent error was universal-only reading, in which a model explains the surface mechanism of the joke, such as a pun or an incongruity, but omits the local referent on which the satire depends. This error occurred in 57% of explanations for culture-bound items and 14% for culture-light items. Providing gold precedent improved recall on both subsets (from 0.28 to 0.44 and from 0.49 to 0.68), yet the gap between them remained at approximately 0.2. Retrieved precedent improved performance only when retrieval succeeded (0.46 on a hit versus 0.27), and distractor precedent raised the rate of universal-only reading on culture-bound items to 67%. Finally, recall declined as the temporal dependency of the joke increased, from 0.43 for low dependency to 0.31 for medium and 0.25 for high.

---

Models explain the joke an outsider would get, not the one Bangladeshis laugh at

Satire is the sharpest version of this problem. A satirical cartoon is compressed shared memory: the cartoonist deliberately leaves out the part every reader already knows, because saying it would kill the joke. Consider the first item in the benchmark:

Man (with shotgun): তোর সাহস তো কম না! আমার দখল করা বাড়ির মধ্যে তুই বাড়ি বানাস!? "You've got some nerve! Building a house inside the house I grabbed!?" Spider: আরে বাড়ি না, এটা আমার ওয়েবসাইট। "It's not a house, it's my website."

Any reader with English sees the pun: a spider's web is its "website". A Bangladeshi reader sees the actual target in two words, দখল করা ("grabbed"): the man openly admits he seized the house, the way local strongmen and political mastans seize homes and land by force, and is outraged that someone else is building on his stolen property. The satire is the hypocrisy of the land‑grabber; the pun is just how the spider escapes.

This was the hardest item in the benchmark (mean closed‑book recall 0.02 across all 13 model). Given the gold transcript, Gemma‑3‑4B explained the joke as an "unexpected situation", a "word ambiguity" and a "mix of technology and tradition", and never mentioned that the man had grabbed the house. Qwen3‑VL‑4B fell into a repetition loop. Every model's answer was classified by the judge as universal_only.

Current multilingual models can read Bangla, and can often tell that something is satire, but they systematically default to the culture‑free layer of a joke and miss the local referent that carries its meaning. We call this universal‑only reading. It is invisible to standard benchmarks for four reasons.

Benchmarks reward recognition and choice, not explanation. Multiple‑choice task is the demonstration: three VLMs scored 100% on it, including in M1 where Gemma's transcription had a character error rate above 2 (it described the picture rather than reading it) and in M4 where the OCR tool read nothing at all for half the items. Multiple‑choice formats are easy to solve from surface cues; open explanation scored against annotated key points is not.

Cultural benchmarks test facts, not tacit use. Benchmarks such as CAMeL and Global‑MMLU show that models lean on Western‑centric knowledge, but they mostly ask encyclopedic questions. Satire tests the opposite: knowledge so widely shared that nobody writes it down, applied pragmatically to infer what was left unsaid. What everyone knows is exactly what no corpus states.

The knowledge is time‑bound. Satire refers to this month's scandal, film release or traffic rule. Our recall gradient by temporal dependency (0.43 → 0.31 → 0.25) shows that the more a joke depends on a specific moment, the less the model can recover it, and static benchmarks never measure this decay.

Bangla visual satire stacks a reading problem on top. Hand lettering, speech bubbles and speaker attribution defeat both general OCR (Tesseract's mean CER was 0.82; it read nothing for 10 of 20 cartoons) and small VLMs' own reading. A model can fail to understand the joke simply because it never read it, and a text‑only benchmark cannot separate the two.

---

Models

Judge

Qwen2.5‑7B‑Instruct (4‑bit) receives the gold explanation, the key points, the misreading list and the model's answer, and returns JSON: which key points are clearly expressed (paraphrase and English count), which misreading the answer mainly commits, whether it hallucinates an anchor, and whether it refuses. The main score is recall over required key points. A 30‑answer validation sample (213 answer × key‑point lines) is exported for human labelling; judge–human agreement (Cohen's κ) has not yet been computed (Section 5).

---

4. Results

4.1 Multiple choice is saturated, and the reason is instructive

Accuracy is 1.00 for every instruction‑tuned model in every mode and every condition (one exception: 0.95). This happens even in M4, where the OCR tool returned nothing usable for 10 of 20 cartoons. The cause: the gold option is the longest option in all 20 items, on average 3.8 times longer than the distractors (minimum 2.0×). The instruction‑tuned models select the most detailed answer; the base model, which has no such preference, scores below chance.

We report this as an artifact of our v0.1 items, not as a model capability, and it shapes the benchmark's design (Section 6). It is also the strongest argument for the "explain" framing: an MCQ‑only version of this benchmark would have concluded that 3–4B models fully understand Bangla satire.

4.2 Open explanation (T2): required key‑point recall

Recall over all key points (required and optional) is much lower, 0.07–0.43. Optional key points, which carry the deeper layers such as the 2024–25 political context or the cartoonist's own history, were hit in under 1.5% of answers in every mode: the models essentially never reach past the first layer of a joke. T4 (naming the real‑world anchor directly) reaches 0.49 pooled in closed book, so models can sometimes name the right social reality when asked directly.

4.3 The cultural gap

T2 required recall, closed book:

culture_bound recall is lower in 12 of 13 settings (in two of them, bloomz‑3b M3 and Qwen2.5‑VL‑3B M2, the difference is negligible). The one reversal, Qwen2.5‑VL‑3B in M1, is a setting where recall is low on both subsets.

The gap also follows the annotated properties of the joke:

4.5 Detection and generation quality

T1 (is this satire?). Every item is satire, so T1 accuracy is uninformative, but P(yes) shows calibration problems: Qwen2.5‑VL‑3B says "not satire" for most cartoons in every mode (mean P(yes) 0.32–0.41), and bloomz‑3b almost never says yes (P(yes) ≈ 0.02). Qwen3‑VL‑4B and Gemma‑3‑4B are near 1.0.

Degenerate Bangla generation. 120 of 1,040 explanations (11.5%) contain repetition loops (the same 25‑character span three or more times), concentrated in the Qwen VL models (15–24% of their answers in M1, M2 and M4). Gemma's answers are long (about 1,000 characters vs about 300 for Qwen) and occasionally leak Chinese characters mid‑sentence (8 answers). bloomz‑3b answers average only 68 characters.

---

Proposal

6.3 Architecture:

Hypothesise, then retrieve, with a time‑aware index. In the pilot, retrieval used the raw transcript as its query, and recall halved when retrieval missed (0.46 → 0.27). The words of a satirical line rarely name its target; "website" retrieved news about website hacking. Instead, the model should first generate candidate anchors ("this may be about forcible land grabbing") and retrieve with those hypotheses, in the spirit of HyDE (Gao et al., 2022). The index should be dated, returning only precedents published before the cartoon and favouring those close to its publication date. A small relevance gate that scores each retrieved card before it enters the prompt would further reduce distraction cost.

A dedicated reading front end for Bangla comics. Gemma's understanding dropped by 0.58 when it had to read the cartoon itself, and Tesseract read nothing on half the items. A modular pipeline would detect speech bubbles and their tails, attribute each bubble to a speaker, and transcribe hand‑lettered Bangla before the language model reasons over it. This builds directly on my layout‑analysis work in BaNLAD (polygon segmentation of degraded Bangla newspaper pages) and can be trained on the bubble‑level annotations that UnmadBench already contains. The transcript would also become a much better retrieval query (gold transcript: 0.75 hit rate; OCR text: 0.35).

6.4 How we would know it worked

Every result would be reported against a human baseline and with judge–human agreement (target κ ≥ 0.6) alongside it. -->

---

---

6.1 Data curation

I want to extend from only Bangla to a South Asian benchmark of 4,000–5,000 items. Coverage is tiered by data availability:

• Tier A (~350 items each): Bangla, Hindi, Urdu, Nepali, Sinhala, Tamil, Telugu, Malayalam, Marathi, Punjabi.
• Tier B (~150 items each): Gujarati, Kannada, Odia, Assamese, Pashto, Dari, Dhivehi, Dzongkha.

Items are drawn from editorial cartoons, satirical magazines, news and web comics. Sampling is stratified by language, decade, target (politics, education, film, transport, family, religion, cricket) and format, with code-mixed and romanised items as a separate stratum. We use a temporal split within each language and a leave-one-language-out split across languages. This tests whether grounding transfers across both time and language.

Each item links to one or more anchor cards. An anchor card is a short, dated note in the source language with an English gloss, tied to an archived source. Anchors are tagged as language-specific, national or shared regional (e.g., Partition, cricket, Bollywood). The cards serve as the retrieval corpus, the training supervision and the scoring key.

Elicit tacit knowledge from native readers. At least three native readers of different ages and regions answer, for each item, "What would someone need to know to get this?" Their answers become candidate anchors, and their disagreements are kept rather than resolved. The same answers form the per-language human baseline.

Build in contrast. Each culture_bound item is paired with:

• a culture_light paraphrase that keeps the joke's mechanism but removes the local referent;
• an anchor-swap counterfactual that pairs the same drawing with a wrong local reading;
• where feasible, a cross-cultural minimal pair: the same joke format with Indian, Pakistani and Bangladeshi referents.

Fix the multiple-choice format. All options are length-matched. universal_only distractors are fluent explanations taken from real model outputs. Every item includes a "none of these" option.

6.2 Modeling

Stage 1: anchor-first SFT with mixed context. We fine-tune a 3–4B multilingual vision-language model with LoRA. Before answering, it produces a structured explanation: the literal content, the universal mechanism, the local anchors and the satirical target. Following RAFT (Zhang et al., 2024), each training example comes with gold cards, gold cards plus distractors, distractors only, or no context. Distractors include cards from neighbouring cultures. This stage targets two pilot failures: no gain from gold precedent in some settings, and universal-only reading rising to 67% under distractors.

Stage 2: preference optimisation. For DPO (Rafailov et al., 2023), the rejected set starts with the pilot's 461 universal_only explanations and is extended with base-model outputs in each new language. The chosen examples are gold or human-corrected explanations. As an alternative, GRPO (Shao et al., 2024) would reward required-anchor recall and penalise four failures:

• hallucinated anchors;
• cross-cultural misattribution;
• political over-attribution;
• repetition loops (11.5% of pilot answers).

The culture_light set guards against reward hacking.

Open question. Should cultural knowledge live in the model's weights or in retrieval? Our hypothesis is that the weights should learn the skill of grounding, while retrieval supplies the facts. We expect this split to matter most for Tier B languages.

6.3 Architecture

Retrieval used the raw transcript as its query, and recall fell from 0.46 to 0.27 when retrieval missed. Instead, the model first generates candidate anchors (HyDE; Gao et al., 2022) and retrieves with those. The cross-lingual, time-aware index returns only precedents published before the cartoon, and a relevance gate filters cards before they enter the prompt.

Script-aware reading front end. Reading the cartoon was a major bottleneck: the M2 − M1 gap reached 0.58, and Tesseract failed on half the items. A modular pipeline will detect speech bubbles, attribute each bubble to a speaker, identify the script and transcribe hand-lettered text. It covers Bengali-Assamese, Devanagari, Nastaliq, Tamil, Sinhala, Thaana, Tibetan and Gurmukhi, and extends our BaNLAD layout-analysis work to multiple scripts. The pilot showed why this matters for retrieval: the hit rate was 0.75 with gold transcripts versus 0.35 with OCR text.

6.4 How we would know it worked

All metrics are reported as macro-averages, worst-language scores, and separately for each tier.

Results are reported against native-reader baselines, with judge–human agreement (κ ≥ 0.6) measured per language.

Acknowledgements and rights

Cartoons are reproduced from Unmad with permission, for research purposes. This work builds on the annotated Unmad archive and Unmad Bot developed at BRAC University with Prof. S M Taiabul Haque, with collaborators including Prof. Syed Ishtiaque Ahmed.

References

Gao, L., Ma, X., Lin, J., and Callan, J. (2022). Precise Zero‑Shot Dense Retrieval without Relevance Labels (HyDE). arXiv:2212.10496.

Naous, T., Ryan, M. J., Ritter, A., and Xu, W. (2024). Having Beer after Prayer? Measuring Cultural Bias in Large Language Models (CAMeL). ACL 2024.

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv:2402.03300.

Singh, S., et al. (2024). Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. arXiv:2412.03304.

Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., and Gonzalez, J. E. (2024). RAFT: Adapting Language Model to Domain Specific RAG. arXiv:2403.10131.