Nirban10/banglat5-bangla-dialect-normalizer

BanglaT5 dialect-to-Standard-Bangla normalizer

Fine-tune of csebuetnlp/banglat5 that rewrites Bangla regional dialect text into Standard Bangla.

Built as the normalization stage of a multi-dialect IVR system for primary healthcare in regional Bangladesh (BRAC University). It sits between speech recognition and intent understanding: the recogniser produces dialectal text, this model standardises it so a single downstream NLU layer can serve every region.

Results

Vashantor held-out test split, 1,875 human-labelled pairs, 375 per dialect. BLEU is sacreBLEU with the 13a tokenizer.

System	Exact match (%)	CER (lower better)	BLEU
Copy (output = input, no change)	0.5	0.345	10.45
Rule-based lexicon	0.5	0.346	10.46
This model	11.1	0.165	47.43
Gain over copy	+10.6 pts	-52.2 %	+36.98

Read the copy baseline alongside the BLEU. Source and target share most of their tokens in this task, so a bare BLEU number is not interpretable on its own. The 36.98-point gain over copying is what the model contributes.

Per dialect, against the benchmark

Dialect	Copy	This model	mT5 (Faria et al.)	BanglaT5 (Faria et al.)
Chittagong	2.31	39.90	36.75	44.03
Noakhali	5.31	41.76	37.43	47.38
Sylhet	14.22	47.55	51.32	51.08
Barishal	6.17	49.22	48.56	53.50
Mymensingh	22.43	58.29	64.74	69.06
Mean	-	47.34	47.76	53.01

Published figures are from the Vashantor paper on the same test split. This single multi-dialect model is level with their per-dialect mT5 in the mean and better on the two hardest dialects, while trailing their BanglaT5 by 5.67.

Two caveats. BLEU on Bangla is strongly tokenizer-dependent: the identical hypotheses score 66.55 under flores200 sentencepiece against 47.43 under 13a, so unless the published work used the same tokenizer part of the difference is measurement. And this is one model for all dialects, which may not be true of the comparison figures.

Usage

The model expects a dialect tag followed by the task prefix. Use [unknown] unless you are certain of the dialect; see the note below on why that is the recommended default.

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "Nirban10/banglat5-bangla-dialect-normalizer"
tok = AutoTokenizer.from_pretrained(repo, use_fast=False)   # slow tokenizer required
model = AutoModelForSeq2SeqLM.from_pretrained(repo).eval()

PROMPT = "আঞ্চলিক ভাষা থেকে প্রমিত বাংলায় রূপান্তর করো: "

def normalize(text, dialect="unknown"):
    enc = tok(f"{PROMPT}[{dialect}] {text}", return_tensors="pt",
              truncation=True, max_length=128)
    with torch.inference_mode():
        out = model.generate(**enc, max_new_tokens=96, num_beams=4,
                             no_repeat_ngram_size=3, early_stopping=True)
    return tok.decode(out[0], skip_special_tokens=True).strip()

print(normalize("আঁর তিন দিন ধরি জ্বর অইয়ের।"))
# -> আমার তিন দিন ধরে জ্বর হয়েছে

use_fast=False is required: fast-tokenizer conversion of the BanglaT5 sentencepiece model fails against current protobuf releases.

Known dialect tags: barishal, chittagong, mymensingh, noakhali, sylhet, unknown.

Why [unknown] is the recommended default

The tag was intended to let one model apply dialect-specific edit strength. It barely does. Measured on the same test set:

Condition	BLEU	CER	Exact %
Correct tag	47.43	0.165	11.1
[unknown] tag	47.13	0.165	10.6
Deliberately wrong tag	46.89	0.166	10.8

The whole spread is 0.54 BLEU. A wrong tag costs almost nothing, and [unknown] costs 0.30 against an oracle. Unless you have a highly reliable dialect identifier, [unknown] is the simpler and safer choice. In the source system the dialect classifier is 78.8 % accurate, and routing it into the normalizer was measured at 47.23, which is 0.10 above always sending [unknown], so the coupling was dropped.

The one real exception is Mymensingh, which gains 2.24 BLEU from a correct tag (58.29 against 56.05). Mymensingh is the dialect closest to Standard Bangla, where editing least is correct.

Decoding

Beam width and n-gram blocking make no measurable difference on this task, which indicates the output distribution is sharp and the bottleneck is the model rather than the search.

Config	BLEU	CER
beams 4 + no_repeat_ngram_size=3	45.49	0.172
beams 4, no blocking	45.49	0.172
beams 5	45.49	0.172
greedy	45.18	0.175

Measured on the previous model revision. Greedy costs 0.31 BLEU and is several times faster, which is worth taking on CPU or low-memory deployments.

Training

Setting	Value
Base model	csebuetnlp/banglat5 (T5, d_model 768)
Training pairs	10,064 (Vashantor Train + Validation)
Dialect tag	[dialect] prefix; 15 % of examples tagged [unknown]
Train / validation	10,064 / 529
Optimiser	AdamW, weight decay 0.01
Learning rate	2e-4, linear schedule, 6 % warmup
Batch size	16
Epochs	13
Precision	bf16 (T5 is numerically unstable in fp16)
Gradient clipping	1.0
Max source / target length	96 / 96 tokens
Best validation loss	0.3013
Hardware / wall-clock	RTX 4080 SUPER, ~20 min

Training deliberately stops at 13 epochs. A 20-epoch run overfits: validation loss bottoms at epoch 13 and drifts upward afterwards while training loss keeps falling.

15 % of examples carry [unknown] instead of the true dialect, so the model stays well-behaved when the tag is absent or unreliable.

Examples

Chittagong   in : আঁর তিন দিন ধরি জ্বর অইয়ের।
             out: আমার তিন দিন ধরে জ্বর হয়েছে

Rangpur      in : মোর মাথা ব্যথা নাগেছে।
             out: আমার মাথা ব্যথা করছে না

Standard     in : আমার জ্বর হয়েছে।
             out: আমার জ্বর হয়েছে

All three produced with the tag [unknown], exactly as the usage snippet above calls the model. Outputs are verbatim, not idealised.

The Chittagong example corrects pronoun, postposition and verb aspect together while preserving the stated duration rather than substituting it, which matters because duration is a clinically relevant field.

The Rangpur example is a failure, and it is shown deliberately. Rangpur is absent from Vashantor, so the model has no parallel data for it. The pronoun is normalised correctly, but the model inserts a negation that is not present in the input, turning a report of head pain into a denial of it. An earlier revision of this model handled the same sentence correctly, so this is a regression rather than a known constant.

On dialects outside the five with parallel data, this model can invert meaning while producing fluent, confident Standard Bangla. Do not rely on it for unseen dialects without a check that also runs on the raw input.

Already-standard input passes through unchanged, so speakers close to Standard Bangla are not mangled by spurious rewriting.

Limitations

• General-domain, not medical. Vashantor is conversational Bangla. These numbers measure the dialect-to-standard transformation, not healthcare accuracy. This revision contains no healthcare-specific training data.
• Five dialects have parallel data. Barishal, Chittagong, Mymensingh, Noakhali, Sylhet. Everything else, including Rangpur, relies on generalisation and is not measured.
• Meaning inversion on unseen dialects. On a Rangpur input the model inserts a negation absent from the source, reversing the sense of the sentence (see the Rangpur example above). Negation handling on out-of-distribution input is not characterised. In any application where a negated and non-negated reading differ materially, and clinical triage is such an application, do not consume this model's output without an independent check.
• Fluency is not faithfulness. Given noisy input, for example a poor speech recognition transcript, the model produces confident, well-formed Standard Bangla that may not reflect what was actually said. In any safety-relevant pipeline, keep a check that also runs on the raw input.
• Exact match is low in absolute terms (11.1 %) because the metric demands character-perfect agreement with one human reference, while valid Standard Bangla renderings vary. CER and BLEU are the informative measures here.
• No confidence intervals. 375 pairs per dialect; differences under about a point should not be over-interpreted.
• Not a medical device. Built for an IVR system that routes and triages and is explicitly designed never to diagnose or prescribe.

Licence

CC BY-NC-SA 4.0, inherited from csebuetnlp/banglat5, whose release states that contents are restricted to non-commercial research purposes under CC BY-NC-SA 4.0. Non-commercial use only, attribution required, derivatives must carry the same licence.

Citation

Training data:

Faria et al., Vashantor: A Large-scale Multilingual Benchmark Dataset for Automated Translation of Bangla Regional Dialects to Bangla Language, arXiv:2311.11142. https://data.mendeley.com/datasets/bj5jgk878b/2

Base model:

Bhattacharjee et al., BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla, Findings of EACL 2023. arXiv:2205.11081

This model:

Md. Nirban Hossain, An Efficient Multi-Dialect IVR System for AI-Enabled Primary Healthcare Services to Regional Communities, BRAC University.