Fine-tune of csebuetnlp/banglat5
that rewrites Bangla regional dialect text into Standard Bangla.
Built as the normalization stage of a multi-dialect IVR system for primary
healthcare in regional Bangladesh (BRAC University). It sits between speech
recognition and intent understanding: the recogniser produces dialectal text, this
model standardises it so a single downstream NLU layer can serve every region.
Results
Vashantor held-out test split, 1,875 human-labelled pairs, 375 per dialect.
BLEU is sacreBLEU with the 13a tokenizer.
System
Exact match (%)
CER (lower better)
BLEU
Copy (output = input, no change)
0.5
0.345
10.45
Rule-based lexicon
0.5
0.346
10.46
This model
11.1
0.165
47.43
Gain over copy
+10.6 pts
-52.2 %
+36.98
Read the copy baseline alongside the BLEU. Source and target share most of
their tokens in this task, so a bare BLEU number is not interpretable on its own.
The 36.98-point gain over copying is what the model contributes.
Per dialect, against the benchmark
Dialect
Copy
This model
mT5 (Faria et al.)
BanglaT5 (Faria et al.)
Chittagong
2.31
39.90
36.75
44.03
Noakhali
5.31
41.76
37.43
47.38
Sylhet
14.22
47.55
51.32
51.08
Barishal
6.17
49.22
48.56
53.50
Mymensingh
22.43
58.29
64.74
69.06
Mean
-
47.34
47.76
53.01
Published figures are from the Vashantor paper on the same test split. This single
multi-dialect model is level with their per-dialect mT5 in the mean and better on
the two hardest dialects, while trailing their BanglaT5 by 5.67.
Two caveats. BLEU on Bangla is strongly tokenizer-dependent: the identical
hypotheses score 66.55 under flores200 sentencepiece against 47.43 under 13a,
so unless the published work used the same tokenizer part of the difference is
measurement. And this is one model for all dialects, which may not be true of the
comparison figures.
Usage
The model expects a dialect tag followed by the task prefix. Use [unknown]
unless you are certain of the dialect; see the note below on why that is the
recommended default.
python
1import torch
2from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
34repo ="Nirban10/banglat5-bangla-dialect-normalizer"5tok = AutoTokenizer.from_pretrained(repo, use_fast=False)# slow tokenizer required6model = AutoModelForSeq2SeqLM.from_pretrained(repo).eval()78PROMPT ="আঞ্চলিক ভাষা থেকে প্রমিত বাংলায় রূপান্তর করো: "910defnormalize(text, dialect="unknown"):11 enc = tok(f"{PROMPT}[{dialect}] {text}", return_tensors="pt",12 truncation=True, max_length=128)13with torch.inference_mode():14 out = model.generate(**enc, max_new_tokens=96, num_beams=4,15 no_repeat_ngram_size=3, early_stopping=True)16return tok.decode(out[0], skip_special_tokens=True).strip()1718print(normalize("আঁর তিন দিন ধরি জ্বর অইয়ের।"))19# -> আমার তিন দিন ধরে জ্বর হয়েছে
use_fast=False is required: fast-tokenizer conversion of the BanglaT5
sentencepiece model fails against current protobuf releases.
Known dialect tags: barishal, chittagong, mymensingh, noakhali, sylhet,
unknown.
Why [unknown] is the recommended default
The tag was intended to let one model apply dialect-specific edit strength. It
barely does. Measured on the same test set:
Condition
BLEU
CER
Exact %
Correct tag
47.43
0.165
11.1
[unknown] tag
47.13
0.165
10.6
Deliberately wrong tag
46.89
0.166
10.8
The whole spread is 0.54 BLEU. A wrong tag costs almost nothing, and [unknown]
costs 0.30 against an oracle. Unless you have a highly reliable dialect
identifier, [unknown] is the simpler and safer choice. In the source system the
dialect classifier is 78.8 % accurate, and routing it into the normalizer was
measured at 47.23, which is 0.10 above always sending [unknown], so the coupling
was dropped.
The one real exception is Mymensingh, which gains 2.24 BLEU from a correct tag
(58.29 against 56.05). Mymensingh is the dialect closest to Standard Bangla, where
editing least is correct.
Decoding
Beam width and n-gram blocking make no measurable difference on this task, which
indicates the output distribution is sharp and the bottleneck is the model rather
than the search.
Config
BLEU
CER
beams 4 + no_repeat_ngram_size=3
45.49
0.172
beams 4, no blocking
45.49
0.172
beams 5
45.49
0.172
greedy
45.18
0.175
Measured on the previous model revision. Greedy costs 0.31 BLEU and is several
times faster, which is worth taking on CPU or low-memory deployments.
Training
Setting
Value
Base model
csebuetnlp/banglat5 (T5, d_model 768)
Training pairs
10,064 (Vashantor Train + Validation)
Dialect tag
[dialect] prefix; 15 % of examples tagged [unknown]
Train / validation
10,064 / 529
Optimiser
AdamW, weight decay 0.01
Learning rate
2e-4, linear schedule, 6 % warmup
Batch size
16
Epochs
13
Precision
bf16 (T5 is numerically unstable in fp16)
Gradient clipping
1.0
Max source / target length
96 / 96 tokens
Best validation loss
0.3013
Hardware / wall-clock
RTX 4080 SUPER, ~20 min
Training deliberately stops at 13 epochs. A 20-epoch run overfits: validation loss
bottoms at epoch 13 and drifts upward afterwards while training loss keeps falling.
15 % of examples carry [unknown] instead of the true dialect, so the model stays
well-behaved when the tag is absent or unreliable.
Examples
Chittagong in : আঁর তিন দিন ধরি জ্বর অইয়ের।
out: আমার তিন দিন ধরে জ্বর হয়েছে
Rangpur in : মোর মাথা ব্যথা নাগেছে।
out: আমার মাথা ব্যথা করছে না
Standard in : আমার জ্বর হয়েছে।
out: আমার জ্বর হয়েছে
All three produced with the tag [unknown], exactly as the usage snippet
above calls the model. Outputs are verbatim, not idealised.
The Chittagong example corrects pronoun, postposition and verb aspect together
while preserving the stated duration rather than substituting it, which matters
because duration is a clinically relevant field.
The Rangpur example is a failure, and it is shown deliberately. Rangpur is
absent from Vashantor, so the model has no parallel data for it. The pronoun is
normalised correctly, but the model inserts a negation that is not present in
the input, turning a report of head pain into a denial of it. An earlier
revision of this model handled the same sentence correctly, so this is a
regression rather than a known constant.
On dialects outside the five with parallel data, this model can invert meaning
while producing fluent, confident Standard Bangla. Do not rely on it for
unseen dialects without a check that also runs on the raw input.
Already-standard input passes through unchanged, so speakers close to Standard
Bangla are not mangled by spurious rewriting.
Limitations
General-domain, not medical. Vashantor is conversational Bangla. These
numbers measure the dialect-to-standard transformation, not healthcare accuracy.
This revision contains no healthcare-specific training data.
Five dialects have parallel data. Barishal, Chittagong, Mymensingh,
Noakhali, Sylhet. Everything else, including Rangpur, relies on generalisation
and is not measured.
Meaning inversion on unseen dialects. On a Rangpur input the model inserts
a negation absent from the source, reversing the sense of the sentence (see
the Rangpur example above). Negation handling on out-of-distribution input is
not characterised. In any application where a negated and non-negated reading
differ materially, and clinical triage is such an application, do not consume
this model's output without an independent check.
Fluency is not faithfulness. Given noisy input, for example a poor speech
recognition transcript, the model produces confident, well-formed Standard
Bangla that may not reflect what was actually said. In any safety-relevant
pipeline, keep a check that also runs on the raw input.
Exact match is low in absolute terms (11.1 %) because the metric demands
character-perfect agreement with one human reference, while valid Standard
Bangla renderings vary. CER and BLEU are the informative measures here.
No confidence intervals. 375 pairs per dialect; differences under about a
point should not be over-interpreted.
Not a medical device. Built for an IVR system that routes and triages and is
explicitly designed never to diagnose or prescribe.
Licence
CC BY-NC-SA 4.0, inherited from csebuetnlp/banglat5, whose release states
that contents are restricted to non-commercial research purposes under
CC BY-NC-SA 4.0. Non-commercial use only, attribution required, derivatives must
carry the same licence.
Citation
Training data:
Faria et al., Vashantor: A Large-scale Multilingual Benchmark Dataset for
Automated Translation of Bangla Regional Dialects to Bangla Language,
arXiv:2311.11142. https://data.mendeley.com/datasets/bj5jgk878b/2
Base model:
Bhattacharjee et al., BanglaNLG and BanglaT5: Benchmarks and Resources for
Evaluating Low-Resource Natural Language Generation in Bangla,
Findings of EACL 2023. arXiv:2205.11081
This model:
Md. Nirban Hossain, An Efficient Multi-Dialect IVR System for AI-Enabled
Primary Healthcare Services to Regional Communities, BRAC University.
