kathe-r11-restorer — Kashmiri short-vowel restoration
Restores kasra (U+0650), damma (U+064F) and fatha (U+064E) to Perso-Arabic
Kashmiri text. A companion to a translation model, not a translation model.
The shipped checkpoint is r11b_dense.pt. The other files in this repo are
earlier or differently-trained arms of the same sweep, kept for reproducibility.
Why this exists
IndicTrans2 cannot emit these three marks. Each appears in exactly one token
of its 122,672-entry target vocabulary — the bare standalone mark — while every
other Kashmiri diacritic is baked into whole-word subwords. Writing چھُس would
require splitting a word to insert a bare diacritic that occurs in no natural
subword context, and beam search never does. Measured across 1,730 sentences the
translation model produced exactly zero of all three, while reproducing the
subword-embedded marks at above-reference rates.
The vocabulary is frozen in the pretrained checkpoint, so no amount of
fine-tuning fixes this and restoration has to be post-hoc.
It is worth a great deal. On the KATHE 2026 test set:
| System | Score |
|---|
| translation model alone | 10.00 |
| + diacritic lexicon | 13.52 |
| + this restorer instead | 13.99 |
| + both, as a union (shipped) | 15.05 |
That is +5.05 from restoration, against +1.17 for corpus selection and +0.83
for fine-tuning. The largest lever in the project, and it needs no GPU.
What it is
A 3.3M-parameter bidirectional transformer encoder used as a per-character
tagger: for each input character it predicts one of {none, kasra, damma, fatha}. The base-letter sequence is never an output, so restoration is
insertion-only by construction — it cannot alter, reorder or drop a
character for any label vector.
How to use it — as a union with a lexicon, not alone
The shipped configuration consults a 48k-form lexicon first and uses the
tagger only on words the lexicon has never seen (restore_merge: known). The
two fail in opposite places: the lexicon is near-unbeatable on words it has seen
and silent on the rest, while the tagger never abstains but can be talked out of
a common word by context.
1git clone https://github.com/AzhadArshad/kathe_2026 && cd kathe_2026
2python scripts/generate_translations.py --input input.csv --output output.csv
The lexicon ships in that repository; the weights are fetched from here
automatically.
Tuning knob
restore_none_bias is a logit offset on the "no mark" class. Under
restore_merge: known it acts only on words the lexicon does not know, so
it is precisely "how freely does the tagger mark the tail":
| bias | marks /100c | score |
|---|
| −1.5 | 13.78 | 14.25 |
| 0.0 | 12.36 | 14.82 |
| +1.6836 (shipped) | 11.30 | 15.05 |
| +2.25 | 11.03 | 15.04 |
| +3.25 | 10.62 | 14.91 |
| +∞ (lexicon only) | 9.85 | 13.52 |
Concave with a broad flat top over [1.68, 2.25], so the shipped value is not
balanced on a knife edge.
Two findings worth carrying elsewhere. First, chrF++ runs at beta=2,
weighting recall four times precision — so emitting a mark the reference lacks
is cheaper than omitting one it has, and tuning output density to match
reference density is a handicap rather than a target. Second, that only holds
where the model is confident: in the tail, extra marks are noise, which is what
the table above measures.
Training data
Monolingual Kashmiri, self-supervised: strip the three marks from any sentence
and (stripped, original) is a training pair.
| source | examples |
|---|
nawabhussain/Kashmiri-Language-Corpus | 41,187 |
BPCC bpcc-seed-v1 | 15,467 |
BPCC daily | 3,059 |
Held-out micro-F1 60.63 (precision 73.18, recall 51.76), best at epoch 20.
The parent project's development sets were excluded by exact stripped-and-
normalized string.
Limitations
- Only these three marks. Others already match reference density to within
5%; "restoring" them would add noise.
- Standalone it is worse than a lexicon lookup (13.81 vs 13.52 was close,
and on the internal dev set the lexicon won). Its value is on the tail —
words the lexicon has never seen — which is why the shipped system is a union
and why the tail is deliberately suppressed.
- Precision 73% / recall 52% on held-out text: it under-marks rather than
over-marks, and the bias knob exists to trade that off.
- Trained on Perso-Arabic Kashmiri only.
Licence and attribution
Apache-2.0 — trained only on BPCC (CC-BY-4.0 / CC0) and
nawabhussain/Kashmiri-Language-Corpus (Apache-2.0), both of which permit
relicensing of derivatives with attribution. Other checkpoints from the same
sweep saw CC-BY-NC-SA text and are not published here.
Attribution remains required for the corpora above regardless of this licence;
see NOTICE in the parent repository.
