A ByT5-small model fine-tuned for Kashmiri/Koshur diacritic restoration: non-diacritic Kashmiri text → diacritic Kashmiri text. the average reviewer-rated accuracy of our model is approximately 77.5%. That's a reasonable first-model score for a low-resource diacritization task , the model captures most patterns but still has room to improve on edge cases and truncation issues.
1from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
2
3repo_id = "Omarrran/koshur-diacritizer-byt5-small"
4tokenizer = AutoTokenizer.from_pretrained(repo_id)
5model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)
6
7text = "کاشر زبان"
8inputs = tokenizer(text, return_tensors="pt")
9out = model.generate(**inputs, max_new_tokens=256)
10print(tokenizer.decode(out[0], skip_special_tokens=True))
1{
2 "validation": {
3 "loss": 0.061068128794431686,
4 "der_marked": 0.10005247507433969,
5 "der_all": 0.03764245052568145,
6 "wer": 0.1231150319412455,
7 "exact_match": 0.24492979719188768,
8 "runtime": 156.3775,
9 "samples_per_second": 8.198,
10 "steps_per_second": 0.262,
11 "epoch": 9.992486851990984
12 },
13 "test": {
14 "der_marked": 0.2011514510633298,
15 "der_all": 0.14687684306471502,
16 "wer": 0.21588209414870216,
17 "exact_match": 0.12782608695652173,
18 "n_sentences": 1150,
19 "n_units": 93255,
20 "n_marked": 17022
21 }
22}
We cast diacritic restoration as byte-level sequence-to-sequence transduction and fine-tune the latest released model, with the retained training checkpoint at training-checkpoints/checkpoint-6650. The extra-dataset run was initialized from an earlier trained model during training, but only the final retained checkpoint is kept in the Hub repo. Byte-level modelling avoids subword tokenisers that corrupt Perso-Arabic combining marks. Input is the un-diacritised (bare) skeleton; the target is the fully diacritised form. At inference a skeleton guard rejects any output that alters the consonant skeleton, so the model can only add marks.
Test set: 1150 sentences, 93255 letters (17022 diacritised).
