This repository is kept unchanged as the published record of a tokenizer defect described below. It contains a known bug and should not be used for new work. Use the successor repo instead.

	this repo	successor
name	`sinhala-print-trocr-pilot`	`sinhala-print-trocr`
role	frozen "before" state	current model
tokenizer	deletes Zero Width Joiner	ZWJ preserved
use it?	no — reference only	yes

What this model is

A TrOCR fine-tune for printed Sinhala text lines, trained on synthetic data rendered from five fonts. Input is a single line image (~2600×128); it is not a page or document recogniser.

Known defect: the decoder tokenizer deletes ZWJ

The decoder tokenizer's BertNormalizer runs with clean_text=True, which strips every Unicode Cf (format) character — including U+200D ZERO WIDTH JOINER. In Sinhala, ZWJ is what binds consonants into conjuncts such as ශ්‍රී and ප්‍රජා.

python

tok = AutoTokenizer.from_pretrained("kaan84/sinhala-print-trocr-pilot")
s = "ශ්\u200Dරී"
tok.decode(tok.encode(s, add_special_tokens=False)) == s   # False

The character is removed silently: no error, no warning, and no signal in the loss curve. The model therefore cannot emit a ZWJ conjunct at any point in training.

How much text this affects

Measured on the training corpus (6,968 usable lines after filtering one null row):

unit	affected	share
codepoints	414 / 487,209	0.085%
grapheme clusters	407 / 344,999	0.118%
lines (≥1 conjunct)	315 / 6,968	4.52%

An earlier description of this work cited ~6.6% of training text. That figure was wrong. The corrected measurements are above; the line-level figure (4.52%) is the closest to the original claim, and it is the weaker statement — a line counts as affected if it contains a single affected character.

Grapheme clusters are computed with grapheme-kit, which folds ZWJ sequences into a single cluster. Default UAX #29 segmentation does not do this and will undercount.

Reported results

Evaluated on a 5% held-out split (349 lines, seed 42) of the training dataset.

metric	value
char-CER (codepoint)	0.1820
g-CER (grapheme cluster)	0.1852 → see successor
conjunct recall	0.0000 (0/19)

Conjunct recall is 0 by construction, not by difficulty.

These numbers do not measure reading ability. Training and evaluation come from the same synthetic generator and the same five fonts, so they largely measure memorisation of one renderer. Real scanned documents are substantially harder — a single page from avishadilhara/sinhala-ocr-lk-acts-1010 scored char-CER 0.4657 with line segmentation applied.

Intended use

Reference and reproduction only. For anything else use kaan84/sinhala-print-trocr.

Citation of the underlying issue

The general problem of pre-tokenization and grapheme handling in complex scripts is treated in:

Velayuthan & Sarveswaran (2025), Egalitarian Language Representation in Language Models: It All Begins with Tokenizers, COLING 2025.
Nisfer et al. (2026), grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP.

Note that the defect in this repository is in the normalizer, not the pre-tokenizer: BertPreTokenizer preserves ZWJ intact and yields one pre-token per conjunct.