Urdu OCR — Fine-Tuned TrOCR (SI-26)
A
TrOCR model fine-tuned end-to-end to read printed Urdu (Nastaliq-style) text from images and output editable Unicode text.
Try the live demo: Streamlit app
Model Description
Optical Character Recognition for Urdu lags far behind Latin-script OCR — Urdu's cursive, context-dependent Nastaliq script and the scarcity of labeled datasets make off-the-shelf tools like Tesseract perform poorly out of the box (a baseline test on this project's own data measured only a 2.1% word-recovery rate from Tesseract). This model fine-tunes TrOCR specifically on Urdu text so it can read real-world Urdu images — for example, digitizing scanned documents, signboards, or book pages.
TrOCR pairs a vision encoder (which "looks" at the image) with a text decoder (which "writes out" what it reads). Its decoder's tokenizer is a byte-level BPE tokenizer that can already represent any Unicode text — including Urdu — via UTF-8 byte-level fallback tokens. This model fine-tunes the full pretrained checkpoint, encoder and decoder together, rather than training a decoder from scratch, so it only has to learn Urdu rather than language modeling from zero.
- Base model: microsoft/trocr-base-printed
- Fine-tuning: two-phase — decoder-only warm-up, then the full model unfrozen at a lower learning rate, 15 epochs total
- Language: Urdu (ur)
- Task: image-to-text (OCR)
- License: Apache 2.0
Intended Use
Extracting Urdu text from clean, printed, single-line images — scanned documents, signboards, book pages, or screenshots of printed text. Not intended for handwriting, multi-line paragraphs, or heavily degraded/noisy images.
How to Use
1from transformers import VisionEncoderDecoderModel, TrOCRProcessor
2from PIL import Image
3import torch
4
5repo_id = "qandeelasim13/urdu-ocr-trocr-si26"
6processor = TrOCRProcessor.from_pretrained(repo_id)
7model = VisionEncoderDecoderModel.from_pretrained(repo_id)
8model.eval()
9
10image = Image.open("your_urdu_image.png").convert("RGB")
11pixel_values = processor(images=image, return_tensors="pt").pixel_values
12
13with torch.no_grad():
14 generated_ids = model.generate(pixel_values, max_length=319, num_beams=4)
15
16text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
17print(text)
Training Data
- Size: 3,160 labeled images (~1,348 unique source images plus augmented rotation/blur/brightness copies)
- Sources: UTRSet-Real (real printed Urdu word images, ICDAR 2023), synthetic sentences rendered from Urdu Wikipedia text in 3 different Urdu fonts, augmented variants (blur/brightness/rotation), and manual screenshots from Dawn Urdu, BBC Urdu, Jang, and Wikipedia.
- Split: leakage-safe, grouped by parent image so augmented copies of the same source image never appear in both train and test.
Evaluation Results
| Metric | Score |
|---|
| Character Error Rate (CER) | 0.52 |
| Character-level accuracy | 47.66% |
| Training loss | 5.00 → 0.18 (avg per epoch, 15 epochs) |
Sample predictions showed exact or near-exact matches on shorter sentences, with more errors on longer, more complex sentences.
Limitations
- Dataset size (~1,348 unique source images) is the primary bottleneck — the training notebook's own working estimate suggests 80–95% accuracy would be achievable with substantially more data.
- Performs best on clean, printed, single-line Urdu text; accuracy drops on handwriting, multi-line paragraphs, low-resolution scans, and heavy noise/blur.
- Character Error Rate of 0.52 means roughly half of characters may be misread on average — always spot-check output for anything beyond casual/exploratory use.
Training Procedure
- Loaded the pretrained microsoft/trocr-base-printed checkpoint (encoder + decoder + processor together).
- Phase 1: froze the encoder, trained the decoder only (warm-up).
- Phase 2: unfroze the full model, continued training at a lower learning rate.
- 15 epochs total, evaluated on a held-out, leakage-safe test split using Character Error Rate.
Frameworks: PyTorch, Hugging Face transformers (VisionEncoderDecoderModel, Seq2SeqTrainer), evaluate + jiwer for CER, trained on a Google Colab GPU runtime (Tesla T4).
Credit
Qandeel Asim — built during the Code Saviours ML/AI Internship, Batch SI-26.
