You can deploy this model and start earning money today!

ianuragbhatt/
hindi-handwritten-ocr-qwen2vl-2b

Name: hindi-handwritten-ocr-qwen2vl-2b
Availability: InStock

2506.02295

Views

No views yet

Hindi Handwritten OCR — Fine-tuned Qwen2-VL-2B

Fine-tuned vision-language model for Hindi handwritten text recognition from images, trained on the IIIT-INDIC-HW-WORDS-Hindi dataset.

Model Details

Property	Value
Base model	`Qwen/Qwen2-VL-2B-Instruct`
Fine-tuned model	`ianuragbhatt/hindi-handwritten-ocr-qwen2vl-2b`
Merged model	`ianuragbhatt/hindi-handwritten-ocr-qwen2vl-2b-merged`
Parameters	~2.2B (base) + LoRA adapter
Dataset	69,853 train / ~12,000 val+test images
Training method	LoRA fine-tuning (rank 16)
Vision encoder	Frozen
Trainable params	~10–15% of total
License	Apache-2.0 (base) / CC-BY-4.0 (dataset)

Architecture

Qwen2-VL-2B-Instruct
├─ Vision Encoder  (675M params, frozen)
├─ MLP Projector   (cross-modal adapter, frozen)
└─ Qwen2-1.5B LLM (1.5B params, LoRA-tuned)
   └─ LoRA rank 16 on attention + MLP layers

Why this model?

Proven OCR fine-tuning recipe from QARI-OCR paper
Native transformers support (no trust_remote_code)
Dynamic image resolution (better for handwritten text of varying sizes)
Works with TRL SFTTrainer + PEFT LoRA out of the box

Quick Start

1. Install dependencies

pip install transformers>=4.56 accelerate peft bitsandbytes

2. Run inference

python

from PIL import Image
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor

model_id = "ianuragbhatt/hindi-handwritten-ocr-qwen2vl-2b-merged"

# Load (use load_in_4bit=True for low VRAM)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

# Load image
image = Image.open("hindi_sample.jpg").convert("RGB")

# Build prompt
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Transcribe the Hindi handwritten text in this image."},
        ],
    }
]

# Prepare
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[image], return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

# Generate
generated = model.generate(**inputs, max_new_tokens=256, do_sample=False)
output = processor.batch_decode(generated[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(output)

3. CLI inference

python inference_hindi_ocr.py --image hindi_sample.jpg --model ianuragbhatt/hindi-handwritten-ocr-qwen2vl-2b-merged

Training Recipe

Hyperparameter	Value
Learning rate	2e-4
LoRA rank	16
LoRA alpha	16
LoRA dropout	0.05
Batch size	2
Gradient accumulation	4
Epochs	1
Warmup ratio	0.03
LR scheduler	linear
Weight decay	0.01
Max sequence length	512
Optimizer	AdamW (via TRL defaults)

Based on: QARI-OCR (Arabic OCR fine-tuning on Qwen2-VL-2B), arxiv:2506.02295

Dataset

Source: IIIT-INDIC-HW-WORDS-Hindi
Format: image (JPEG) + text (Hindi Devanagari string)
Size: 69,853 train / 11,882 test / 11,882 validation
License: CC-BY-4.0

Hardware Requirements

Mode	GPU VRAM	Notes
Training (LoRA, bf16)	~12–16 GB	T4 / RTX 3060 / Colab T4 works
Inference (bf16)	~6–8 GB	Any modern GPU
Inference (4-bit)	~3–4 GB	GTX 1060 6GB, laptop GPUs
Inference (CPU)	~8–12 GB RAM	Very slow but works

How to Train (Free GPU)

Option A: Google Colab (Recommended)

Open Google Colab
Set Runtime → Change runtime type → GPU (T4)
Copy the cells from colab_hindi_ocr.ipynb.py (in this repo)
Run all cells (~3–4 hours for 1 epoch)

Option B: Kaggle

Open Kaggle Notebooks
Set Accelerator → GPU T4 x2
Use the same script
Run (~2–3 hours)

Option C: Local GPU

bash

pip install transformers>=4.56 trl peft accelerate datasets bitsandbytes
python train_hindi_ocr_colab.py

Deployment Options for Cheaper Devices

A. 4-bit Quantization (already <4 GB)

python

model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    load_in_4bit=True,
    device_map="auto",
)

B. ONNX Export for Edge Devices

bash

optimum-cli export onnx \
  --model ianuragbhatt/hindi-handwritten-ocr-qwen2vl-2b-merged \
  ./onnx_model/

C. Even Smaller Alternative (Under 1.2B params)

If you must stay under 1.2B params, use:

OpenGVLab/InternVL2_5-1B (938M params)
- Same training approach, swap BASE_MODEL in the script
- Requires trust_remote_code=True and may need older transformers
- Less proven for OCR but viable with same dataset

Files in this Repo

File	Description
`train_hindi_ocr_colab.py`	Full training script (run on Colab/Kaggle/Local)
`train_internvl2_5_1b.py`	Alternative training script for InternVL2_5-1B (sub-1B)
`inference_hindi_ocr.py`	CLI inference script
`colab_hindi_ocr.ipynb.py`	Colab notebook cells (copy-paste ready)
`README.md`	This file

License

Model weights: Apache-2.0 (inherited from Qwen2-VL)
Dataset: CC-BY-4.0
Code: MIT