The model accepts a document image and an extraction instruction, then generates a structured JSON object containing the fields visible in the document.
1Document Image
2 ↓
3Frozen SigLIP Base Patch16-384
4 ↓
5576 visual tokens × 768 dimensions
6 ↓
7Two-layer MLP projector
8768 → 1024 → 896
9 ↓
10Adaptive average pooling
11576 → 384 visual tokens
12 ↓
13Qwen2-0.5B + LoRA
14 ↓
15Structured JSON
16Training Curriculum
17Stage 1 — Indic Language Adaptation
18
19Qwen2-0.5B was adapted using LoRA on multilingual text to strengthen its capabilities in Hindi, Bengali and Tamil.
20
21Trainable component: LoRA adapter
22Vision encoder: Not used
23
24Stage 2 — Vision Projector Alignment
25
26A projector was trained to map frozen SigLIP Base visual representations into Qwen's embedding space using a multilingual image-caption dataset.
27
28Trainable component: Vision projector
29Frozen components: SigLIP Base, Qwen2 base weights and Stage-1 LoRA
30
31Stage 2B joint projector-LoRA alignment was not performed.
32
33Stage 3 — Document Understanding
34
35The Stage-2 projector and Stage-1 LoRA were jointly fine-tuned on multilingual synthetic document images for structured JSON extraction.
36
37Trainable components: Vision projector and LoRA adapter
38Frozen components: SigLIP Base and Qwen2 base weights
39
40Training Dataset
41
42Stage 3 uses:
43
44arikatokachi/indic-synthetic-documents
45
46The dataset contains:
47
482,400 synthetic document images
494 languages
506 document types
51100 samples for each language-document-type pair
52
53The dataset was divided into:
54
551,920 training samples
56240 validation samples
57240 held-out test samples
58
59The split was stratified by language and document type.
60
61Model Files
62File Purpose
63projector.safetensors Stage-3 visual projector
64projector_config.json Projector dimensions and pooling configuration
65lora_adapter/ Stage-3 LoRA adapter
66tokenizer/ Tokenizer files used during training
67training_config.json Stage-3 training hyperparameters
68evaluation_results.json Evaluation summary
69architecture.json Architecture and training-stage metadata
70sample_predictions.jsonl Example held-out test predictions
71
72The repository does not duplicate the Qwen2 or SigLIP base weights. They must be loaded from their original Hugging Face repositories.
73
74Loading the Artifacts
75from huggingface_hub import snapshot_download
76
77model_dir = snapshot_download(
78 repo_id="arikatokachi/Indic-Document-VLM"
79)
80
81Reconstruct the model using:
82
83google/siglip-base-patch16-384
84+
85projector.safetensors
86+
87Qwen/Qwen2-0.5B
88+
89lora_adapter/
90
91The project GitHub repository will provide the complete inference implementation.
92
93Input Format
94
95The model was trained with prompts in the following form:
96
97Instruction:
98Extract all visible fields from the document and return valid JSON.
99
100Answer:
101
102The visual embeddings are prepended to the text embeddings before being passed to Qwen.
103
104Preliminary Evaluation
105
106Qualitative evaluation on held-out synthetic test images shows:
107
108Strengths
109Consistent valid-JSON generation
110Strong document-type and schema recognition
111Good extraction of recurring semantic fields
112Multilingual output in English, Hindi, Bengali and Tamil
113Strong performance on departments, subjects, recipients, authorities and long notice text
114Known Weaknesses
115
116The model is less reliable on high-entropy fields requiring exact character-level recognition:
117
118Dates
119Dates of birth
120Identification numbers
121Reference numbers
122Some names and demographic fields
123
124The model may generate a plausible value matching the expected format rather than reading the exact characters in the image.
125
126This behaviour reflects a limitation of using:
127
128a generic SigLIP visual encoder rather than an OCR-specialized encoder
129384 × 384 document resolution
130adaptive pooling from 576 to 384 visual tokens
131a relatively small 0.5B language-model backbone
132Intended Use
133
134This model is intended for:
135
136multimodal-learning research
137multilingual document-understanding experiments
138synthetic document extraction
139educational demonstrations
140prototyping structured document pipelines
141Out-of-Scope Use
142
143Do not use this model for:
144
145official identity verification
146financial or legal decision-making
147government eligibility decisions
148production OCR without independent verification
149processing sensitive personal documents without appropriate privacy controls
150Limitations
151Trained primarily on synthetic document templates
152Limited visual and layout diversity
153Not evaluated on production government documents
154Exact OCR accuracy is weaker than semantic field extraction
155May hallucinate plausible values for fields it cannot read
156May not generalize to handwriting, blur, occlusion or unfamiliar layouts
157Does not provide confidence scores or uncertainty calibration
158Ethical Considerations
159
160The training dataset is synthetic and was designed to avoid exposing real personal identity information. However, a deployed document-understanding system can process sensitive information.
161
162Users should:
163
164obtain appropriate consent
165minimize collection and storage
166encrypt uploaded documents
167avoid logging personally identifiable information
168independently verify generated values
169provide human review for consequential workflows
170Future Work
171Evaluate all 240 held-out test samples with field-level metrics
172Increase document-image resolution
173Preserve more visual tokens
174Add region-based image tiling
175Introduce OCR transcription supervision
176Explore OCR-aware document encoders
177Evaluate on real-world documents with privacy safeguards
178Add confidence estimation and field-level uncertainty
179Deploy an interactive Hugging Face Space
180Citation
181@misc{walia2026indicdocumentvlm,
182 author = {Sanat Walia},
183 title = {Indic-Document-VLM: Multilingual Structured Document Understanding},
184 year = {2026},
185 publisher = {Hugging Face},
186 howpublished = {\url{https://huggingface.co/arikatokachi/Indic-Document-VLM}}
187}
188Acknowledgements
189
190This project builds upon:
191
192Qwen2 by the Qwen team
193SigLIP by Google
194Hugging Face Transformers and PEFT
195AI4Bharat resources used during multilingual data preparation
196Disclaimer
197
198This is a research prototype trained primarily on synthetic data. Generated outputs may be incomplete or incorrect and must be independently verified before use.
