You can deploy this model and start earning money today!

Shriyansh11/
qwen2-vl-id-card-lora

Name: qwen2-vl-id-card-lora
Availability: InStock

document-understanding

ocr

image-to-text

Qwen/Qwen2-VL-2B-Instruct

Views

No views yet

Qwen2-VL-2B-Instruct LoRA for Synthetic ID Card OCR & Document Understanding

This repository contains a Low-Rank Adaptation (LoRA) adapter for Qwen2-VL-2B-Instruct fine-tuned to extract key structural fields from Employee Identification Cards. It was trained and evaluated on programmatically generated synthetic cards to ensure data privacy (PII compliance) and robustness against scanning distortions.

Model Details

Base Model: Qwen/Qwen2-VL-2B-Instruct
Adapter Type: PEFT / LoRA
Language(s): English (text & document layout understanding)
Task: Key-Value Field Extraction / OCR Document QA
Fine-Tuning Method: QLoRA (4-bit quantized base weights + 16-bit LoRA adapter projection layers)

Intended Fields to Extract:

Company Name (e.g., APEX ENTERPRISES)
Full Name (e.g., STEVEN RODRIGUEZ)
Employee ID (e.g., NEX-5239-Y)
Role/Department (e.g., Project Manager)
Date of Birth (e.g., 2000-11-19)
Expiry Date (e.g., 2035-01-25)

Evaluation & Robustness Benchmarks

The model was evaluated on a dedicated, completely unseen Test Set of 50 synthetic ID cards across four distinct environmental noise scenarios (Clean, Gaussian Blur, 10-degree Tilt/Rotation, and Low Light).

Below are the actual measured results comparing the pre-trained Base Model against the Fine-Tuned Model:

Scenario	Base Model CER	Fine-Tuned (LoRA) CER	Base Field Exact Match	Fine-Tuned (LoRA) Exact Match
Clean Set	82.80%	0.00%	92.00%	100.00%
Blurry (Gaussian)	37.29%	7.41%	35.00%	51.00%
Rotated (10°)	78.48%	0.00%	100.00%	100.00%
Low Light (0.25x)	79.26%	0.00%	96.00%	100.00%

Key Observations:

OCR Format Learning: The Base Model struggled significantly with the layout formatting (represented by high baseline Character Error Rates), whereas the Fine-Tuned Model achieved a perfect 0.00% CER on Clean, Rotated, and Low Light images.
Robustness: Under harsh rotation (10-degree scan tilt) and extreme low light (75% luminance drop), the fine-tuned model maintained a 100% exact match rate across all fields, proving layout generalization.

Training Details

Training Data

Dataset: 500 programmatically generated corporate ID cards with random names, roles, company logos, and barcodes.
Split: 400 Train / 50 Validation / 50 Test.

Training Procedure & Hyperparameters

Epochs: 3
Batch Size: 2 (Gradient Accumulation Steps = 8, Effective Batch Size = 16)
Optimizer: paged_adamw_8bit (memory pages optimization)
Precision: fp16 (mixed precision training)
Learning Rate: 2e-4 (cosine scheduler decay)
LoRA configuration:
- Rank (r): 16
- Alpha: 32
- Dropout: 0.05
- Target Modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

How to Get Started with the Model

You can load this model along with its adapter using the following snippet:

python

import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel

model_id = "Qwen/Qwen2-VL-2B-Instruct"
adapter_id = "YOUR_HF_USERNAME/qwen2-vl-id-card-lora" # Replace with your repo identifier

# Load processor
processor = AutoProcessor.from_pretrained(model_id)

# Load base model (quantized or full precision)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)

Shriyansh11/qwen2-vl-id-card-lora