Model Card for phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora
This model is a LoRA fine-tuned version of Qwen/Qwen2.5-VL-3B-Instruct specifically optimized for Optical Character Recognition (OCR) tasks using the MJSynth dataset.
Model Details
Model Description
This model utilizes Parameter-Efficient Fine-Tuning (PEFT) via LoRA and 4-bit quantization to adapt the base Qwen2.5-VL-3B-Instruct model for highly accurate text recognition from images. It achieves significant improvements in Character Error Rate (CER) and Exact Match accuracy compared to the base model.
- Developed by: phucdu
- Model type: Vision-Language Model (VLM) / LoRA Adapter
- Language(s) (NLP): English
- License: Apache 2.0 (inheriting from Qwen)
- Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct
Uses
Direct Use
Extracting and reading text from images (Optical Character Recognition - OCR). It is particularly tuned on single-word crops but generalizes well to standard text reading tasks.
How to Get Started with the Model
(Mục này đã được để trống theo yêu cầu của tác giả - Không chứa code tải mô hình).
Training Details
Training Data
A subset of priyank-m/MJSynth_text_recognition.
- Pool size: 8000 examples
- Split: 6400 Train / 800 Validation / 800 Test.
Training Procedure
Training Hyperparameters
- Training regime: LoRA with 4-bit quantization (nf4, double quant) and bfloat16 compute dtype.
- LoRA Rank (r): 16
- LoRA Alpha: 32
- LoRA Dropout: 0.05
- Target Modules:
down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj
- Epochs: 3
- Batch Size: 16 (Train) / 8 (Eval)
- Learning Rate: 1e-4
- LR Scheduler: Cosine
- Warmup Ratio: 0.03
- Weight Decay: 0.0
Speeds, Sizes, Times
- Hardware: 1x NVIDIA L4 (24GB) GPU on Google Colab Pro.
- Training Time: ~35 minutes (2115 seconds for 3 epochs).
Evaluation
Testing Data, Factors & Metrics
- Testing Data: 800 held-out samples from the MJSynth dataset.
- Metrics: Character Error Rate (CER), Word Error Rate (WER), Exact Match Accuracy.
Results
Comparing Base Model vs. Fine-tuned Model on the test set:
- CER: 1.3056 ➔ 0.0842 (93.5% relative improvement)
- WER: 2.3462 ➔ 0.1437 (93.8% relative improvement)
- Exact Match Accuracy: 62.37% ➔ 85.62% (37.2% relative improvement)
- Avg Inference Latency: ~0.768 sec/image
Summary
The LoRA fine-tuning process significantly enhanced the model's OCR capabilities on the target dataset, bringing the exact match accuracy up to over 85% while massively dropping both CER and WER.
