Finetuned from model : unsloth/Qwen2-VL-2B-Instruct-16Bit
This qwen2_vl model was trained 2x faster with Unsloth and Huggingface's TRL library.
Model Description
通义千问 QWEN OCR is a proprietary model focused on text extraction, specifically designed for extracting text from images of documents, tables, and payslips. The primary goal of this model is to extract COMPLETE/FULL text from images while ensuring that no information is missed.
Qwen2-VL-2B-OCR is a fine-tuned variant of unsloth/Qwen2-VL-2B-Instruct, optimized specifically for Optical Character Recognition (OCR). This model is trained to extract full and complete text from images, with a focus on documents such as payslips, invoices, and tables. The model aims to provide accurate text extraction with minimal loss of information, ensuring that every detail is captured.
This model uses cutting-edge techniques for text-to-text generation from images and works seamlessly for various OCR tasks, including text from complex documents with structured layouts.
Intended Use
The primary purpose of the model is to extract data from images or documents, especially from payslips and tables, without missing any critical details. It can be applied in various domains such as payroll systems, finance, legal document analysis, and any field where document extraction is required.
Prompt Example:
text: The model will BEST WORK to this "Extract all text from image/payslip without miss anything".
Model Benchmark
Benchmark Results
Benchmark
InternVL2-2B
MiniCPM-V 2.0
Qwen2-VL-2B
MMMUval
36.3
38.2
41.1
DocVQAtest
86.9
-
90.1
InfoVQAtest
58.9
-
65.5
ChartQAtest
76.2
-
73.5
TextVQAval
73.4
-
79.7
OCRBench
781
605
794
MTVQA
-
-
20.0
VCRen easy
-
-
81.45
VCRzh easy
-
-
46.16
RealWorldQA
57.3
55.8
62.9
MMEsum
1876.8
1808.6
1872.0
MMBench-ENtest
73.2
69.1
74.9
MMBench-CNtest
70.9
66.5
73.5
MMBench-V1.1test
69.6
65.8
72.2
MMT-Benchtest
-
-
54.5
MMStar
49.8
39.1
48.0
MMVetGPT-4-Turbo
39.7
41.0
49.5
HallBenchavg
38.0
36.1
41.7
MathVistatestmini
46.0
39.8
43.0
MathVision
-
-
12.4
After fine-tuning, this model has significantly improved in extracting all relevant sections from the payslip, including the previously missing Deductions section.
Example Output Comparison
image/png
Fine-tuned Model:
Here is the extracted data from the payslip:
Employee Information:
Date of Joining: 2018-06-23
Pay Period: August 2021
Employee Name: Sally Harley
Designation: Marketing Executive
Department: Marketing
Earnings:
Earnings
Amount
Deductions
Amount
Basic
10000
Provident Fund
1200
Incentive
1000
Professional Tax
500
House Rent
400
Loan
400
Meal Allowance
200
9500
Total Earnings: $11,600 Total Deductions: $2,100 Net Pay: $9,500
Employer Signature Employee Signature
Original Model:
The original model extracted the following data but missed the Deductions section:
Date of Joining: 2018-06-23
Pay Period: August 2021
Employee Name: Sally Harley
Designation: Marketing Executive
Department: Marketing
Earnings:
Basic: $10,000
Incentive Pay: $1,000
House Rent Allowance: $400
Meal Allowance: $200
Total Earnings: $11,600
Total Deductions: $2,100
Net Pay: $9,500
Employer Signature: [Signature]
Employee Signature: [Signature]
This is system-generated payslip
Quick Start
Here’s an example code snippet to get started with this model:
1# Load your image2from PIL import Image
3image_path ="xxxxx"# Replace with your image path4image = Image.open(image_path)
Preparing the Model, Preprocessing Inputs, and Performing Inference
python
1import requests
2import torch
3from torchvision import io
4from typing import Dict
56model = model.to("cuda")7conversation =[8{9"role":"user",10"content":[11{12"type":"image",13},14{15"type":"text",16"text":"extract all data from this payslip without miss anything"17}18]19}20]212223# Preprocess the inputs24text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)25# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\n<|im_start|>assistant\n'2627inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")28inputs = inputs.to('cuda')2930# Inference: Generation of the output31output_ids = model.generate(**inputs, max_new_tokens=2048)32generated_ids =[output_ids[len(input_ids):]for input_ids, output_ids inzip(inputs.input_ids, output_ids)]33output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)34print(output_text)
Handling CUDA Memory Issues During Inference
If you encounter CUDA memory issues during model inference, a common solution is to resize the input image to reduce its size. This helps in reducing the memory footprint and allows the model to process the image more efficiently.
python
1# Resize the image to reduce its size (e.g., scale to half its original size)2image = image.resize((image.width //2, image.height //2))
Model Fine-Tuning Details
The model was fine-tuned using the Unsloth framework, which accelerated training by 2x using Huggingface's TRL (Training Reinforcement Learning) library. LoRA (Low-Rank Adaptation) was applied to fine-tune only a small subset of the parameters, which significantly reduces training time and computational resources. Fine-tuning focused on both vision and language layers, ensuring that the model could handle complex OCR tasks efficiently.
Total Trainable Parameters: 57,901,056
Hardware Requirements
To run this model, it is recommended to have access to a GPU with at least 16 GB of VRAM. Training requires significant memory, so smaller batch sizes or gradient accumulation may be necessary for GPUs with less memory.
Model Architecture
If you'd like to learn more about the model's architecture and its detailed specifications, you can view the source page on Hugging Face at the following link:
