You can deploy this model and start earning money today!

Abhiram1404/
qwen2vl-doc-ocr-lora

Name: qwen2vl-doc-ocr-lora
Availability: InStock

transformers

safetensors

text-generation-inference

unsloth

qwen2_vl

trl

unsloth/Qwen2-VL-7B-Instruct-unsloth-bnb-4bit

finetune

apache-2.0

endpoints_compatible

Views

No views yet

qwen2vl-doc-ocr-lora

A LoRA adapter for Qwen2-VL-7B-Instruct fine-tuned for document OCR — given a scanned document image, it outputs the extracted text in reading order.

Developed by: Abhiram1404
License: apache-2.0
Finetuned from: unsloth/Qwen2-VL-7B-Instruct-unsloth-bnb-4bit
Task: image-to-text (OCR) via a fixed instruction prompt: "Extract all the text from this document image, preserving line order."

This qwen2_vl model was trained 2x faster with Unsloth and TRL's SFTTrainer.

Training data

Trained on real scanned documents from RVL-CDIP (38.7 GB tarball, 16 document classes — letter, form, invoice, resume, memo, etc.) paired with real per-image OCR annotations from jordyvl/rvl_cdip_easyocr (EasyOCR word/box outputs, reconstructed into reading-order text lines).

Caveat: RVL-CDIP is built from 1990s tobacco-litigation document scans — real scan artifacts, but era-appropriate layouts (e.g. its "resume" class looks typewriter/dot-matrix-era, not like a modern resume). Ground truth is EasyOCR's own output, not human transcription, so it inherits EasyOCR's error rate on these low-quality scans.

Training procedure

Method: LoRA via FastVisionModel.get_peft_model, r=16, lora_alpha=16, lora_dropout=0, target_modules="all-linear", adapters on both vision and language layers (attention + MLP).
Base model: loaded in 4-bit (load_in_4bit=True) with "unsloth" gradient checkpointing.
Trainer: TRL SFTTrainer with UnslothVisionDataCollator, batch size 1, gradient accumulation 8 (effective batch size 8), 120 steps, learning_rate=2e-4, linear schedule with 5 warmup steps, adamw_8bit, bf16/fp16 mixed precision, max_seq_length=2048.
Data: ~2,000 training examples / 100 eval examples sampled from a pooled subset of RVL-CDIP images with usable OCR text (≥15 words).

How to use

python

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "Abhiram1404/qwen2vl-doc-ocr-lora",
    load_in_4bit=True,
)
FastVisionModel.for_inference(model)

instruction = "Extract all the text from this document image, preserving line order."
messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": instruction}]}]
input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
inputs = tokenizer(image, input_text, add_special_tokens=False, return_tensors="pt").to("cuda")

output_ids = model.generate(**inputs, max_new_tokens=512, temperature=0.2, do_sample=True)
print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
  