Model Card for Model ID
This is a LoRA adapter fine-tuned on the Qwen2.5-VL-3B-Instruct base model for Optical Character Recognition (OCR) of mathematical formulas, converting images directly into LaTeX code.
Model Details
Model Description
This model is a Vision-Language Model (VLM) adapter fine-tuned to transcribe images of math equations into their corresponding LaTeX representation. It uses 4-bit QLoRA quantization to achieve efficient fine-tuning on a single consumer-grade GPU (e.g., Tesla T4).
- Developed by: ericmrib
- Model type: Vision-Language Model Adapter (LoRA)
- Language(s) (NLP): English, LaTeX
- Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct
Out-of-Scope Use
The model is highly specialized for LaTeX formula extraction and may not perform well on general image captioning, document parsing without equations, or natural language reasoning tasks outside its fine-tuned scope.
Bias, Risks, and Limitations
As with any vision-language model, the performance relies heavily on the quality, clarity, and handwriting style (if applicable) of the input images. Complex or highly stylized formatting might lead to inaccuracies in the generated LaTeX.
How to Get Started with the Model
Use the code below to get started with the model.
1from peft import PeftModel
2from transformers import AutoProcessor, AutoModelForImageTextToText
3
4# Load base model and processor
5base_model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", device_map="auto")
6processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
7
8# Load adapter
9model = PeftModel.from_pretrained(base_model, "ericmrib/math-ocr")
Training Details
Training Data
The model was fine-tuned on the unsloth/LaTeX_OCR dataset, which contains approximately 76,000 samples of image-to-LaTeX pairs.
Training Hyperparameters
- Training regime: fp16 mixed precision
- Quantization: 4-bit (QLoRA) with NF4
- Batch size: 1 (with 16 gradient accumulation steps; effective batch size = 16)
- Learning rate: 2e-4
- Optimizer: adamw_torch
- Scheduler: cosine
- Max Steps: 500
- LoRA configuration: r=16, alpha=32, dropout=0.05, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
Technical Specifications
Hardware
- Hardware Type: Tesla T4 (1x)
- Hours used: ~4.2 hours
- Cloud Provider: Google Cloud (Google Colab)
