Part of the edge-vlm-extraction project — a methodology for fine-tuning and honestly measuring a vision-language model under edge hardware constraints.

Results — before vs after (same 40 held-out test examples)

Metric	Base model	Fine-tuned	Δ
Valid JSON rate	7.5%	100%	+92.5 pts
Field exact-match	0%	6.2%	+6.2 pts
Menu-item recall	0%	7.5%	+7.5 pts

Reading the numbers honestly. The jump to 100% valid JSON is the core result. Field exact-match and recall remain low because 3 epochs on 600 examples teaches structure, not value memorisation. Both numbers are reported deliberately.

Training dynamics. Loss: 1.69 → 0.16 over 220 steps. Under 45 min on RTX 4090.

Training setup

Base model: Qwen/Qwen2-VL-7B-Instruct
Method: QLoRA — 4-bit NF4, LoRA on attention + MLP projections
Trainable parameters: ~40M / 8.33B (0.48%)
Dataset: CORD-v2, 600 train examples, 3 epochs
Hardware: 1x RTX 4090 (24 GB VRAM)

Usage

python

from peft import PeftModel
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor

base = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "saliousk/qwen2vl-cord-lora")
processor = AutoProcessor.from_pretrained("saliousk/qwen2vl-cord-lora")

saliousk/qwen2vl-cord-lora

qwen2vl-cord-lora

Results — before vs after (same 40 held-out test examples)

Training setup

Usage

Links

saliousk/
qwen2vl-cord-lora