Complete training output from fine-tuning Qwen/Qwen2-VL-7B-Instruct to detect nutrition tables in product images: LoRA adapters, per-stage checkpoints, merged weights, and full optimizer/scheduler/RNG state for both stages.

Serving this model? Use MayaKD/qwen2-vl-7b-nutrition-vllm instead. Weights here live in subfolders, which vLLM cannot load directly (there is no subfolder= parameter). That repo is a flat copy of the final merged weights, and carries the full model card, usage examples, benchmarks, and limitations.

What is here

exp13_mod121925_full_visio/     Stage 1 — full vision encoder
├── adapter_model.safetensors   LoRA adapter (43 MB)
├── checkpoint-102/             Trainer state: optimizer, scheduler, RNG, tokenizer
└── merged/                     Stage 1 adapter merged into base (16.6 GB)

exp13_mod121925_joint/          Stage 2 — joint vision + language
├── adapter_model.safetensors   LoRA adapter (205 MB)
├── checkpoint-102/             Trainer state: optimizer, scheduler, RNG, tokenizer
└── merged/                     Final model, adapter merged into base (16.6 GB)

exp13_mod121925_joint/merged/ is the shipped model. Stage 2 begins from stage 1's merged/ directory, not from stacked adapters.

Training recipe

LoRA (r=16, α=32) applied to the frozen base in two stages, merging between them so each stage starts from clean full weights.

Stage	Targets	Epochs	Batch × grad-accum	LR
1 — Full vision	All vision blocks (attn + MLP) + merger MLP	6	2 × 16	1e-4
2 — Joint	Vision + LLM `q/k/v/o_proj`, `gate/up/down_proj`	6	1 × 32	1e-5

SFTTrainer + accelerate on 2× RTX Pro 6000. BF16, Flash Attention 2, fused AdamW. Best checkpoint by eval_loss, early stopping patience 3. Image resolution min_pixels=784, max_pixels=705600; sequence length is intentionally uncapped, since truncation corrupts image tokens.

An optional vision-warmup stage (last 8 vision blocks only) was evaluated and dropped — it added training time without measurable IoU gain once stage 2 targets the full vision encoder.

Results

Metric (123 val samples)	Value
Mean IoU	0.82	primary metric
Precision@0.5	≤ 0.91	upper bound
Recall@0.5	≤ 0.89	upper bound
F1@0.5	≤ 0.90	upper bound

The threshold metrics were computed without one-to-one matching, which overcounts true positives when a table draws duplicate detections; the removed bias is strictly upward, so true values sit at or below those shown. Mean IoU is unaffected and is the metric to use.

A three-stage schedule reached 0.893 mean IoU in exploratory runs, showing headroom above this checkpoint. Those weights were not preserved; the two-stage model published here is the reproducible artifact.

Note on multi-GPU merging

Merging LoRA adapters after distributed training silently corrupts weights unless the save is guarded. When every rank calls merge_and_unload() and save_pretrained() concurrently, the processes interleave writes to the same shards. The result loads without error but scores 0.300 IoU against 0.794 for the identical in-memory model.

The fix is to perform the merge and save only on accelerator.is_main_process, fenced by wait_for_everyone() barriers on both sides. All checkpoints in this repo were produced after that fix; pre-merge and post-merge evaluations agree within ~1%.

License

Apache-2.0, inherited from Qwen2-VL-7B-Instruct. Project code is MIT.

MayaKD/qwen2-vl-7b-nutrition

Qwen2-VL-7B Nutrition Table Detection — training artifacts

What is here

Training recipe

Results

Note on multi-GPU merging

Links

License

MayaKD/
qwen2-vl-7b-nutrition