LoRA adapter for a compact generative Vision-Language Model (unsloth/Qwen3.5-0.8B) that extracts Pallet Load Manifests from document images directly into a strict JSON schema — without a classical OCR pipeline. Trained as part of a Master's thesis at Fachhochschule Südwestfalen.

Kurzfassung (DE): Überwacht (LoRA-SFT) feinjustierter, kompakter VLM-Adapter zur schemakonformen Extraktion von Paletten-Manifesten aus Dokumentenbildern. Ende-zu-Ende (Bild + Prompt → JSON), ohne klassische Texterkennung (OCR).

Model at a glance

Base model: unsloth/Qwen3.5-0.8B (4-bit NF4 quantised)
Adapter type: LoRA (PEFT), rank r = 16, alpha = 16, dropout 0
Target modules: vision + language, attention + MLP (full four-block coverage)
Task: image → structured JSON (image-text-to-text)
Domain: logistics / supply-chain document understanding

Results (test split, n = 10)

Comparison of the frozen base model vs. this LoRA adapter, same prompt and evaluation protocol for both.

Metric	Base (zero-shot)	LoRA (this adapter)
WES (Weighted Extraction Score)	0.049	0.996
Schema compliance	0 %	100 %
CER (full JSON, lower=better)	0.672	0.013
Exact match	0 / 10	9 / 10
Field accuracy	0.000	0.996
Products list-F1	0.00	1.00
ROUGE-L-F1	0.58	0.99
BERTScore-F1	0.815	0.996
LLM-judge (GPT-4o+4.1 ensemble, 1–5)	1.36	4.96

The main effect is structural: the base model already reads part of the field values but serialises them into an unusable JSON envelope (schema compliance 0 %). The adapter fixes the serialisation into the binding schema — schema compliance rises to 100 % and the character error rate drops by ~50×.

Note on scope: All figures are point estimates over a small test split (n = 10) on a synthetically homogeneous layout. They characterise this domain; transfer to other layouts, real photographs or foreign schemas has not been evaluated.

Intended use

Convert single-page pallet-load-manifest images into schema JSON (shipment_id, asn_number, dock, route, products[], totals{}).
On-premise, privacy-preserving alternative to API frontier models: the training run takes ≈ 27 min on a single A100 and ≈ 24 GB VRAM, producing a reusable adapter.

Out of scope

Multi-page documents, layouts/schemas outside the training domain, and any use where a wrong field silently propagates without human/downstream validation.

How to use

python

from unsloth import FastVisionModel
from peft import PeftModel

model, processor = FastVisionModel.from_pretrained(
    "unsloth/Qwen3.5-0.8B", load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "NghonseFiassap/Qwen3.5-0.8B-pallet-manifest-lora")
FastVisionModel.for_inference(model)

# messages = [{"role": "user", "content": [
#     {"type": "image", "image": image},
#     {"type": "text", "text": prompt},
# ]}]
# inputs = processor.apply_chat_template(messages, add_generation_prompt=True,
#                                        tokenize=True, return_tensors="pt").to("cuda")
# out = model.generate(**inputs, max_new_tokens=1024)

The recommended production prompt is the schema-guided variant (P2): it is the strongest prompt on both models and is robust after fine-tuning (WES spread across five prompt variants drops from ≈ 0.85 to ≈ 0.002).

Training

Framework: Unsloth + TRL SFTTrainer
Optimiser: adamw_8bit (QLoRA), learning rate 1e-4, linear schedule, warmup 20
Steps: max_steps = 70 (< 1 epoch equivalent, deliberate short run against memorisation)
Effective batch size: 16 (4 × 4 gradient accumulation), max_length = 2048
Seed: 3407
Data: Roboflow pallet load manifest json (CC BY 4.0), 150 training documents

Modular ablation (where the gain sits)

A six-configuration sweep (identical rank, steps and protocol) localises the gain in the language layers: language-only LoRA replicates the full run to within 0.002–0.004 WES points, while vision-only configurations stay at schema 0 % — one vision variant even regresses the CER (defect mode only visible through schema-aware metrics).

Limitations & responsible use

Small evaluation set (n = 10): point estimates, no confidence intervals.
Synthetic, homogeneous layout → residual memorisation risk.
No direct comparison against API frontier models on the same test set.
Always validate critical fields downstream before ERP ingestion.

Citation

Master's thesis, Fachhochschule Südwestfalen (2026): Strukturierte Informationsextraktion aus Dokumentenbildern mit kompakten generativen Vision-Language-Modellen.

License

Adapter released under apache-2.0. Please also observe the license of the base model unsloth/Qwen3.5-0.8B and of the training dataset (CC BY 4.0).