The model was fine-tuned on a cleaned KORIE receipt SFT dataset using assistant-only JSON loss. The previous training path used TRL VLM messages format, which computed loss over prompt and image tokens as well as the assistant answer. This v4 run switches to prompt + completion format with completion_only_loss=True.

Intended Output

The adapter is intended to return one strict JSON object with receipt fields:

json

{
  "schema_version": "korie_receipt_ie.v2",
  "merchant_name": null,
  "merchant_address": null,
  "merchant_phone_number": null,
  "receipt_number": null,
  "transaction_date": null,
  "transaction_time": null,
  "subtotal": null,
  "total_tax": null,
  "total_discount": null,
  "tip": null,
  "total": null,
  "currency": "KRW",
  "item_barcodes": [],
  "items": [],
  "field_status": {"missing": [], "uncertain": []}
}

Training Summary

Base model: Qwen/Qwen2.5-VL-3B-Instruct
Method: QLoRA SFT
Dataset: cleaned KORIE receipt SFT v3
Train samples: 408
Validation samples: 135
Epochs: 2
LoRA rank: 32
LoRA alpha: 64
LoRA dropout: 0.05
Target modules: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
Completion-only loss: true

Final training metrics:

train loss: 0.1270
validation loss: 0.0979
validation mean token accuracy: 0.9739

Small Generation Evaluation

Same 5 KORIE validation samples used for quick comparison: IMG00016, IMG00018, IMG00025, IMG00028, IMG00048.

Model / decoding	Valid JSON	Scalar field accuracy
Base model + strict prompt	0.80	0.600
Previous public LoRA + strict prompt	1.00	0.667
v3 incorrect full-message loss	1.00	0.583
v4, repetition penalty 1.15	1.00	0.683
v4, repetition penalty 1.0	1.00	0.750

Recommended deterministic decoding for this adapter:

do_sample=False
repetition_penalty=1.0
max_new_tokens=2048

The 5-sample evaluation is a smoke test, not a production benchmark. Remaining weak fields are subtotal, total_tax, receipt number disambiguation, and phone-number formatting. OCR text injection and deterministic post-processing are recommended for higher production accuracy.

Files

receipt_vlm_training_summary.json: final training metrics
eval/korie_val5_v4_lora_rp100_eval.json: best quick generation eval
eval/korie_val5_v4_lora_rp115_eval.json: comparison eval
dataset/dataset_report.json: cleaned dataset report

Loading

python

from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

base = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter = "hoin1218/receipt-qwen25vl-3b-korie-v4-lora"

processor = AutoProcessor.from_pretrained(base)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base,
    device_map="auto",
    torch_dtype="auto",
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()