from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
import torch

processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

base_model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-2B-Instruct",
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(
    base_model,
    "bharath-0201/omni-shield-qwen2vl-pii-adapter"
)
model = model.merge_and_unload()
model.eval()

Key finding: label consistency

Fine-tuning teaches the model to emit structured LABEL | VALUE pairs consistently. The base model detects values correctly but frequently omits field labels, causing downstream pipeline failures. Fine-tuning fixes this without degrading value detection accuracy.

Example (German ID card):

Field	Base model output	Fine-tuned output
Name	`Name \| Leonie Dörr`	`Name \| Leonie Dörr`
DOB	`Date of Birth \| 01.08.1991`	`Date of Birth \| 01.08.1991`
ID Number	`[empty] \| M898585166`	`ID No. \| M898585166`
Signature	`Signature \| [none]`	`Signature \| [none]`

Citation

bibtex

@article{{omni-shield-2025,
  title={{Omni-Shield: Local-First Multi-Modal PII Redaction
         with Cryptographic Audit Trail}},
  author={{[Anonymous for review]}},
  year={{2025}}
}}

Paper

Submitted to PETS 2026. arXiv preprint forthcoming.

Field	Base model output	Fine-tuned output
Name	`Name \| Leonie Dörr`	`Name \| Leonie Dörr`
DOB	`Date of Birth \| 01.08.1991`	`Date of Birth \| 01.08.1991`
ID Number	`[empty] \| M898585166`	`ID No. \| M898585166`
Signature	`Signature \| [none]`	`Signature \| [none]`