You can deploy this model and start earning money today!

lamao-ab/
paligemma-blind-assist-lora-merged-v1

Name: paligemma-blind-assist-lora-merged-v1
Availability: InStock

google/paligemma-3b-mix-224

adapter

gemma

text-generation-inference

endpoints_compatible

Views

No views yet

👁️ PaliGemma-BlindAssist: A parameter-efficient vision-language model adapter for blind assistance VQA & Captioning using LoRA

🎯 Model Overview

A full-precision (bfloat16) adaptation of google/paligemma-3b-mix-224 for accessibility vision-language tasks, trained with LoRA on the VizWiz datasets and merged into a standalone checkpoint. It performs visual question answering and scene captioning for blind and low-vision users, and represents the quality ceiling of our adaptation: the 4-bit QLoRA variant matches it to within a fraction of a point at 40% of the memory footprint, and is the variant intended for on-device deployment.

Adaptation improves the target task substantially while leaving general-domain performance largely intact. On VizWiz captioning, CIDEr-D rises from 55.31 to 98.08 and mean caption length from 5.03 to 10.30 words, matching the dataset's more descriptive reference style; VizWiz-VQA accuracy rises from 73.95 to 75.80. General-domain performance declines only modestly, by 0.50 points on VQAv2 and 5% relative on COCO-Caps.

🔧 Specifications

Base: google/paligemma-3b-mix-224 (bfloat16)
Adapter: LoRA, rank 8, alpha 16, dropout 0.05 · 11.3 M trainable params (0.385 %)
Target modules: q/k/v/o_proj, gate/up/down_proj
Data: 137,678 train / 27,569 val · seed 123
Schedule: 3 epochs, 3,225 steps · effective batch 128 (16 × 8 grad-accum) · max seq 512
Optimiser: adamw_torch_fused · LR 2e-4, cosine (warmup 0.03)
Parameters: 2.92 B (adapter merged into the base weights)
Precision: bf16 (merged, standalone)
In-memory size: 5.45 GB (model.get_memory_footprint())
Training peak VRAM: 53.1 GB

📊 Measured Performance

Blind-domain (target task) — VizWiz

VizWiz-VQA

Model	Overall	Yes/No	Number	Other	Unans.
Base	73.95	88.71	66.10	66.10	91.67
LoRA	75.80 ±0.21	86.55	67.56	66.54	97.73

VizWiz-Caps

Model	CIDEr-D	BLEU-4	METEOR	ROUGE-L	Len.
Base	55.31	12.07	14.08	28.76	5.03
LoRA	98.08 ±1.51	30.77 ±0.36	23.45 ±0.31	49.54 ±0.47	10.30

General-domain (control) — VQAv2 / COCO-Caps

VQAv2

Model	Overall	Yes/No	Number	Other
Base	81.65	94.74	67.53	73.49
LoRA	81.15 ±0.04	94.57	66.38	72.85

COCO-Caps

Model	CIDEr-D	BLEU-4	METEOR	ROUGE-L	Len.
Base	131.21	31.96	30.62	59.17	12.40
LoRA	124.54 ±1.36	34.58 ±0.39	30.63 ±0.06	58.62 ±0.19	11.20

_{LoRA/QLoRA scores are mean ± std over 3 seeds (42, 123, 7). This checkpoint is the seed-123 run.}

🚀 Usage

python

from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch

repo = "lamao-ab/paligemma-blind-assist-lora-merged-v1"
model = PaliGemmaForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = PaliGemmaProcessor.from_pretrained(repo)

image = Image.open("example.jpg").convert("RGB")
prompt = "describe the scene for a blind person"  
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

📄 License

Inherits the Gemma license from the base model.