
Vision encoder	`google/siglip2-base-patch16-224` (frozen)
LLM	`Qwen/Qwen2.5-0.5B-Instruct` (LoRA: q,k,v,o,gate,up,down)
Visual tokens	49 per image (14x14 patches, 2x2 pixel-shuffle)
Image placeholder	`<
Stage 1	projector alignment on Flickr8k + Flickr30k captions
Stage 2	LoRA instruction tuning on VQAv2 + LLaVA-ReCap + captions

Answer length is prompt-controlled

Stage 2 mixes one-word VQA answers with multi-sentence descriptions, and VQA examples carry an explicit hint. Append it for short answers, omit it for prose:

What color is the bus?
Answer the question using a single word or phrase.     -> "red"

What color is the bus?                                 -> "The bus is red."

Files

projector.pt -- projector weights (pool_stride is recoverable from the tensor shapes)
lora_adapter/ -- PEFT adapter for the LLM

Usage

Needs the companion code from the training repo (Model/model.py, Model/dataset.py, Inference/inference.py):

python

from huggingface_hub import hf_hub_download, snapshot_download
from inference import load_model, answer

projector = hf_hub_download(repo_id="dhruvpatel93/tinyvlm-vqa", filename="projector.pt")
lora = snapshot_download(repo_id="dhruvpatel93/tinyvlm-vqa", allow_patterns=["lora_adapter/*"]) + "/lora_adapter"

model = load_model(projector, lora, device="cuda")
print(answer(model, "photo.jpg", "What is in this image?"))
print(answer(model, "photo.jpg", "What color is the car?", short_answer=True))

Limitations

224x224 input, 49 visual tokens, and a 0.5B LLM. Expect everyday-scene captioning and simple VQA -- not OCR, fine detail, counting, or multi-step visual reasoning.

Training

optimizer steps: 10518
final held-out loss: 0.9565
batch size 4 x 4 accum, lr 0.0002