You can deploy this model and start earning money today!

Rishabh3365/
smolvlm2-blv-500m

Name: smolvlm2-blv-500m
Availability: InStock

Views

No views yet

SmolVLM2-BLV: Fine-Tuned Vision-Language Model for Blind & Low-Vision Navigation Assistance

GitHub

arXiv

Fine-tuned from SmolVLM2-500M via an SFT → GRPO → SFT pipeline to generate real-time, spatially-grounded navigation captions for blind and low-vision (BLV) users — deployed on-device (Samsung Galaxy A55) via GGUF/llama.cpp.

Overview

Existing captioning models describe what is in a scene but rarely give BLV users what they actually need: concise, actionable, hazard-aware guidance for moving safely through an environment. This model adapts SmolVLM2-500M, a lightweight vision-language model, specifically for real-time navigation assistance — balancing caption usefulness against the latency constraints of running fully on a phone.

Figure 1. SmolVLM2-BLV Architecture: Multimodal Fusion for Navigation Captioning

Method

Training pipeline: SFT → GRPO → SFT

Stage 1 — SFT: Initial supervised fine-tuning on BLV-navigation captioning data to teach the base task format and domain vocabulary.
Stage 2 — GRPO: Reinforcement learning (Group Relative Policy Optimization) to reward captions for hazard awareness, spatial precision, and actionability rather than generic descriptiveness.
Stage 3 — SFT patch: A final supervised pass to correct regressions/artifacts introduced during GRPO and stabilize output formatting.

Deployment: Quantized to GGUF and run via llama.cpp for real-time, on-device inference on Android (tested on Samsung Galaxy A55).

Model Details


Base model	SmolVLM2-500M-Video-Instruct
Parameters	0.5B
Precision (this repo)	Unquantized (F16/BF16 safetensors)
Quantized variant	[Link to your GGUF repo, if uploaded separately]
Task	Image/Video-Text-to-Text (navigation captioning)
Training pipeline	SFT → GRPO → SFT
Deployment target	Android via GGUF + llama.cpp
License	Apache 2.0

Results

Replace with your actual eval numbers — MCF/NAF framework scores, LLM-judge (Qwen2.5-32B) comparisons, GPT-4o baseline comparison, etc.

Model	Hazard Coverage	Actionability	Human Eval	Latency
Base SmolVLM2-500M	–	–	–	–
GPT-4o (baseline)	–	–	–	–
Ours (SFT→GRPO→SFT)	–	–	–	–

On-device (GGUF / llama.cpp)

bash

# See [GGUF repo link] for the quantized checkpoint used in the Android deployment
./llama-mtmd-cli -m smolvlm2-blv-500m-q4_k_m.gguf --image scene.jpg -p "Describe the path ahead."

Limitations

[Hazard-coverage limitation — add your camera-ready writeup summary here]
Trained/evaluated primarily on [dataset domain] — may not generalize to unseen environments (e.g., outdoor/crowded scenes) without further fine-tuning.
Latency and accuracy trade-offs differ between the unquantized and GGUF-quantized variants; see [GGUF repo] for on-device benchmarks.

Contact

[r33choudhary@gmail.com]

Rishabh3365/smolvlm2-blv-500m