LLaVA-via is a PEFT LoRA adapter fine-tuned on top of LLaVA-1.5-7B to improve scene narration and navigation guidance for assistive vision-language applications.

The objective of this work is to generate contextually grounded, coherent, and navigation-aware responses that can support visually impaired assistance scenarios. Rather than answering general visual questions, the adapter is optimized to describe surrounding environments and provide guidance-oriented information from input images.

This repository contains only the LoRA adapter and associated tokenizer/processor configuration files. It does not contain the full LLaVA model weights.

Base Model

This adapter must be loaded on top of the following Hugging Face model:

Base Model

llava-hf/llava-1.5-7b-hf

The base model is not included in this repository.

Repository Contents

This repository includes:

adapter_model.safetensors — Fine-tuned LoRA adapter weights
adapter_config.json — PEFT configuration
Tokenizer files
Processor configuration
Image preprocessor configuration
Chat template
Model card (README)

Training checkpoints, optimizer states, and scheduler files are intentionally excluded.

Intended Use

LLaVA-via is intended for research and development involving assistive vision-language systems, including:

Scene narration
Environmental understanding
Navigation guidance
Assistive AI research
Vision-language model benchmarking

Example applications include:

Assistive mobile applications
Smart glasses
Vision-language research
Accessibility-focused AI systems
Human-centered AI

Fine-Tuning Method

Architecture: LLaVA-1.5-7B
Fine-tuning: PEFT LoRA
Task: Vision-Language Conditional Generation
LoRA Rank (r): 16
LoRA Alpha: 32
Target Modules: q_proj, v_proj
LoRA Dropout: 0.05

Evaluation

The fine-tuned adapter was evaluated using multiple automatic metrics covering lexical similarity, semantic similarity, vision-language alignment, and hallucination analysis.

Metric	Score
BLEU-1	0.3125
BLEU-2	0.1842
BLEU-3	0.1228
BLEU-4	0.0860
METEOR	0.3165
ROUGE-L	0.2755
BERTScore-F1	0.8872
SBERTScore	0.7116
CLIPScore (ViT-B/32)	0.2839
CLIPScore (ViT-B/16)	0.2329
CLIPScore (ViT-L/14)	0.1828
PickScore (CLIP-ViT-H-14)	0.2030
SPECS-VIA Score	3.2770 / 5
PickScore-VIA	0.8132
CHAIR-i (Semantically Refined)	0.410

The evaluation includes lexical overlap metrics (BLEU, METEOR, ROUGE-L), semantic similarity metrics (BERTScore and SBERTScore), vision-language alignment metrics (CLIPScore and PickScore), a task-specific evaluation (SPECS-VIA and PickScore-VIA), and hallucination analysis using the semantically refined CHAIR-i metric.

Loading the Adapter

The adapter can be loaded using the Hugging Face Transformers and PEFT libraries.

python

from transformers import AutoProcessor, LlavaForConditionalGeneration
from peft import PeftModel

base_model = LlavaForConditionalGeneration.from_pretrained(
    "llava-hf/llava-1.5-7b-hf"
)

processor = AutoProcessor.from_pretrained(
    "llava-hf/llava-1.5-7b-hf"
)

model = PeftModel.from_pretrained(
    base_model,
    "SobanHM/LLaVA-via"
)

Example Use Cases

The adapter is designed for prompts such as:

Describe the surrounding environment.
Identify important objects in the scene.
Provide navigation guidance.
Describe possible obstacles.
Explain the scene for a visually impaired user.

Limitations

This repository provides a fine-tuned PEFT LoRA adapter evaluated using offline automatic metrics.

The reported evaluation does not include:

Real-time deployment on electronic white canes
Smart glasses deployment
Mobile-device latency evaluation
Embedded hardware benchmarking
User studies involving visually impaired participants

Accordingly, inference latency, energy consumption, real-time responsiveness, and human-subject usability remain outside the scope of the reported experiments.

Citation

If you use this repository in your research, please cite this work and the original LLaVA and PEFT publications.

A BibTeX entry for this work may be added once the associated thesis or publication becomes available.

Authors

Soban Hussain Department of Computer Science Sukkur IBA University

Praih Alias Faiza Department of Computer Science Sukkur IBA University

Tasmia Department of Computer Science Sukkur IBA University

Acknowledgements

This work builds upon the following open-source projects:

LLaVA
Hugging Face Transformers
PEFT
PyTorch

We gratefully acknowledge the developers and research community whose contributions made this work possible.