rico-blip-lora
This model is a fine-tuned version of Salesforce/blip-image-captioning-base on the RICO Screen2Words dataset for the task of generating textual descriptions of mobile UI screenshots.
The model adapts the BLIP vision-language architecture to better understand the layout and semantics of mobile application interfaces.
Model Description
This project performs multimodal fine-tuning of the BLIP image captioning model on mobile UI screenshots.
The base model Salesforce/blip-image-captioning-base is a vision-language transformer capable of generating captions from images. However, it is primarily trained on natural images.
To adapt the model to the domain of mobile user interfaces, it was fine-tuned on the RICO Screen2Words dataset, which contains screenshots of mobile apps paired with textual descriptions.
Fine-tuning was performed using LoRA (Low-Rank Adaptation), a parameter-efficient technique that allows training a small number of additional parameters while keeping the majority of the base model weights frozen.
This allows efficient training on limited hardware such as a single T4 GPU.
Intended Use
This model is intended for:
- Automatic captioning of mobile application screenshots
- UI understanding tasks
- Multimodal research involving vision-language models
- Demonstrating multimodal fine-tuning with small language models (SLMs)
Dataset
The model was fine-tuned on the RICO Screen2Words dataset, which contains mobile application screenshots paired with textual descriptions of the screen content.
Dataset link:
Dataset characteristics:
- Image: Mobile UI screenshot
- Caption: Natural language description of the interface
The dataset enables training models to understand app layouts, menus, and interface components.
Training Procedure
Fine-tuning was performed using LoRA (Low-Rank Adaptation) via the PEFT library.
Instead of updating all model parameters, LoRA introduces small trainable matrices into attention layers. This significantly reduces the number of trainable parameters and allows efficient training.
The model was trained using Hugging Face Transformers on T4 GPU compute.
Training Hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: AdamW (fused)
- lr_scheduler_type: linear
- num_epochs: 2
- mixed_precision_training: Native AMP
Training Results
| Training Loss | Epoch | Step | Validation Loss |
|---|
| 8.1654 | 0.2823 | 500 | 8.1154 |
| 7.9349 | 0.5647 | 1000 | 7.9267 |
| 7.8727 | 0.8470 | 1500 | 7.8694 |
| 7.8471 | 1.1293 | 2000 | 7.8467 |
| 7.8374 | 1.4116 | 2500 | 7.8362 |
| 7.8324 | 1.6940 | 3000 | 7.8311 |
| 7.8319 | 1.9763 | 3500 | 7.8293 |
How to Run Inference
The model can be loaded directly from the Hugging Face Hub.
1from transformers import BlipProcessor, BlipForConditionalGeneration
2from PIL import Image
3import requests
4
5model_name = "PMN23/rico-blip-lora"
6
7processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
8model = BlipForConditionalGeneration.from_pretrained(model_name)
9
10url = "https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg"
11image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
12
13inputs = processor(image, return_tensors="pt")
14
15out = model.generate(**inputs)
16
17print(processor.decode(out[0], skip_special_tokens=True))
Merging LoRA Adapters with Base Model
If the uploaded model contains LoRA adapters, they can be merged with the base model before inference.
1from transformers import BlipForConditionalGeneration
2from peft import PeftModel
3
4base_model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
5
6model = PeftModel.from_pretrained(base_model, "PMN23/rico-blip-lora")
7
8model = model.merge_and_unload()
Framework Versions
- PEFT 0.18.1
- Transformers 5.0.0
- PyTorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
Repository
Full code, training notebooks, and documentation are available in the GitHub repository associated with this project.
