RICO BLIP LoRA Model
Model Description
This model is a BLIP image captioning model fine-tuned using LoRA (Low Rank Adaptation) on the RICO Screen2Words dataset.
The goal of the model is to generate textual descriptions of mobile user interface screenshots. The model takes an image of a UI screen as input and produces a caption describing the interface.
Model Details
- Developed by: suhas2468
- Model type: Vision-Language Model (Image Captioning)
- Language(s): English
- License: Apache 2.0
- Finetuned from model: Salesforce/blip-image-captioning-base
Model Sources
Intended Use
Direct Use
This model can be used to generate captions describing mobile UI screenshots. It can help automatically summarize UI layouts and components present in mobile application interfaces.
Example use cases include:
- UI understanding
- UI documentation
- Automated interface description
Out-of-Scope Use
The model is not designed for:
- general image captioning outside UI screenshots
- safety-critical applications
- decision-making systems
How to Use the Model
Example inference code:
1from transformers import pipeline
2
3pipe = pipeline(
4 "image-to-text",
5 model="suhas2468/rico-blip-lora-model"
6)
7
8result = pipe("example_image.png")
9print(result)
Training Details
Training Data
The model was fine-tuned using the RICO Screen2Words dataset.
This dataset contains mobile UI screenshots paired with captions describing the interface.
Training Setup
- GPU: NVIDIA Tesla T4
- Epochs: 1
- Batch size: 4
- Learning rate: 2e-5
- Training samples: 500
Training was performed using Hugging Face Transformers, PEFT, and PyTorch.
Training Method
The model was fine-tuned using LoRA (Low Rank Adaptation).
LoRA enables efficient fine-tuning by adding small trainable adapters to the base model while keeping most of the original model parameters frozen. This reduces GPU memory usage and speeds up training.
Limitations
- The model was trained on a relatively small subset of the dataset (500 samples).
- Performance may vary for UI layouts not represented in the training data.
- The model is primarily intended for educational and research purposes.
Hardware
Training was performed on:
- GPU: NVIDIA Tesla T4
- Platform: Google Colab
Additional Resources
GitHub repository containing the training notebook and code:
