Vision-Language Model for Automatic Mobile UI Screen Captioning
Overview
RICO UI Captioner is a vision-language model that automatically
generates natural language descriptions for mobile UI screenshots.
The system fine‑tunes the BLIP (Bootstrapping Language Image
Pre‑training) vision‑language model to understand UI layouts and
produce captions describing the screen content.
Possible uses:
UI accessibility
UI search engines
Automated UI documentation
Mobile design analysis
Model
Base model: BLIP (Bootstrapping Language Image Pre‑training)
The goal of this project is to generate natural language descriptions for mobile UI screenshots using a vision-language model.
The approach consists of the following steps:
Dataset Selection
The RICO Dataset was used as the primary dataset. It contains thousands of Android application screenshots along with UI hierarchy information.
Model Selection
The project uses the BLIP architecture, which is a vision-language transformer designed for image captioning tasks.
BLIP was chosen because it:
supports multimodal learning
performs well on caption generation
integrates easily with the **Transformers ecosystem.
Fine-Tuning Strategy
Instead of training the entire model from scratch, the project uses parameter-efficient fine-tuning with LoRA.
Benefits of LoRA:
reduces the number of trainable parameters
decreases GPU memory requirements
allows faster training
Training Process
The training pipeline involved:
Loading the pretrained BLIP model
Preparing image-caption pairs from the dataset
Applying LoRA adapters to attention layers
Training the model using the Hugging Face Trainer API
Saving the fine-tuned model
Uploading the model to Hugging Face
Training was performed using Google Colab with GPU acceleration.
Inference Pipeline
During inference:
The UI screenshot is passed to the processor
The image is encoded by the vision encoder
Cross-attention aligns visual features with text tokens
The decoder generates the caption sequentially
Assumptions
Several assumptions were made during the development of the model.
UI Screens Contain Recognizable Layout Patterns
The model assumes that UI screens follow common design patterns such as navigation bars, lists, buttons, and forms.
Screen-Level Captioning is Sufficient
The model generates captions for the entire screen, rather than describing each individual UI component.
Dataset Diversity is Adequate
The RICO dataset is assumed to contain sufficient variety in UI layouts and application types for the model to generalize.
Visual Context is Enough
The model assumes that meaningful captions can be generated from visual information alone, without additional metadata.
Observations
During training and inference, several observations were made.
The Model Learns UI Structure
The model is able to identify common UI elements such as:
navigation bars
lists
buttons
profile screens
and produce meaningful captions describing them.
Generic Captions for Complex Screens
For screens with many components, the model tends to produce more generalized captions rather than detailed descriptions.
Example:
Input: complex dashboard UI
Output:
“a mobile application interface with multiple options and icons”
Small Elements Are Hard to Detect
Very small UI elements such as icons or labels are sometimes ignored by the model.
Dataset Quality Affects Performance
The accuracy of captions strongly depends on the quality and diversity of training examples.
LoRA Significantly Reduces Training Cost
Using LoRA allowed fine-tuning the model with significantly fewer trainable parameters while maintaining good performance.
Summary
The project demonstrates that a pretrained vision-language model such as BLIP can be successfully adapted to the UI domain using parameter-efficient fine-tuning.
Despite limitations in detecting small elements, the model is able to generate meaningful captions for mobile UI screens and shows potential for applications in accessibility and automated UI documentation.
Applications
UI Accessibility
Generate descriptions for visually impaired users.
UI Search
Search UI screenshots using natural language queries.
Automated Documentation
Generate descriptions of application screens.
UI Design Analysis
Understand layout patterns across apps.
Limitations
Complex layouts may produce generic captions
Small UI elements may be missed
Accuracy depends on training dataset diversity
Future Improvements
Component‑level UI captioning
Larger UI datasets
Multilingual caption generation
Real‑time UI analysis
Citation
@misc{rico_ui_captioner,
title={RICO UI Captioner: Vision-Language Model for UI Screen Captioning},
author={Surabhi Muralidhar},
year={2026},
url={https://huggingface.co/surabhimuralidhar/rico-ui-captioner-final}
}
Author
Surabhi Muralidhar
Acknowledgements
BLIP model research by Salesforce Research
RICO dataset developed by Stanford University
Model hosting by Hugging Face
