V-Droid
V-Droid is a novel mobile GUI task automation agent that leverages Large Language Models (LLMs) as verifiers rather than generators. This verifier-driven paradigm, combined with a discretized action space and a prefilling-only workflow, allows V-Droid to achieve state-of-the-art performance on public benchmarks while maintaining near-real-time decision-making capabilities.
Model Details
Model Description
V-Droid is a mobile GUI agent designed to automate tasks on Android devices. Unlike conventional approaches where LLMs generate actions directly, V-Droid first identifies a set of candidate actions from the current screen and then uses a verifier model to evaluate and select the most promising action. This approach significantly improves accuracy and reduces latency.
The core components of the V-Droid framework include:
- Discretized Action Space: V-Droid operates on a defined set of actions (e.g., click, type), enabling efficient identification of potential actions on a given screen.
- Verifier-based Decision Making: An LLM-based verifier assesses the utility of each candidate action in the context of the current task, leading to more reliable decisions.
- Pair-wise Progress Preference Training (P3T): A novel training strategy that enhances the verifier's ability to distinguish between beneficial and detrimental actions.
- Human-Agent Joint Annotation: A scalable and efficient method for collecting high-quality training data.
This model card corresponds to the verifier component of the V-Droid agent.
- Developed by: Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, Lili Qiu
- Model type: Verifier for Mobile GUI Agent
- Language(s) (NLP): English
- Finetuned from model: Llama-3.1-8B-4bit
Model Sources
Uses
Direct Use
The V-Droid verifier model is intended to be used as a core component within the V-Droid agent framework. It takes a HTML description of a mobile device, a task description, and a candidate action as input, and outputs a score that the action will contribute to completing the task.
Downstream Use
The principles and architecture of V-Droid can be adapted for other GUI automation tasks beyond mobile environments, such as web or desktop applications. The verifier-driven approach could also be explored for other types of autonomous agents where action validation is critical.
Out-of-Scope Use
This model is not designed for generating free-form text or engaging in conversational chat. It is specifically tailored for the task of action verification in a mobile GUI context. Using it for purposes outside of this scope is likely to yield poor results. The model should not be used for any applications that could cause harm, violate privacy, or conduct malicious activities.
Bias, Risks, and Limitations
The performance of V-Droid is dependent on the quality and diversity of the training data. As with any model trained on specific data distributions, it may exhibit biases present in the training set. The model's ability to generalize to unseen applications or radically different UI layouts may be limited.
Recommendations
Users should be aware of the potential for errors and biases. For critical tasks, human oversight is recommended. To improve performance on specific applications, fine-tuning the verifier on a targeted dataset of interactions with those applications would be beneficial.
Training Details
Training Data
V-Droid was trained on data collected through a human-agent joint annotation process. This involved human annotators interacting with V-Droid, correcting its mistakes, and providing demonstrations of successful task completion. The paper does not point to a specific public dataset release at this time.
Training Procedure
The training data consists of state-action pairs, where the state includes the screen information and the task description. Actions are categorized as either positive (contributing to task completion) or negative.
Citation
BibTeX:
1 @article{dai2025advancing,
2 title={Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment},
3 author={Dai, Gaole and Jiang, Shiqi and Cao, Ting and Li, Yuanchun and Yang, Yuqing and Tan, Rui and Li, Mo and Qiu, Lili},
4 journal={arXiv preprint arXiv:2503.15937},
5 year={2025}
6 }
