Views
No views yet
pip install transformers
pip install qwen-vl-utilsfrom transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
# Default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
"HongxinLi/UIPro_1stage", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("HongxinLi/UIPro_1stage")
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "./web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
},
# For ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench Action Grounding
{"type": "text", "text": "I want to {goal_info}. Please locate the target element I should interact with. (Output the center coordinates of the target)"},
# For AutoGUI
{"type": "text", "text": "Locate the element according to its detailed functionality description. {goal_info} (Output the center coordinates of the target)"},
# For VisualWebBench Element Grounding
{"type": "text", "text": "Locate the text "{goal_info}" (Output the center coordinates of the target)"},
],
}
]| Model | Size | Input Res. | FuncGnd | ScreenSpot | ScreenSpot-v2 | MOTIF | RefExp | VWB EG | VWB AG |
|---|---|---|---|---|---|---|---|---|---|
| GPT-4o | - | AnyRes | 9.8 | 17.8 | 20.4 | 30.5 | 21.8 | 5.6 | 6.8 |
| Qwen2VL [1] | 72B | AnyRes | 47.7 | 71.4 | 73.2 | 80.3 | 77.7 | 60.5 | 62.1 |
| Qwen2VL [1] | 7B | AnyRes | 38.7 | 66.4 | 66.9 | 75.1 | 64.8 | 55.9 | 62.1 |
| CogAgent [2] | 18B | 1120 | 29.3 | 47.4 | 49.2 | 46.7 | 35.0 | 55.7 | 59.2 |
| SeeClick [3] | 10B | 448 | 19.8 | 53.4 | 54.0 | 11.1 | 58.1 | 39.2 | 27.2 |
| Ferret-UI [4] | 8B | AnyRes | 1.2 | 7.1 | 7.8 | 15.9 | 5.5 | 3.9 | 1.9 |
| UGround [5] | 7B | AnyRes | 48.8 | 74.8 | 76.5 | 72.4 | 73.6 | 85.2 | 63.1 |
| OS-ATLAS-Base [6] | 7B | AnyRes | 52.1 | 82.5 | 84.1 | 78.8 | 66.5 | 82.6 | 69.9 |
| UIPro-Qwen2VL (ours) | 7B | AnyRes | 58.8 | 82.5 | 86.9 | 80.6 | 81.9 | 94.9 | 70.9 |
| Qwen2-VL [4] | 2B | AnyRes | 7.1 | 17.9 | 18.6 | 28.8 | 29.2 | 17.9 | 17.5 |
| UIPro-SLiME (ours) | 3B | AnyRes | 58.3 | 60.7 | 61.1 | 73.3 | 59.0 | 60.0 | 40.8 |
@InProceedings{Li_2025_ICCV,
author = {Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},
title = {UIPro: Unleashing Superior Interaction Capability For GUI Agents},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2025},
pages = {1613-1623}
}