📊 Performance Comparison on GUI Grounding Benchmarks
Table 1. Main results on ScreenSpot-Pro, ScreenSpot, and ScreenSpot-v2 with Qwen2-VL as the backbone. † indicates scores obtained from our own evaluation of the official models on Huggingface.
Method
Backbone VLM
ScreenSpot-Pro
ScreenSpot
ScreenSpot-v2
72B models:
AGUVIS-72B
Qwen2-VL
-
89.2
-
UGround-V1-72B
Qwen2-VL
34.5
89.4
-
UI-TARS-72B
Qwen2-VL
38.1
88.4
90.3
7B models:
OS-Atlas-7B
Qwen2-VL
18.9
82.5
84.1
AGUVIS-7B
Qwen2-VL
22.9
84.4
86.0†
UGround-V1-7B
Qwen2-VL
31.1
86.3
87.6†
UI-TARS-7B
Qwen2-VL
35.7
89.5
91.6
GUI-Actor-7B
Qwen2-VL
40.7
88.3
89.5
GUI-Actor-7B + Verifier
Qwen2-VL
44.2
89.7
90.9
2B models:
UGround-V1-2B
Qwen2-VL
26.6
77.1
-
UI-TARS-2B
Qwen2-VL
27.7
82.3
84.7
GUI-Actor-2B
Qwen2-VL
36.7
86.5
88.6
GUI-Actor-2B + Verifier
Qwen2-VL
41.8
86.9
89.3
Table 2. Main results on the ScreenSpot-Pro and ScreenSpot-v2 with Qwen2.5-VL as the backbone.
Method
Backbone VLM
ScreenSpot-Pro
ScreenSpot-v2
7B models:
Qwen2.5-VL-7B
Qwen2.5-VL
27.6
88.8
Jedi-7B
Qwen2.5-VL
39.5
91.7
GUI-Actor-7B
Qwen2.5-VL
44.6
92.1
GUI-Actor-7B + Verifier
Qwen2.5-VL
47.7
92.5
3B models:
Qwen2.5-VL-3B
Qwen2.5-VL
25.9
80.9
Jedi-3B
Qwen2.5-VL
36.1
88.6
GUI-Actor-3B
Qwen2.5-VL
42.2
91.0
GUI-Actor-3B + Verifier
Qwen2.5-VL
45.9
92.4
🚀 Usage
python
1import torch
23from qwen_vl_utils import process_vision_info
4from datasets import load_dataset
5from transformers import AutoProcessor
6from gui_actor.constants import chat_template
7from gui_actor.modeling_qwen25vl import Qwen2_5_VLForConditionalGenerationWithPointer
8from gui_actor.inference import inference
91011# load model12model_name_or_path ="microsoft/GUI-Actor-7B-Qwen2.5-VL"13data_processor = AutoProcessor.from_pretrained(model_name_or_path)14tokenizer = data_processor.tokenizer
15model = Qwen2_5_VLForConditionalGenerationWithPointer.from_pretrained(16 model_name_or_path,17 torch_dtype=torch.bfloat16,18 device_map="cuda:0",19 attn_implementation="flash_attention_2"20).eval()2122# prepare example23dataset = load_dataset("rootsautomation/ScreenSpot")["test"]24example = dataset[0]25print(f"Intruction: {example['instruction']}")26print(f"ground-truth action region (x1, y1, x2, y2): {[round(i,2)for i in example['bbox']]}")2728conversation =[29{30"role":"system",31"content":[32{33"type":"text",34"text":"You are a GUI agent. Given a screenshot of the current GUI and a human instruction, your task is to locate the screen element that corresponds to the instruction. You should output a PyAutoGUI action that performs a click on the correct position. To indicate the click location, we will use some special tokens, which is used to refer to a visual patch later. For example, you can output: pyautogui.click(<your_special_token_here>).",35}36]37},38{39"role":"user",40"content":[41{42"type":"image",43"image": example["image"],# PIL.Image.Image or str to path44# "image_url": "https://xxxxx.png" or "https://xxxxx.jpg" or "file://xxxxx.png" or "data:image/png;base64,xxxxxxxx", will be split by "base64,"45},46{47"type":"text",48"text": example["instruction"]49},50],51},52]5354# inference55pred = inference(conversation, model, tokenizer, data_processor, use_placeholder=True, topk=3)56px, py = pred["topk_points"][0]57print(f"Predicted click point: [{round(px,4)}, {round(py,4)}]")5859# >> Model Response60# Intruction: close this window61# ground-truth action region (x1, y1, x2, y2): [0.9479, 0.1444, 0.9938, 0.2074]62# Predicted click point: [0.9709, 0.1548]
📝 Citation
@article{wu2025gui,
title={GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents},
author={Wu, Qianhui and Cheng, Kanzhi and Yang, Rui and Zhang, Chaoyun and Yang, Jianwei and Jiang, Huiqiang and Mu, Jian and Peng, Baolin and Qiao, Bo and Tan, Reuben and others},
journal={arXiv preprint arXiv:2506.03143},
year={2025}
}
