LZXzju/

Qwen2.5-VL-3B-UI-R1

Benchmark 1: ScreenSpotV2

Benchmark 2: ScreenSpot-Pro

Leaderboard: UI-I2E-Bench

Evaluation Code for GUI Grounding

1. Generation for UI-R1-E-3B：
   python

   1model = Qwen2_5_VLForConditionalGeneration.from_pretrained( 2 args.model_path, 3 torch_dtype=torch.bfloat16, 4 attn_implementation="flash_attention_2", 5 device_map="cpu", 6) 7model = model.to(torch.device(rank)) 8model = model.eval() 9processor = AutoProcessor.from_pretrained(ori_processor_path) 10question_template = ( 11 f"In this UI screenshot, I want to perform the command '{task_prompt}'.\n" 12 "Please provide the action to perform (enumerate in ['click', 'scroll']) and the coordinate where the cursor is moved to(integer) if click is performed.\n" 13 "Output the thinking process in <think> </think> and final answer in <answer> </answer> tags." 14 "The output answer format should be as follows:\n" 15 "<think> ... </think> <answer>[{'action': enum['click', 'scroll'], 'coordinate': [x, y]}]</answer>\n" 16 "Please strictly follow the format." 17) 18query = '<image>\n' + question_template 19messages = [ 20 { 21 "role": "user", 22 "content": [ 23 {"type": "image", "image": image_path} 24 ] + [{"type": "text", "text": query}], 25 } 26] 27text = processor.apply_chat_template( 28 messages, tokenize=False, add_generation_prompt=True 29) 30image_inputs, video_inputs = process_vision_info(messages) 31inputs = processor( 32 text=[text], 33 images=image_inputs, 34 videos=video_inputs, 35 padding=True, 36 return_tensors="pt", 37) 38generated_ids = model.generate(**inputs, max_new_tokens=1024) 39generated_ids_trimmed = [ 40 out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) 41] 42response = processor.batch_decode( 43 generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False 44) 45response = response[0] 46pred_coord, _ = extract_coord(response)
2. Rescale the predicted coordinate according to the image resize (especially image_size > 12845056)
   python

   1image = Image.open(image_path) 2origin_width, origin_height = image.size 3resized_height,resized_width = smart_resize(origin_height,origin_width,max_pixels=12845056) 4scale_x = origin_width / resized_width 5scale_y = origin_height / resized_height 6pred_coord[0] = int(pred_coord[0] * scale_x) 7pred_coord[1] = int(pred_coord[1] * scale_y)
   Function smart_resize is from Qwen2VL：
   python

   1import math 2def smart_resize( 3 height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280 4): 5 """Rescales the image so that the following conditions are met: 6 7 1. Both dimensions (height and width) are divisible by 'factor'. 8 9 2. The total number of pixels is within the range ['min_pixels', 'max_pixels']. 10 11 3. The aspect ratio of the image is maintained as closely as possible. 12 13 """ 14 if height < factor or width < factor: 15 raise ValueError(f"height:{height} or width:{width} must be larger than factor:{factor}") 16 elif max(height, width) / min(height, width) > 200: 17 raise ValueError( 18 f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}" 19 ) 20 h_bar = round(height / factor) * factor 21 w_bar = round(width / factor) * factor 22 if h_bar * w_bar > max_pixels: 23 beta = math.sqrt((height * width) / max_pixels) 24 h_bar = math.floor(height / beta / factor) * factor 25 w_bar = math.floor(width / beta / factor) * factor 26 elif h_bar * w_bar < min_pixels: 27 beta = math.sqrt(min_pixels / (height * width)) 28 h_bar = math.ceil(height * beta / factor) * factor 29 w_bar = math.ceil(width * beta / factor) * factor 30 return h_bar, w_bar