GUI-Shift Implementation

Overview

Key Components

• build_data.py: Constructs K-step GUI Transition dataset from AndroidControl
• train_gui_shift.py: GRPO training with rule-based rewards (format + action correctness)
• run_train.sh: Launch script with hyperparameters matching the paper

Workflow

1. Build dataset: python build_data.py
   • Downloads AndroidControl (ckg/AndroidControlParsedWithImages-20k)
   • Extracts K-step state pairs (default k=1)
   • Saves 2000 transition samples as JSONL
2. Train: bash run_train.sh
   • Loads Qwen2.5-VL-7B-Instruct
   • Freezes vision encoder + projector
   • Trains with GRPO (format_reward + action_reward)
   • Pushes to Hugging Face Hub

Hyperparameters (from paper Appendix A)

Action Space

• click, long_press (with x,y coordinates)
• scroll (with direction)
• open_app (with app_name)
• input_text (with text content)
• navigate_back, navigate_home, wait

Reward Design

• Format reward: +1 if output contains <answer>...</answer> tags
• Action reward: +1 if parsed action matches ground truth
  • Click/long_press: point falls within ground-truth bounding box
  • Scroll/open_app/input_text: exact parameter match
  • navigate_back/navigate_home/wait: action type match only

References

• Paper: https://arxiv.org/abs/2505.12493
• Dataset: ckg/AndroidControlParsedWithImages-20k
• Base Model: Qwen/Qwen2.5-VL-7B-Instruct