This model is a fine-tuned version of Qwen's Qwen3-VL-30B-A3B-Instruct base model with LoRA adaptation for Android UI control tasks. This model demonstrates strong performance in GUI Grounding tasks, particularly excelling in coordinate prediction accuracy for click actions.

Key Performance Highlights

Strong GUI Grounding Performance:

Click L2 Distance: 87.04 pixels - showing competitive performance compared to other models in the benchmark
Demonstrates strong coordinate prediction capabilities for GUI interaction tasks
Solid performance across other action types (input text match: 0.8455, scroll direction match: 0.8689)

The model demonstrates strong spatial understanding for GUI elements, making it suitable for automated UI testing and accessibility applications.

Training Data

Dataset: OfficerChul/Android-Control-84k
Data Format: Mobile UI screenshots paired with user instructions to perform appropriate actions (click, scroll, input, etc.)
Training Samples: 84,000+ UI interaction examples

Training Data Format Example

json

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant that can identify what action to perform on mobile UI Screenshot given the user instruction."
    },
    {
      "role": "user",
      "content": "<image>Click on the Recording 2"
    },
    {
      "role": "assistant",
      "content": "{\"action_type\": \"click\", \"x\": 561, \"y\": 535}"
    }
  ],
  "images": ["and_ctrl/out_episode_18557_step_001.png"]
}

Training Method

LoRA fine-tuning performed using LLaMA-Factory framework

Training Configuration (`qwen_3_vl_30b.yaml`)

Base Model: Qwen/Qwen3-VL-30B-A3B-Instruct
Training Method: LoRA (Low-Rank Adaptation)
LoRA Configuration:
- Rank: 8
- Target modules: all
- Image max pixels: 128,000
Training Parameters:
- Batch size: 4 (gradient accumulation: 48, effective batch size: 192)
- Learning rate: 1e-4
- Epochs: 5
- LR scheduler: Cosine
- Warmup ratio: 0.1
- Optimizer: AdamW (fused)
- Precision: bf16
- Weight decay: 0.01
- Cutoff length: 2048 tokens
Additional Settings:
- Gradient checkpointing enabled
- Flash Attention 2 enabled
- Vision tower, multi-modal projector, and language model all trainable
- DeepSpeed ZeRO-3 utilized
- Validation size: 5%
- Evaluation steps: 100

Training Results

Total Steps: 2,055
Final Training Loss: 0.2086
Final Evaluation Loss: 0.1190
Training Runtime: ~104 hours
Samples per Second: 1.049

Supported Action Types

click: Click on specific coordinates (x, y)
long_press: Long press action
scroll: Scroll (up/down/left/right)
input_text: Text input
navigate_back: Navigate back
navigate_home: Navigate to home screen
open_app: Open application
wait: Wait action

Usage

The merged model can be directly loaded using the Hugging Face Transformers library.

python

from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image

model_path = "OfficerChul/Qwen3-VL-30B-Android-Control"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto"
)

# Prepare your UI screenshot
image = Image.open("path/to/screenshot.png")
instruction = "Click on the Settings button"

# Prepare conversation
messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant that can identify what action to perform on mobile UI Screenshot given the user instruction."
    },
    {
        "role": "user",
        "content": f"<image>{instruction}"
    }
]

# Process and generate
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
result = processor.batch_decode(outputs, skip_special_tokens=True)[0]
print(result)

Evaluation Results

Comprehensive Benchmark Comparison

Model	Action Type Accuracy	Click L2 Distance	Input Text Match	Scroll Direction Match
Qwen/Qwen2.5-VL-3B-Instruct	0.6645	88.21 (n=165)	0.7889 (n=90)	0.3519 (n=108)
OfficerChul/Qwen2.5-VL-3B-Instruct	0.9965	446.54 (n=1467)	0.9363 (n=157)	0.9738 (n=267)
InfiX-ai/InfiGUI-G1-3B	0.8745	102.39 (n=1020)	0.7700 (n=100)	0.2299 (n=174)
OfficerChul/InfiGUI-G1-3B	0.9980	449.73 (n=1467)	0.9625 (n=160)	0.9625 (n=267)
Qwen/Qwen3-VL-30B-A3B-Instruct	0.9090	705.72 (n=812)	0.8264 (n=121)	0.3226 (n=248)
OfficerChul/Qwen3-VL-30B-A3B-Instruct_lora_sft	0.5907	87.04	0.8455	0.8689
Qwen/Qwen2.5-VL-72B-Instruct	0.6594	64.98 (n=125)	0.8879 (n=107)	0.2925 (n=106)
OfficerChul/Qwen2.5-VL-72B-Instruct	0.8838	529.23	0.9032	0.9512
google/gemma-3n-E4B-it	0.5398	824.09	0.7521	0.5217
OfficerChul/gemma-3n-E4B-it	0.5088	878.66 (n=124)	0.8763 (n=97)	0.3689 (n=103)

License

This model follows the Apache 2.0 license of the Qwen3-VL base model.

Acknowledgments

Base model: Qwen3-VL-30B-A3B-Instruct by Qwen team
Training framework: LLaMA-Factory
Dataset: Android-Control-84k

Notes

This model was developed for research purposes in mobile UI automation and accessibility enhancement
The strong GUI grounding performance makes it suitable for applications requiring precise coordinate prediction
Proper validation is required when using in production environments
For best results, ensure input images are clear and at appropriate resolution

Generated with LLaMA-Factory | For questions or issues, please open an issue on the model repository.

OfficerChul/Qwen3-VL-30B-A3B-Instruct-Android-Control-84k

Qwen3-VL-30B-A3B-Instruct Android Control LoRA Fine-tuned Model

Model Overview