You can deploy this model and start earning money today!

Views

No views yet

Vocaela-500M: A Tiny Mighty GUI Agent Model

Name: Vocaela-500M
Availability: InStock

** Vocaela-2 comes, 3x faster: Vocaela-2-500M-1024R2 **

TL;DR: A compact 500M-parameter Vision-Language Model (VLM) designed for GUI agents. Given a screenshot and a simple instruction (e.g., “click the submit button”), it outputs structured JSON actions with precise pixel coordinates. Despite its small size, it delivers surprisingly strong performance—often surpassing much larger models—while running smoothly on laptops and even mobile devices.

Model description

A growing number of models can now operate computer and mobile GUIs on behalf of users. However, most are massive and impractical for everyday devices like laptops or phones. While many GUI agent models chase higher autonomy, Vocaela-500M explores a different path: a smaller, efficient model focused on precise low-level control.

Given a screenshot and an explicit instruction such as “click the submit button,” it produces structured JSON actions with pixel coordinates. By narrowing the scope, we maximize efficiency, achieving smooth performance on laptops and even mobile devices.

Despite its compact 500M parameters, Vocaela-500M performs surprisingly well on grounding and GUI control tasks—often matching or surpassing larger models. This marks a new step in scaling GUI agent models downward toward lightweight, practical deployment.

Type: Vision-Language Model (VLM) for Computer GUI Agents
Size: 500M parameters
Input: Screenshot + natural language instruction (specific GUI action)
Output: Structured JSON describing GUI action(s), including pixel coordinates
Recommended image resolution: longer edge < 2048
Fine-tuned from: HuggingFaceTB/SmolVLM2-500M-Video-Instruct (brief as SmolVLM2-500M below)
Quantized model: Vocaela-500M-GGUF
License: CC BY-NC-SA 4.0
Developed by: Vocaela AI

Action space

The following table lists the default action schema used during training. Users may extend or redefine it via system prompts.

	Action	Parameters	Parameters' Values	Example	Meaning
Common	type	text	string, the text to type in	{"action": "type", "text": "example"}	Typing in specified text
	click	coordinate	[x,y], scaled [0, 1), position to click on	{"action": "click", "coordinate": [0.1,0.5]}	Click using mouse or tap using finger at specified position
Desktop Only	mouse_move	coordinate	[x,y], scaled [0, 1), position to move to	{"action": "mouse_move", "coordinate": [0.1,0.5]}	Move mouse to specified position
	drag	coordinate, coordinate2	[x,y], scaled [0, 1), start (`coordinate`) and end (`coordinate2`) position to drag	{"action": "drag", "coordinate": [0.1,0.5], "coordinate2": [0.2,0.6]}	Drag mouse (click left button and hold) from specified start position to end position
	right_click	coordinate	[x,y], scaled [0, 1), position to click on	{"action": "right_click", "coordinate": [0.1,0.5]}	Click right mouse button at specified position
	middle_click	coordinate	[x,y], scaled [0, 1), position to click on	{"action": "middle_click", "coordinate": [0.1,0.5]}	Click middle mouse button at specified position
	double_click	coordinate	[x,y], scaled [0, 1), position to click on	{"action": "double_click", "coordinate": [0.1,0.5]}	Double click left mouse button at specified position
	scroll	scroll_direction	enum: {'up', 'down'}	{"action": "scroll", "scroll_direction": "up"}	Scroll mouse wheel with specified direction
	press_key	key, presses	`key`: string, single key pressed; `presses`, integer, number of times to press	{"action": "press_key", "key": 'enter'}	Press a single key
	hotkey	hotkeys	list of string, combination of keys to press, e.g, ['ctrl', 'c']	{"action": "hotkey", "hotkeys": ["ctrl", "c"]}	Press hotkey combinations e.g., Ctrl+C
Mobile Only	long_press	coordinate, time	`coordinate`: [x,y], scaled [0, 1), position to press on; `time`: seconds to hold	{"action": "long_press", "coordinate": [0.1,0.5], "time": 5}	Press at specified position and hold for specified time (s)
	swipe	swipe_direction, swipe_from, coordinate	`swipe_direction`: direction swipe towards, enum {'up', 'down', 'left', 'right'}, `swipe_from`: general area to swipe from, enum {'top', 'bottom', 'left', 'right', 'center', 'top_left', 'top_right', 'bottom_left', 'bottom_right'}, `coordinate`: [x,y], scaled [0, 1), accurate position to swipe from. `swipe_from` and `coordinate` are optional.	{"action": "swipe", "coordinate": [0.1,0.5], "swipe_direction": 'up'}	Swipe from specified start position towards specified direction
	system_button	button	string, system button to press, enum: {'Back', 'Home', 'Menu', 'Enter'}	{"action": "system_button", "button": "home"}	Press a specified system button
	open	text	string, name of app to open	{"action": "open", "text": "Google Chrome"}	Open a specified app

See below Section System messages for example of how to instruct the action space.

How to use

The model is used in the same way as SmolVLM2-500M. The example below shows how to load the model and processor, construct multimodal messages, and perform inference. For system messages, please refer to Section System messages. For a completed running example, please refer to the simple play demo vocaela-500m-demo, which is a completed example of loading model, creating messages, preprocessing, and doing model inference. It also provides an example on how to inference via LlamaCpp.

python

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_path = "vocaela/Vocaela-500M"
processor = AutoProcessor.from_pretrained(model_path)
torch_dtype = torch.float16 # using torch.bfloat16 if your device supports it
device = 'cuda' # using 'cpu' if want to inference on cpu
_attn_implementation = 'sdpa' # using "flash_attention_2" if it available in your env
model = AutoModelForImageTextToText.from_pretrained(model_path, torch_dtype=torch_dtype, _attn_implementation=_attn_implementation).to(device)

# Ensure the 'content' field uses a list format for every message, even for single items; otherwise, apply_chat_template's result will be wrong without raising any exception.
messages = [
    {
      "role": "system",
      "content": [
        { "type": "text", "text": "<SYSTEM_MESSAGE>"}, # please reference section [System messages](#system-messages) for choices of using message for computer use or mobile use.
      ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "<image full path>"},
            {"type": "text", "text": "Click the ..."},
        ]
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device, dtype=torch_dtype)

generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)
generated_texts = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True, # set it False if you want to inspect the special tokens in the generated text
)
print(generated_texts[0])

Evaluation results

We evaluated the model on two levels of tasks:

Grounding: the model is asked to directly output screen coordinate (x,y) of a concerned GUI element. In the past, related works show a trend of improving this low-level capability by scaling up model size. However, this model impresses us by how a tiny model can still perform remarkablly on grounding.
Low-level GUI agent task: the model is asked to execute low-level GUI instructions such as "click the submit button", "type 'diet food' in the search box", "scroll up the page", "open chrome", etc. Although the task is not like those popular highly autonomous agentic ones, we hope it is still a self-contained "agent" model instead of only a "grounding" model.

Grounding evaluation

Screenspot

The following table compares Vocaela-500M with other small (<=4B) specialized GUI models on the ScreenSpot benchmark. Numbers are from original papers/pages unless otherwise noted.

Model	Mobile-Text	Mobile-Icon	Desktop-Text	Desktop-Icon	Web-Text	Web-Icon	Overall
General purpose models
QWen2.5-VL-3B [1]	-	-	-	-	-	-	55.5 (from [10])
InternVL3.5-4B [2]	-	-	-	-	-	-	83.6
QWen3-VL-4B Instruct [3]	-	-	-	-	-	-	94.0
Specialized GUI models
OS-Atlas-4B [4]	85.7	58.5	72.2	45.7	82.6	63.1	70.1
ShowUI-2B [5]	92.3	75.5	76.3	61.1	81.7	63.6	75.1
UGround-V1-2B [6]	89.4	72.0	88.7	65.7	81.3	68.9	77.7
UI-Tars-2B [7]	93.0	75.5	90.7	68.6	84.3	74.8	82.3
Vocaela-500M	92.7	70.3	90.7	75.0	87.4	78.2	83.1
TongUI-3B [8]	92.6	77.7	92.3	77.1	87.8	74.8	83.6
GUI-Actor-2B [9]	93.0	79.9	88.1	78.6	90.9	84.0	86.5
InfiGUI-R1-3B [10]	97.1	81.2	94.3	77.1	91.7	77.6	87.5
UI-R1-E-3B	97.1	83.0	95.4	77.9	91.7	85.0	89.2

ScreenspotV2

The following table compares Vocaela-500M with other small (<=4B) specialized GUI models on the ScreenSpotV2 benchmark. Numbers are from the ScreenSpotV2/ScreenSpotPro leaderboard page except Phi-Ground-4B and GUI-Actor-2B from the original papers.

Model	Mobile-Text	Mobile-Icon	Desktop-Text	Desktop-Icon	Web-Text	Web-Icon	Overall
General purpose models
QWen2.5-VL-3B [1]	62.1	46.4	54.1	30.0	31.2	48.3	46.9
Specialized GUI models
OS-Atlas-4B [4]	82.8	64.0	64.4	46.4	78.6	60.1	68.5
ShowUI-2B [5]	92.1	75.4	78.9	59.3	84.2	61.1	77.3
Phi-Ground-4B [11]	94.1	62.0	91.7	77.1	94.4	78.3	84.1
TongUI-3B [8]	94.4	79.6	92.8	75.0	87.6	77.8	85.5
Vocaela-500M	95.9	73.93	95.4	75.7	91.0	75.4	85.8
ZonUI-3B [14]	98.6	82.9	92.3	74.3	88.0	74.4	86.6
GUI-Actor-2B [9]	95.0	82.2	92.2	81.8	92.9	82.7	88.6
UI-R1-E-3B	98.2	83.9	94.8	75.0	93.2	83.7	89.5
Holo1.5-3B [12]	99.2	88.0	95.0	89.7	91.8	84.8	91.7

Showdown

The following table compares Vocaela-500M with other small specialized GUI models on the Showdown benchmark. Numbers are from the Phi-Ground-4B [11] paper except Holo1.5-3B/Holo1.5-7B from their model pages. Compared with ScreenSpot and ScreenSpotV2, the Showdown dataset includes many examples requiring app-specific knowledge rather than pure visual grounding—posing a greater challenge for compact models like Vocaela-500M. Despite this, it still outperforms several 4B–7B models.

Model	Acc
OS-Atlas-4B [4]	15.8
SeeClick-9.6B [15]	24.6
OS-Atlas-7B [4]	41.1
UGround-7B [6]	46.5
Vocaela-500M	52.1
UGround-v1-7B [6]	57.8
Phi-Ground-4B [11]	58.2
UI-TARS-2B [7]	59.8
Phi-Ground-7B [11]	62.5
UI-TARS-7B [7]	66.1
UI-TARS-1.5-7B [7]	67.2
Holo1.5-3B [12]	67.5
Holo1.5-7B [13]	72.2

Low-level agent evaluation

Following convention of related works, we report the three metrics:

Type: Accuracy of predict the action type, e.g., 'click', 'type' etc.
Grounding: The accuracy of the coordinate of some actions requiring output screen coordinate, such as 'click'.
SR: The step success rate.

AndroidControl-Low

Model	Type	Grounding	SR
OS-Atlas-4B [4]	64.58	71.19	40.62
OS-Atlas-7B [4]	73.00	73.37	50.94
GUI-R1-3B [16]	83.68	81.59	64.41
GUI-R1-7B [16]	85.17	84.02	66.52
Aria-UI (3.5B act. of 24.9B) [17]	-	87.7	67.3
Vocaela-500M	83.98	81.52	69.68
Aguvis-7B [18]	–	–	80.5
UI-R1-3B [19]	94.3	82.6	88.5
UI-TARS-2B [7]	98.1	87.3	89.3 (from [10])
InfiGUI-R1-3B [10]	96.0	93.2	92.1

Numbers other than Vocaela-500M are from each own reference except UI-TARS-2B which is from InfiGUI-R1-3B [10].

GUI-Act-Web

Model	Type	Grounding	SR
OS-Atlas-4B [4]	79.22	58.57	42.62
OS-Atlas-7B [4]	86.95	75.61	57.02
UI-R1-3B [19]	75.89	79.43	67.31 (from [10])
GUI-R1-3B [16]	89.86	87.42	76.31
GUI-R1-7B [16]	90.85	88.06	80.31
Vocaela-500M	90.28	79.71	80.43

OmniAct-Web

Model	Type	Grounding	SR
OS-Atlas-4B [4]	46.74	49.24	22.99
OS-Atlas-7B [4]	85.63	69.35	59.15
UI-R1-3B [19]	75.42	61.35	61.33 (from [16])
Vocaela-500M	88.16	72.42	67.13
GUI-R1-3B [16]	88.58	75.10	75.08
GUI-R1-7B [16]	91.16	77.29	77.35

OmniAct-Desktop

Model	Type	Grounding	SR
OS-Atlas-4B [4]	63.30	42.55	26.94
OS-Atlas-7B [4]	90.24	62.87	56.73
UI-R1-3B [19]	73.41	64.12	63.98 (from [16])
GUI-R1-3B [16]	91.86	78.37	78.31
Vocaela-500M	89.23	83.05	79.12
GUI-R1-7B [16]	92.20	83.36	83.33

Training strategy

The model architecture and configuration remain identical to the base model, except for its slightly customized chat template (see Section Special tokens & chat template). Training followed three stages: Two stages of Supervised Fine-Tuning (SFT) and then Reinforcement Fine-Tuning (RFT) using GRPO.

SFT Stage 1: ~7M examples from public datasets after extensive preprocessing, unifying action spaces, synthesis, and balancing.

SFT Stage 2: ~256K examples after filtering, re-sampling to balance action distribution, synthesis to enrich rare actions

RFT: ~40K examples

Limitations

Not suitable for high-resolution images

There are two factors in the base model SmolVLM2-500M which limit the capability of handling high resolution images:
1. Aggressive pixel shuffle (r=4), compressing 64x64 pixels into one token.
2. Fixed scaling to 2048px on the longest side.
Together, they severely impact grounding on high-resolution screens. We evaluated Vocaela-500M on ScreenSpotPro. The average score is 15.1. Although still better than several 2B/3B/7B models, the relative superiority is much worse compared to that on ScreenSpot/ScreenSpotV2. The result verifies the limitation.
Not suitable for high-level agentic tasks / lacks reasoning capability

The base model SmolVLM2-500M does not exhibit reasoning capabilities. Considering that the main goal of this model is for low-level command execution and the very compact model size, the model was not trained with reasoning capabilities. We evaluated Vocaela-500M on high level task of AndroidControl-High. Shown as below, the result verifies this limitation.

Task Model Type Grounding SR
AndroidControl-High Vocaela-500M 25.9 13.1 13.1
Loss of general-purpose capabilities

The model is heavily-tuned on this specific scenario and hence loses general-purpose capabilities, such as chat, QA, instruct-following etc.
No video support

The model was not trained with any video data in SFT/RFT.

System messages

Below system messages were used in training and hence recommended to use for inference.

System message for computer use

python

Vocaelam_Computer_Use_System_Message = """
You are an assistant trained to navigate the computer screen. 
Given a task instruction, a screen observation, and an action history sequence, 
output the next actions and wait for the next observation. 

## Allowed ACTION_TYPEs and parameters:
1. `PRESS_KEY`: Press one specified key. Two parameters: `key`, string, the single key to press; `presses`, integer, the number of times to press the key (default is 1).
2. `TYPE`: Type a string into an element. Parameter: `text`, string, the text to type.
3. `MOUSE_MOVE`: Move the mouse cursor to a specified position. Parameter: `coordinate`, formatted as [x,y], the position to move the cursor to.
4. `CLICK`: Click left mouse button once on an element. Parameter: `coordinate`, formatted as [x,y], the position to click on.
5. `DRAG`: Drag the cursor with the left mouse button pressed, start and end positions are specified. Two parameters: `coordinate`, formatted as [x,y], the start position to drag from; `coordinate2`, formatted as [x2,y2], the end position to drag to.
6. `RIGHT_CLICK`: Click right mouse button once on an element. Parameter: `coordinate`, formatted as [x,y], the position to right click on.
7. `MIDDLE_CLICK`: Click middle mouse button once on an element. Parameter: `coordinate`, formatted as [x,y], the position to middle click on.
8. `DOUBLE_CLICK`: Click left mouse button twice on an element. Parameter: `coordinate`, formatted as [x,y], the position to double click on.
9. `SCROLL`: Scroll the screen (via mouse wheel). Parameter: `scroll_direction`, the direction (`up`/`down`/`left`/`right`) to scroll.
10. `WAIT`: Wait for several seconds. Parameter: `time`, duration in seconds to wait.
11. `TERMINATE`: Terminate the task. Parameter: `status`, the status of the task, `success`/`failure`.
12. `REFUSE`: Refuse to perform the task if not feasible. No parameter.
13. `HOTKEY`: Press a combination of keys simultaneously. Parameter: `hotkeys`, list of strings, the keys to press together.

* NOTE *: The `coordinate` and `coordinate2` parameters (formatted as [x,y]) are the relative coordinates on the screenshot scaled to range of 0-1, [0,0] is the top-left corner and [1,1] is the bottom-right corner.

## Format your response as
<Action>the next actions</Action>

`The next actions` can be one or multiple actions. Format `the next actions` as a JSON array of objects as below, each object is an action:
[{"action": "<ACTION_TYPE>", "key": "<key>", "presses": <presses>, "hotkeys": ["<hotkeys>"], "text": "<text>", "coordinate": [x,y], "coordinate2": [x2,y2], "time": <time>, "status": "<status>", "scroll_direction": "<scroll_direction>"}]

If a parameter is not applicable, don't include it in the JSON object.
"""

System message for mobile phone use

python

Vocaela_Mobile_Use_System_Message = """
You are an assistant trained to navigate the mobile phone. 
Given a task instruction, a screen observation, and an action history sequence, 
output the next actions and wait for the next observation. 

## Allowed ACTION_TYPEs and parameters:
1. `CLICK`: Click/tap on the screen. Parameter: `coordinate`, formatted as [x,y], the position to click on.
2. `LONG_PRESS`: Long press on the screen. Two parameters: `coordinate`, formatted as [x,y], the position to long press on; `time`, duration in seconds to long press.
3. `SWIPE`: Swipe on the screen. Two parameters: `swipe_from`, the start area to swipe from, only allowed value in {'top', 'bottom', 'left', 'right', 'center', `top_left`, `top_right`, `bottom_left`, `bottom_right`}; `swipe_direction`, the direction (`up`/`down`/`left`/`right`) to swipe towards.
4. `TYPE`: Type a string into an element. Parameter: `text`, string, the text to type.
5. `SYSTEM_BUTTON`: Press a system button. Parameter: `button`, the system button to press, allowed button values: 'Back', 'Home', 'Menu', 'Enter'.
6. `OPEN`: Open an app. Parameter: `text`, string, the app name to open.

* NOTE *: The `coordinate` parameter (formatted as [x,y]) is the relative coordinates on the screenshot scaled to range of 0-1, [0,0] is the top-left corner and [1,1] is the bottom-right corner.

## Format your response as
<Action>the next actions</Action>

`The next actions` can be one or multiple actions. Format `the next actions` as a JSON array of objects as below, each object is an action:
[{"action": "<ACTION_TYPE>", "text": "<text>", "coordinate": [x,y], "swipe_from": "<swipe_from>", "swipe_direction": "<swipe_direction>", "button": "<button>"}]

If a parameter is not applicable, don't include it in the JSON object.
"""

Special tokens & chat template

The base model SmolVLM2-500M does not provided special token to identify user or assistant role. For convenience of accurately masking user-turn messages in SFT, two existing special tokens were used to mark the beginning and end of an assistant message, <|reserved_special_token_50|> for the beginning and <|reserved_special_token_51|> for the end. Consequently, if looking into the chat_template.jinja file of the model folder, you will find the chat template added the prefix token <|reserved_special_token_50|> for inference:

<|im_start|>{% for message in messages %}{{message['role'] | capitalize}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>
{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:<|reserved_special_token_50|>' }}{% endif %}

For a normal generation, if you configured tokenizer decoding not skipping special tokens, a completed sequence ends with two successive special tokens <|reserved_special_token_51|><end_of_utterance>, where <end_of_utterance> is the default end token of the base model and <|reserved_special_token_51|> is introduced in by our training process.

License

This model is made available under the CC BY-NC-SA 4.0 license. To comply with the license, you may use, modify, and share the dataset or derivative works for non-commercial purposes only. Any derivative works must be shared under the same license.

Reason of adopting CC BY-NC-SA 4.0 license: model training used datasets under CC BY-NC-SA 4.0 license.

Please see the full license here.

Acknowledgements

Thanks to Microsoft Azure startup credit offer for partially funding the computing
Thanks to related projects Jedi, TongUI, UGround, Aguvis, OS-ATLAS, GTA-1, OpenCUA, GUI-R1, etc. We leveraged datasets, code, and insights shared out from them.

References

[1] Bai, Shuai, et al. "Qwen2. 5-vl technical report." arXiv preprint arXiv:2502.13923 (2025).

[2] Wang, Weiyun, et al. "Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency." arXiv preprint arXiv:2508.18265 (2025).

[3] Qwen3-VL. https://github.com/QwenLM/Qwen3-VL

[4] Wu, Zhiyong, et al. "Os-atlas: A foundation action model for generalist gui agents." arXiv preprint arXiv:2410.23218 (2024).

[5] Lin, Kevin Qinghong, et al. "Showui: One vision-language-action model for gui visual agent." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[6] Gou, Boyu, et al. "Navigating the digital world as humans do: Universal visual grounding for gui agents." arXiv preprint arXiv:2410.05243 (2024).

[7] Qin, Yujia, et al. "Ui-tars: Pioneering automated gui interaction with native agents." arXiv preprint arXiv:2501.12326 (2025).

[8] Zhang, Bofei, et al. "TongUI: Building Generalized GUI Agents by Learning from Multimodal Web Tutorials." arXiv preprint arXiv:2504.12679 (2025).

[9] Wu, Qianhui, et al. "GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents." arXiv preprint arXiv:2506.03143 (2025).

[10] Liu, Yuhang, et al. "Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners." arXiv preprint arXiv:2504.14239 (2025).

[11] Zhang, Miaosen, et al. "Phi-ground tech report: Advancing perception in gui grounding." arXiv preprint arXiv:2507.23779 (2025).

[12] Holo1.5-3B. https://huggingface.co/Hcompany/Holo1.5-3B

[13] Holo1.5-7B. https://huggingface.co/Hcompany/Holo1.5-7B

[14] Hsieh, ZongHan, and Tzer-Jen Wei. "ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding" arXiv preprint arXiv:2506.23491 (2025).

[15] Cheng, Kanzhi, et al. "Seeclick: Harnessing gui grounding for advanced visual gui agents." arXiv preprint arXiv:2401.10935 (2024).

[16] Luo, Run, et al. "Gui-r1: A generalist r1-style vision-language action model for gui agents." arXiv preprint arXiv:2504.10458 (2025).

[17] Yang, Yuhao, et al. "Aria-ui: Visual grounding for gui instructions." arXiv preprint arXiv:2412.16256 (2024).

[18] Xu, Yiheng, et al. "Aguvis: Unified pure vision agents for autonomous gui interaction." arXiv preprint arXiv:2412.04454 (2024).

[19] Lu, Zhengxi, et al. "UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning." arXiv preprint arXiv:2503.21620 (2025).