A 27B vision-language GUI agent, produced by full-parameter supervised fine-tuning of Qwen/Qwen3.8-27B on FigmaTrace which contains 3,469 trajectories distilled from 200+ hours of screen-recorded expert Figma work. Training on in-domain design workflows transfers out of domain: the model beats its base checkpoint on six of the seven benchmark configurations and outscores both closed frontier baselines on two of them.

Model Details

Model Description

FigmaTrace converts expert design sessions into agent trajectories using design phase-based segmentation rather than fixed-length sharding. This model is the best-performing arm of that study. It predicts the next GUI action from a screenshot plus an instruction, emitting Playwright-MCP-style tool calls with norm-1000 coordinates (both axes scaled to 0–1000, not raw pixels).

Developed by: Patronus AI
Model type: Vision-language model for next-action prediction in GUI agents (qwen3_5, Qwen3_5ForConditionalGeneration)
Language(s) (NLP): English
License: Inherited from the Qwen/Qwen3.8-27B base model; see that repository for terms. Training data is released under CC-BY-4.0.
Finetuned from model: Qwen/Qwen3.8-27B

Model Sources

Repository: https://huggingface.co/PatronusAI/Qwen3.8-27B-Figmatrace-SFT
Paper: FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows
Dataset: https://huggingface.co/datasets/PatronusAI/figmatrace

Uses

Direct Use

Next-action prediction for GUI and design agents: given a screenshot and a task instruction, the model returns the tool call to execute. It was trained on Figma workflows but evaluated out of domain on mobile, web, and desktop navigation, and generalizes to all three.

Downstream Use

A starting checkpoint for further SFT or RL on computer-use tasks, or a base for design-specific agents (component construction, layout blocking, theming, accessibility remediation).

Bias, Risks, and Limitations

Failure modes below come from a manual inspection of every item where fine-tuning flipped the base model's outcome (paper §5, RQ3):

Regression on undirected VideoGUI: 26.0 → 19.3. Fine-tuning traded open-ended exploration for directed execution.
Repetition: on Android flows the model can emit near-identical coordinates on consecutive steps — e.g. (331, 989) then (331, 988) — while the ground-truth trajectory advances down a list. Attributed to noisy actions surviving preprocessing. It inflates trajectory length rather than failing outright, but strains agent context budgets.
Screen-center bias: targets in browser chrome are sometimes abandoned in favor of content in the middle of the frame which could be a plausible leak of FigmaTrace's canvas-centric framing. Clusters in utility and browser flows.
Design taste is SME-specific: open-ended source tasks (theming, sketch-to-Figma, prototyping) encode the preferences of the individual experts recorded.
Small evaluation samples: benchmark scores come from 150-item probes, so differences of a few points are not robustly separated.

Recommendations

De-normalize the model's norm-1000 output against actual viewport dimensions before dispatching clicks. Add a repeat-action guard in agent loops. Prefer step-directed prompting over open-ended goals, and keep a human in the loop for irreversible actions.

How to Get Started with the Model

python

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "PatronusAI/Qwen3.8-27B-Figmatrace-SFT"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "screenshot.png"},
        {"type": "text", "text": "Open the color picker and set the fill to #00DC82."},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    tools=PLAYWRIGHT_MCP_TOOLS,   # 17-tool Playwright-MCP schema used in training (or other compatible design toolset)
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Coordinates in the returned tool call are norm-1000: multiply x by width/1000 and y by height/1000. Evaluation used greedy decoding with reasoning off and tool use forced.

Training Details

Training Data

FigmaTrace — 126 long-horizon tasks across 8 designer workflow categories, covering a 10-skill expert taxonomy. This run used 92,472 actions sampled from 35 sessions, averaging 47.6 hours of expert work, packed into 1,554 phase-aware training rows of up to 44 frames each.

Training Procedure

Preprocessing

OS-level events were filtered (~95% removed, mostly idle mouse movement) and mapped to the closest Playwright-MCP action. Frames were extracted in two passes with settle detection: a candidate at time t pairs the frame at t−0.15s with the first frame in [t+0.2, t+2.0] where consecutive frames satisfy mean |Δ| < 0.75. Action-frame pairs whose changed-pixel fraction fell below 5×10⁻⁴ were dropped as having no visible effect. observe probes were inserted every 2s inside gaps longer than 4s so input-free screen transitions become first-class steps. Phase boundaries were assigned by Gemini-3.6-Flash over a closed 12-label vocabulary, keeping only boundaries that ≥2 of three shardings (3/6/12-way) placed within ±30s. Total compaction: 179× versus raw OS events.

Training Hyperparameters

Training regime: bf16 mixed precision, full-parameter SFT with the vision tower frozen and the aligner trainable
Framework: ms-swift with the Megatron mcore-bridge
Parallelism: TP4 / CP2 / DP2, sequence parallel 2, distributed optimizer, full activation recomputation
Learning rate: 2e-6, cosine decay to 2e-7, 3% warmup
Optimizer: Adam (β = 0.9/0.95), weight decay 0.1, gradient clipping 1.0
Batch: global batch size 6, micro-batch 1
Steps: 259 iterations (≈1 epoch); published checkpoint is checkpoint-259
Sequence length: 65,536 tokens
Visual budget: 1,296,000 pixels (≈1,260 visual tokens/frame), up to 44 frames per row
Attention: FlashAttention
Seed: 42

Speeds, Sizes, Times

Checkpoint is 54.7 GB in bf16 across 12 safetensors shards. Wall-clock training time and hardware were not recorded in the released artifacts.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Four out-of-distribution GUI agent benchmarks — GUI-Odyssey (multi-app, multi-viewport navigation), AndroidControl (instruction granularity), Mind2Web (open-web instruction grounding, two viewport configurations), and VideoGUI (planning, narration, execution), plus the in-domain ScreenSpot-Pro Creative split. VideoGUI matters as a generalization check because it is built with a different video-to-trajectory pipeline than FigmaTrace's.

Results

Step-wise accuracy across GUI agent benchmarks

Summary

Fine-tuning on FigmaTrace lifts the base model on six of the seven benchmark configurations, most sharply on AndroidControl (+16.4) and GUI-Odyssey (+9.7), and by +8.3 on the in-domain ScreenSpot-Pro Creative split. At 27B it exceeds Claude Opus 5 on GUI-Odyssey (+6.4) and AndroidControl (+11.8), while remaining well behind both frontier models on Mind2Web and on undirected VideoGUI. Phase-based trajectory construction beats length-based sharding by 7.3 points on average, confirming that the gains come from the segmentation method and not merely from more design data.

Model Examination

Manual inspection of every flipped item identified three sources of gain:

Element selection — two-thirds of GUI-Odyssey gains are cases where the base model selects an entirely different UI element. Base median error ≈457 px; fine-tuned ≈15 px. Largest category improvements: Media (+22 pp) and Social (+17 pp).
Coordinate understanding — the base model emits raw pixel values that land far off target when read as norm-1000, with y > 1000 overflowing the viewport on 10/150 GUI-Odyssey items. The fine-tuned model does this on none.
Decisiveness — every AndroidControl gain is an item where the base model emitted no coordinates or picked a clearly wrong element; the fine-tuned model always answers, landing ≈11 px from target when it corrects the element choice.

Technical Specifications

Model Architecture and Objective

Qwen3_5ForConditionalGeneration, a hybrid-attention VLM: hidden size 5120, head dim 256, one full attention layer every four (the rest linear attention), 248k-token vocabulary, Qwen3VLProcessor vision stack (patch size 16, merge size 2). Objective is standard next-token cross-entropy over assistant tool-call turns.

Compute Infrastructure

Software

ms-swift, Megatron-LM (mcore-bridge), PyTorch, Transformers, FlashAttention.

Citation

BibTeX:

bibtex

@article{deshpande2026figmatrace,
  title={FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows},
  author={Deshpande, Darshan and Fujinuma, Yoshinari and Markiewicz, Martyna
          and Bansal, Devanshu and Jain, Shivani and Saban, Nicholas
          and Maheshwari, Chirag and Kannappan, Anand},
  journal={https://cdn.patronus.ai/FigmaTrace.pdf},
  year={2026}
}

Glossary

norm-1000 coordinates — screen positions rescaled so both axes run 0–1000, independent of actual resolution.
Phase — a contiguous span of a design session labeled from a closed taxonomy (e.g. blocking_layout, componentising, refinement_polish as listed in the paper).
observe probe — a synthetic no-input step inserted where the screen changes on its own, such as a render completing or a plugin loading.
directed / undirected — whether the agent receives step-wise instructions or only an open-ended goal.

Model Card Authors

Patronus AI

Model Card Contact

darshan@patronus.ai