Share and message this to Bob Steve. Click send button
Robot Manipulation
Pick Place Hotdog Sausage
Put Mushroom Place Pot
Push Cloth Left to Right (Out-of-Dist.)
Gaming
Task: Model controls the robot to collect green blocks.
Magma v.s. LLaVA-OneVision
Magma v.s. GPT4o-minni
Model Details
Model Description
Magma is a multimodal agentic AI model that can generate text based on the input text and image. The model is designed for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, in particular the multimodal agentic AI. The main innovation of this model lies on the introduction of two technical innovations: Set-of-Mark and Trace-of-Mark, and the leverage of a large amount of unlabeled video data to learn the spatial-temporal grounding and planning. Please refer to our paper for more technical details.
Highlights
Digital and Physical Worlds: Magma is the first-ever foundation model for multimodal AI agents, designed to handle complex interactions across both virtual and real environments!
Versatile Capabilities: Magma as a single model not only possesses generic image and videos understanding ability, but also generate goal-driven visual plans and actions, making it versatile for different agentic tasks!
State-of-the-art Performance: Magma achieves state-of-the-art performance on various multimodal tasks, including UI navigation, robotics manipulation, as well as generic image and video understanding, in particular the spatial understanding and reasoning!
Scalable Pretraining Strategy: Magma is designed to be learned scalably from unlabeled videos in the wild in addition to the existing agentic data, making it strong generalization ability and suitable for real-world applications!
License
The model is developed by Microsoft and is funded by Microsoft Research. The model is shared by Microsoft Research and is licensed under the MIT License.
How to Get Started with the Model
To get started with the model, you first need to make sure that transformers and torch are installed, as well as installing the following dependencies:
pip install torchvision Pillow open_clip_torch
⚠️ Please note that you need to install our customized transformers lib:
The data collection process involved sourcing information from publicly available documents, with a meticulous approach to filtering out undesirable documents and images. To safeguard privacy, we carefully filtered various image and text data sources to remove or scrub any potentially personal data from the training data.
In addition to the text-related preprocessing, we mainly undertake the following image and video preprocessing steps:
UI Grounding and Navigation Data: For each UI screenshot, we extract the bounding boxes for the UI elements, and apply Set-of-Mark Prompting to overlay numeric marks on the raw image. The model is trained to generate the UI grounding text based on the image and the Set-of-Mark prompts.
Instruction Video Data: For each video clip, we apply Co-Tracker to extract the grid traces and then apply filtering algorithm to remove the noisy or static points. For videos that bear camera motion, we further apply homography transformation to stabilize the video clips. In the end, we assign a numeric mark for each trace which gives us a set of trace-of-mark. The model is trained to generate the trace-of-mark given the video clips and instructional text.
Robotics Manipulation Data: For robotics data in Open-X Embodiment, we extract the 7 DoF robot gripper state and also extract the trace-of-mark from the video clips. Similar filtering and stabilization steps are applied to the video clips. The model is trained to generate the robot manipulation action as well as the trace-of-mark given the video clips and instructional text.
After all these preprocessing, we combine them with existing text annotations to form our final multimodal training data. We refer to our paper for more technical details.
Training Hyperparameters
We used bf16 mixed precision for training on H100s and MI300s. We used the following hyperparameters for training:
Batch size: 1024
Learning rate: 1e-5
Max sequence length: 4096
Resolution: maximally 1024x1024 for image, 512x512 for video frame.
Pretraining Epochs: 3
Evaluation
We evaluate the model in zero-shot manner on a wide range of tasks, mostly agent-related tasks.
Testing Data, Factors & Metrics
Zero-shot Testing Data
We evaluate the model's zero-shot performance on the following datasets:
We follow the individual dataset's evaluation metrics for the evaluation. Please refer to the original dataset for more details.
Results on Agentic Intelligence
Zero-shot evaluation on agentic intelligence. We report the results for pretrained Magma without any domain-specific finetuning. Magma is the only model that can conduct the full task spectrum.
Model
VQAv2
TextVQA
POPE
SS-Mobile
SS-Desktop
SS-Web
VWB-Ele-G
VWB-Act-G
SE-Google Robot
SE-Bridge
GPT-4V
77.2
78.0
n/a
23.6
16.0
9.0
67.5
75.7
-
-
GPT-4V-OmniParser
n/a
n/a
n/a
71.1
45.6
58.5
-
-
-
-
LLava-1.5
78.5
58.2
85.9
-
-
-
12.1
13.6
-
-
LLava-Next
81.3
64.9
86.5
-
-
-
15.0
8.7
-
-
Qwen-VL
78.8
63.8
n/a
6.2
6.3
3.0
14.0
0.7
-
-
Qwen-VL-Chat
78.2
61.5
n/a
-
-
-
-
-
-
-
Fuyu
74.2
n/a
n/a
21.2
20.8
19.2
19.4
15.5
-
-
SeeClick
-
-
-
65.0
51.1
44.1
9.9
1.9
-
-
Octo
-
-
-
-
-
-
-
-
-
-
RT-1-X
-
-
-
-
-
-
-
-
6.0
15.9
OpenVLA
-
-
-
-
-
-
-
-
34.2
1.1
Magma-8B
80.0
66.5
87.4
59.5
64.1
60.6
96.3
71.8
52.3
35.4
Notes: SS - ScreenSpot, VWB - VisualWebArena, SE - SimplerEnv
Technical Specifications
Model Architecture and Objective
Language Model: We use Meta LLama-3 as the backbone LLM.
Vision Encoder: We use CLIP-ConvneXt-XXLarge trained by LAION team as the vision encoder to tokenize the images and videos.
The whole pipeline follows the common practice in the multimodal LLMs, where the vision encoder is used to tokenize the images and videos, and then the visual tokens are fed into the LLM along with the textual tokens to generate the text outputs.
This model is intended for broad research use in English. It is designed only for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, particularly in multimodal agentic AI. It is intended to be used by domain experts who are independently capable of evaluating the quality of outputs before acting on them.
Direct Use
The model takes images and text as inputs, and produces the textual outputs for the following uses:
Image/Video-Conditioned Text Generation: The model can generate text (e.g., descriptions, answers) based on the input text and image.
Visual Planning Capabilities: The model can also produce the visual trace as the future planning to accomplish a task (e.g., move object from one place to another).
Agentic Capabilities: The model can also generate UI grounding (e.g., click ``search'' button) and robotics manipulations (e.g., 7 DoF for the robot gripper).
Downstream Use
The model can be further finetuned for different downstream tasks, such as:
Image Captioning and QA: We can further finetune this model for image captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better spatial understanding and reasoning on these tasks.
Video Captioning and QA: We can further finetune this model for video captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better temporal understanding and reasoning on these tasks.
UI Navigation: We can finetune this model for specific UI navigation tasks, such as web navigation or mobile navigation. The model can achieve superior performance on these tasks.
Robotics Manipulation: Our model can be further finetuned for robotics tasks given its general agentic capabilities as a vision-language-action model. After finetuning, our model significantly outperforms the state-of-the-art models such as OpenVLA on robotics manipulation tasks.
Bias, Risks, and Limitations
Please note that this model is not specifically designed or evaluated for all downstream purposes.
The model is not intended to be deployed in production settings. It should not be used in high-risk scenarios, such as military and defense, financial services, and critical infrastructure systems.
Developers should consider common limitations of multimodal models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case.
Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case. Like other multimodal models, Magma can potentially behave in ways that are unfair, unreliable, or offensive.
The models' outputs do not reflect the opinions of Microsoft.
Some of the limiting behaviors to be aware of include:
Quality of Service: The model is trained primarily on English text. Languages other than English will experience worse performance. English language varieties with less representation in the training data might experience worse performance than standard American English. Magma is not intended to support multilingual use.
Representation of Harms & Perpetuation of Stereotypes: These models can over- or under-represent groups of people, erase representation of some groups, or reinforce demeaning or negative stereotypes. Despite safety post-training, these limitations may still be present due to differing levels of representation of different groups or prevalence of examples of negative stereotypes in training data that reflect real-world patterns and societal biases.
Inappropriate or Offensive Content: These models may produce other types of inappropriate or offensive content, which may make it inappropriate to deploy for sensitive contexts without additional mitigations that are specific to the use case.
Information Reliability: Multimodal models can generate nonsensical content or fabricate content that might sound reasonable but is inaccurate or outdated.
Developers should apply responsible AI best practices and are responsible for ensuring that a specific use case complies with relevant laws and regulations (e.g. privacy, trade, etc.). Using safety services like Azure AI Content Safety that have advanced guardrails is highly recommended.
Recommendations
Magma was developed for research purposes only. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
The recommended usage for the finetuned models is within the research settings they were trained on — namely,
an android simulator running on a computer for UI manipulation.
an enclosure equipped with a robotic arm and everyday objects for Robotic manipulation
For UI navigation task, researchers should make sure a human is in the loop and in control for every action the agentic system generates. Since the model cannot act by itself, the sub-module a researcher uses to actually perform the UI navigation action should ensure no unintended consequences can occur as a result of performing the UI action proposed by the model.
For the robotic manipulation task, some mitigation strategies to use for human safety when operating robotic arms include:
Safety Zones and Barriers: Establish physical barriers or safety zones around robotic workspaces to prevent unauthorized access.
Emergency Stop Systems: Equip robotic arms with easily accessible emergency stop buttons. Implement a fail-safe mechanism that triggers an immediate stop of operations in case of an emergency
Safety Standards and Compliance: Adhere to established safety standards (e.g., ISO 10218, ISO/TS 15066) for industrial robots and collaborative robots.
User Training and Awareness: Provide comprehensive training for all personnel working around robotic arms to understand their functions, safety features, and emergency procedures. Promote awareness of the potential risks associated with robotic manipulation.
Citation
bibtex
1@misc{yang2025magmafoundationmodelmultimodal,
2 title={Magma: A Foundation Model for Multimodal AI Agents},
3 author={Jianwei Yang and Reuben Tan and Qianhui Wu and Ruijie Zheng and Baolin Peng and Yongyuan Liang and Yu Gu and Mu Cai and Seonghyeon Ye and Joel Jang and Yuquan Deng and Lars Liden and Jianfeng Gao},
4 year={2025},
5 eprint={2502.13130},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2502.13130},
9}
