A 16-bit floating-point format designed for faster computation while retaining good precision.
Provides similar dynamic range as FP32 but with lower memory usage.
Recommended if your hardware supports BF16 acceleration (check your device's specs).
Ideal for high-performance inference with reduced memory footprint compared to FP32.

📌 Use BF16 if:
✔ Your hardware has native BF16 support (e.g., newer GPUs, TPUs).
✔ You want higher precision while saving memory.
✔ You plan to requantize the model into another format.

📌 Avoid BF16 if:
❌ Your hardware does not support BF16 (it may fall back to FP32 and run slower).
❌ You need compatibility with older devices that lack BF16 optimization.

F16 (Float 16) – More widely supported than BF16

A 16-bit floating-point high precision but with less of range of values than BF16.
Works on most devices with FP16 acceleration support (including many GPUs and some CPUs).
Slightly lower numerical precision than BF16 but generally sufficient for inference.

📌 Use F16 if:
✔ Your hardware supports FP16 but not BF16.
✔ You need a balance between speed, memory usage, and accuracy.
✔ You are running on a GPU or another device optimized for FP16 computations.

📌 Avoid F16 if:
❌ Your device lacks native FP16 support (it may run slower than expected).
❌ You have memory limitations.

Quantized Models (Q4_K, Q6_K, Q8, etc.) – For CPU & Low-VRAM Inference

Quantization reduces model size and memory usage while maintaining as much accuracy as possible.

Lower-bit models (Q4_K) → Best for minimal memory usage, may have lower precision.
Higher-bit models (Q6_K, Q8_0) → Better accuracy, requires more memory.

📌 Use Quantized Models if:
✔ You are running inference on a CPU and need an optimized model.
✔ Your device has low VRAM and cannot load full-precision models.
✔ You want to reduce memory footprint while keeping reasonable accuracy.

📌 Avoid Quantized Models if:
❌ You need maximum accuracy (full-precision models are better for this).
❌ Your hardware has enough VRAM for higher-precision formats (BF16/F16).

Very Low-Bit Quantization (IQ3_XS, IQ3_S, IQ3_M, Q4_K, Q4_0)

These models are optimized for extreme memory efficiency, making them ideal for low-power devices or large-scale deployments where memory is a critical constraint.

IQ3_XS: Ultra-low-bit quantization (3-bit) with extreme memory efficiency.
- Use case: Best for ultra-low-memory devices where even Q4_K is too large.
- Trade-off: Lower accuracy compared to higher-bit quantizations.
IQ3_S: Small block size for maximum memory efficiency.
- Use case: Best for low-memory devices where IQ3_XS is too aggressive.
IQ3_M: Medium block size for better accuracy than IQ3_S.
- Use case: Suitable for low-memory devices where IQ3_S is too limiting.
Q4_K: 4-bit quantization with block-wise optimization for better accuracy.
- Use case: Best for low-memory devices where Q6_K is too large.
Q4_0: Pure 4-bit quantization, optimized for ARM devices.
- Use case: Best for ARM-based devices or low-memory environments.

Summary Table: Model Format Selection

Model Format	Precision	Memory Usage	Device Requirements	Best Use Case
BF16	Highest	High	BF16-supported GPU/CPUs	High-speed inference with reduced memory
F16	High	High	FP16-supported devices	GPU inference when BF16 isn't available
Q4_K	Medium Low	Low	CPU or Low-VRAM devices	Best for memory-constrained environments
Q6_K	Medium	Moderate	CPU with more memory	Better accuracy while still being quantized
Q8_0	High	Moderate	CPU or GPU with enough VRAM	Best accuracy among quantized models
IQ3_XS	Very Low	Very Low	Ultra-low-memory devices	Extreme memory efficiency and low accuracy
Q4_0	Low	Low	ARM or low-memory devices	llama.cpp can optimize for ARM devices

Included Files & Details

`Magma-8B-bf16.gguf`

Model weights preserved in BF16.
Use this if you want to requantize the model into a different format.
Best if your device supports BF16 acceleration.

`Magma-8B-f16.gguf`

Model weights stored in F16.
Use if your device supports FP16, especially if BF16 is not available.

`Magma-8B-bf16-q8_0.gguf`

Output & embeddings remain in BF16.
All other layers quantized to Q8_0.
Use if your device supports BF16 and you want a quantized version.

`Magma-8B-f16-q8_0.gguf`

Output & embeddings remain in F16.
All other layers quantized to Q8_0.

`Magma-8B-q4_k.gguf`

Output & embeddings quantized to Q8_0.
All other layers quantized to Q4_K.
Good for CPU inference with limited memory.

`Magma-8B-q4_k_s.gguf`

Smallest Q4_K variant, using less memory at the cost of accuracy.
Best for very low-memory setups.

`Magma-8B-q6_k.gguf`

Output & embeddings quantized to Q8_0.
All other layers quantized to Q6_K .

`Magma-8B-q8_0.gguf`

Fully Q8 quantized model for better accuracy.
Requires more memory but offers higher precision.

`Magma-8B-iq3_xs.gguf`

IQ3_XS quantization, optimized for extreme memory efficiency.
Best for ultra-low-memory devices.

`Magma-8B-iq3_m.gguf`

IQ3_M quantization, offering a medium block size for better accuracy.
Suitable for low-memory devices.

`Magma-8B-q4_0.gguf`

Pure Q4_0 quantization, optimized for ARM devices.
Best for low-memory environments.
Prefer IQ4_NL for better accuracy.

🚀 If you find these models useful

❤ Please click "Like" if you find this useful!
Help me test my AI-Powered Network Monitor Assistant with quantum-ready security checks:
👉 Quantum Network Monitor

💬 How to test:
Choose an AI assistant type:

TurboLLM (GPT-4o-mini)
HugLLM (Hugginface Open-source)
TestLLM (Experimental CPU-only)

What I’m Testing

I’m pushing the limits of small open-source models for AI network monitoring, specifically:

Function calling against live network services
How small can a model go while still handling:
- Automated Nmap scans
- Quantum-readiness checks
- Network Monitoring tasks

🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads):

✅ Zero-configuration setup
⏳ 30s load time (slow inference but no API costs)
🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!

Other Assistants

🟢 TurboLLM – Uses gpt-4o-mini for:

Create custom cmd processors to run .net code on Quantum Network Monitor Agents
Real-time network diagnostics and monitoring
Security Audits
Penetration testing (Nmap/Metasploit)

🔵 HugLLM – Latest Open-source models:

🌐 Runs on Hugging Face Inference API

💡 Example commands to you could test:

"Give me info on my websites SSL certificate"
"Check if my server is using quantum safe encyption for communication"
"Run a comprehensive security audit on my server"
'"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code from. This is a very flexible and powerful feature. Use with caution!

Final Word

I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.

If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.

I'm also open to job opportunities or sponsorship.

Thank you! 😊

Model Card for Magma-8B

Magma: A Foundation Model for Multimodal AI Agents

Jianwei Yang^*¹^† Reuben Tan¹^† Qianhui Wu¹^† Ruijie Zheng²^‡ Baolin Peng¹^‡ Yongyuan Liang²^‡

Yu Gu¹ Mu Cai³ Seonghyeon Ye⁴ Joel Jang⁵ Yuquan Deng⁵ Lars Liden¹ Jianfeng Gao¹^▽

¹ Microsoft Research; ² University of Maryland; ³ University of Wisconsin-Madison
⁴ KAIST; ⁵ University of Washington

^* Project lead ^† First authors ^‡ Second authors ^▽ Leadership

[arXiv Paper] [Project Page] [Hugging Face Paper] [Github Repo] [Video]

Agents

UI Navigation

What's weather in Seattle? & turn on flight mode

Share and message this to Bob Steve. Click send button

Robot Manipulation

Pick Place Hotdog Sausage

Put Mushroom Place Pot

Push Cloth Left to Right (Out-of-Dist.)

Gaming

Task: Model controls the robot to collect green blocks.

Magma v.s. LLaVA-OneVision

Magma v.s. GPT4o-minni

Model Details

Model Description

Magma is a multimodal agentic AI model that can generate text based on the input text and image. The model is designed for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, in particular the multimodal agentic AI. The main innovation of this model lies on the introduction of two technical innovations: Set-of-Mark and Trace-of-Mark, and the leverage of a large amount of unlabeled video data to learn the spatial-temporal grounding and planning. Please refer to our paper for more technical details.

Highlights

Digital and Physical Worlds: Magma is the first-ever foundation model for multimodal AI agents, designed to handle complex interactions across both virtual and real environments!
Versatile Capabilities: Magma as a single model not only possesses generic image and videos understanding ability, but also generate goal-driven visual plans and actions, making it versatile for different agentic tasks!
State-of-the-art Performance: Magma achieves state-of-the-art performance on various multimodal tasks, including UI navigation, robotics manipulation, as well as generic image and video understanding, in particular the spatial understanding and reasoning!
Scalable Pretraining Strategy: Magma is designed to be learned scalably from unlabeled videos in the wild in addition to the existing agentic data, making it strong generalization ability and suitable for real-world applications!

License

The model is developed by Microsoft and is funded by Microsoft Research. The model is shared by Microsoft Research and is licensed under the MIT License.

How to Get Started with the Model

To get started with the model, you first need to make sure that transformers and torch are installed, as well as installing the following dependencies:

pip install torchvision Pillow open_clip_torch

⚠️ Please note that you need to install our customized transformers lib:

pip install git+https://github.com/jwyang/transformers.git@dev/jwyang-v4.48.2

See here for the reason why you need this.

Then you can run the following code:

python

import torch
from PIL import Image
from io import BytesIO
import requests

from transformers import AutoModelForCausalLM, AutoProcessor

# Load the model and processor
dtype = torch.bfloat16
model = AutoModelForCausalLM.from_pretrained("microsoft/Magma-8B", trust_remote_code=True, torch_dtype=dtype)
processor = AutoProcessor.from_pretrained("microsoft/Magma-8B", trust_remote_code=True)
model.to("cuda")

# Inference
url = "https://assets-c4akfrf5b4d3f4b7.z01.azurefd.net/assets/2024/04/BMDataViz_661fb89f3845e.png"
image = Image.open(BytesIO(requests.get(url, stream=True).content))
image = image.convert("RGB")

convs = [
    {"role": "system", "content": "You are agent that can see, talk and act."},
    {"role": "user", "content": "<image_start><image><image_end>\nWhat is in this image?"},
]
prompt = processor.tokenizer.apply_chat_template(convs, tokenize=False, add_generation_prompt=True)
inputs = processor(images=[image], texts=prompt, return_tensors="pt")
inputs['pixel_values'] = inputs['pixel_values'].unsqueeze(0)
inputs['image_sizes'] = inputs['image_sizes'].unsqueeze(0)
inputs = inputs.to("cuda").to(dtype)

generation_args = { 
    "max_new_tokens": 128, 
    "temperature": 0.0, 
    "do_sample": False, 
    "use_cache": True,
    "num_beams": 1,
}

with torch.inference_mode():
    generate_ids = model.generate(**inputs, **generation_args)

generate_ids = generate_ids[:, inputs["input_ids"].shape[-1] :]
response = processor.decode(generate_ids[0], skip_special_tokens=True).strip()
print(response)

Training Details

Training Data

Our training data consists of:

Generic Image SFT Data: LLaVA-Next, InfoGrpahicVQA, ChartQA_Augmented, FigureQA, TQA, ScienceQA.
Generic Video SFT Data: ShareGPT4Video and LLaVA-Video.
Instructional Video Data: Ego4d, Somethingv2, Epic-Kitchen and other related instructional videos.
Robotics Manipulation Data: Open-X-Embodiment.
UI Grounding Data: SeeClick.
UI Navigation Data: Mind2web and AITW.

The data collection process involved sourcing information from publicly available documents, with a meticulous approach to filtering out undesirable documents and images. To safeguard privacy, we carefully filtered various image and text data sources to remove or scrub any potentially personal data from the training data.

More details can be found in our paper.

Microsoft Privacy Notice

Training Procedure

Preprocessing

In addition to the text-related preprocessing, we mainly undertake the following image and video preprocessing steps:

UI Grounding and Navigation Data: For each UI screenshot, we extract the bounding boxes for the UI elements, and apply Set-of-Mark Prompting to overlay numeric marks on the raw image. The model is trained to generate the UI grounding text based on the image and the Set-of-Mark prompts.
Instruction Video Data: For each video clip, we apply Co-Tracker to extract the grid traces and then apply filtering algorithm to remove the noisy or static points. For videos that bear camera motion, we further apply homography transformation to stabilize the video clips. In the end, we assign a numeric mark for each trace which gives us a set of trace-of-mark. The model is trained to generate the trace-of-mark given the video clips and instructional text.
Robotics Manipulation Data: For robotics data in Open-X Embodiment, we extract the 7 DoF robot gripper state and also extract the trace-of-mark from the video clips. Similar filtering and stabilization steps are applied to the video clips. The model is trained to generate the robot manipulation action as well as the trace-of-mark given the video clips and instructional text.

After all these preprocessing, we combine them with existing text annotations to form our final multimodal training data. We refer to our paper for more technical details.

Training Hyperparameters

We used bf16 mixed precision for training on H100s and MI300s. We used the following hyperparameters for training:

Batch size: 1024
Learning rate: 1e-5
Max sequence length: 4096
Resolution: maximally 1024x1024 for image, 512x512 for video frame.
Pretraining Epochs: 3

Evaluation

We evaluate the model in zero-shot manner on a wide range of tasks, mostly agent-related tasks.

Testing Data, Factors & Metrics

Zero-shot Testing Data

We evaluate the model's zero-shot performance on the following datasets:

UI Grounding: ScreenSpot and VisualWebArena.
Robotics Manipulation: SimplerEnv and WidowX real robot.
Spatial Understanding and Reasoning: VSR, BLINK and SpatialEval.

Finetuned Testing Data

We evaluate the model's performance after finetuning on the following datasets:

UI Navigation: Mind2Web and AITW.
Robotics Manipulation: SimplerEnv and WidowX real robot.
Multimodal Image Understanding and Reasoning: VQAv2, GQA, MME, POPE, TextVQA, ChartQA, DocVQA.
Multimodal Video Understanding and Reasoning: Next-QA, VideoMME, MVBench.

Metrics

We follow the individual dataset's evaluation metrics for the evaluation. Please refer to the original dataset for more details.

Results on Agentic Intelligence

Zero-shot evaluation on agentic intelligence. We report the results for pretrained Magma without any domain-specific finetuning. Magma is the only model that can conduct the full task spectrum.

Model	VQAv2	TextVQA	POPE	SS-Mobile	SS-Desktop	SS-Web	VWB-Ele-G	VWB-Act-G	SE-Google Robot	SE-Bridge
GPT-4V	77.2	78.0	n/a	23.6	16.0	9.0	67.5	75.7	-	-
GPT-4V-OmniParser	n/a	n/a	n/a	71.1	45.6	58.5	-	-	-	-
LLava-1.5	78.5	58.2	85.9	-	-	-	12.1	13.6	-	-
LLava-Next	81.3	64.9	86.5	-	-	-	15.0	8.7	-	-
Qwen-VL	78.8	63.8	n/a	6.2	6.3	3.0	14.0	0.7	-	-
Qwen-VL-Chat	78.2	61.5	n/a	-	-	-	-	-	-	-
Fuyu	74.2	n/a	n/a	21.2	20.8	19.2	19.4	15.5	-	-
SeeClick	-	-	-	65.0	51.1	44.1	9.9	1.9	-	-
Octo	-	-	-	-	-	-	-	-	-	-
RT-1-X	-	-	-	-	-	-	-	-	6.0	15.9
OpenVLA	-	-	-	-	-	-	-	-	34.2	1.1
Magma-8B	80.0	66.5	87.4	59.5	64.1	60.6	96.3	71.8	52.3	35.4

Notes: SS - ScreenSpot, VWB - VisualWebArena, SE - SimplerEnv

Technical Specifications

Model Architecture and Objective

Language Model: We use Meta LLama-3 as the backbone LLM.
Vision Encoder: We use CLIP-ConvneXt-XXLarge trained by LAION team as the vision encoder to tokenize the images and videos.

The whole pipeline follows the common practice in the multimodal LLMs, where the vision encoder is used to tokenize the images and videos, and then the visual tokens are fed into the LLM along with the textual tokens to generate the text outputs.

Compute Infrastructure

We used Azure ML for our model training.

Hardware

Our model is trained on two GPUs:

Nvidia H100
AMD MI300

Software

Our model is built based on:

Intended Uses

This model is intended for broad research use in English. It is designed only for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, particularly in multimodal agentic AI. It is intended to be used by domain experts who are independently capable of evaluating the quality of outputs before acting on them.

Direct Use

The model takes images and text as inputs, and produces the textual outputs for the following uses:

Image/Video-Conditioned Text Generation: The model can generate text (e.g., descriptions, answers) based on the input text and image.
Visual Planning Capabilities: The model can also produce the visual trace as the future planning to accomplish a task (e.g., move object from one place to another).
Agentic Capabilities: The model can also generate UI grounding (e.g., click ``search'' button) and robotics manipulations (e.g., 7 DoF for the robot gripper).

Downstream Use

The model can be further finetuned for different downstream tasks, such as:

Image Captioning and QA: We can further finetune this model for image captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better spatial understanding and reasoning on these tasks.
Video Captioning and QA: We can further finetune this model for video captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better temporal understanding and reasoning on these tasks.
UI Navigation: We can finetune this model for specific UI navigation tasks, such as web navigation or mobile navigation. The model can achieve superior performance on these tasks.
Robotics Manipulation: Our model can be further finetuned for robotics tasks given its general agentic capabilities as a vision-language-action model. After finetuning, our model significantly outperforms the state-of-the-art models such as OpenVLA on robotics manipulation tasks.

Bias, Risks, and Limitations

Please note that this model is not specifically designed or evaluated for all downstream purposes.

The model is not intended to be deployed in production settings. It should not be used in high-risk scenarios, such as military and defense, financial services, and critical infrastructure systems.

Developers should consider common limitations of multimodal models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case.

Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case. Like other multimodal models, Magma can potentially behave in ways that are unfair, unreliable, or offensive.

The models' outputs do not reflect the opinions of Microsoft.

Some of the limiting behaviors to be aware of include:

Quality of Service: The model is trained primarily on English text. Languages other than English will experience worse performance. English language varieties with less representation in the training data might experience worse performance than standard American English. Magma is not intended to support multilingual use.
Representation of Harms & Perpetuation of Stereotypes: These models can over- or under-represent groups of people, erase representation of some groups, or reinforce demeaning or negative stereotypes. Despite safety post-training, these limitations may still be present due to differing levels of representation of different groups or prevalence of examples of negative stereotypes in training data that reflect real-world patterns and societal biases.
Inappropriate or Offensive Content: These models may produce other types of inappropriate or offensive content, which may make it inappropriate to deploy for sensitive contexts without additional mitigations that are specific to the use case.
Information Reliability: Multimodal models can generate nonsensical content or fabricate content that might sound reasonable but is inaccurate or outdated.

Developers should apply responsible AI best practices and are responsible for ensuring that a specific use case complies with relevant laws and regulations (e.g. privacy, trade, etc.). Using safety services like Azure AI Content Safety that have advanced guardrails is highly recommended.

Recommendations

Magma was developed for research purposes only. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

The recommended usage for the finetuned models is within the research settings they were trained on — namely,

an android simulator running on a computer for UI manipulation.
an enclosure equipped with a robotic arm and everyday objects for Robotic manipulation

For UI navigation task, researchers should make sure a human is in the loop and in control for every action the agentic system generates. Since the model cannot act by itself, the sub-module a researcher uses to actually perform the UI navigation action should ensure no unintended consequences can occur as a result of performing the UI action proposed by the model.

For the robotic manipulation task, some mitigation strategies to use for human safety when operating robotic arms include:

Safety Zones and Barriers: Establish physical barriers or safety zones around robotic workspaces to prevent unauthorized access.
Emergency Stop Systems: Equip robotic arms with easily accessible emergency stop buttons. Implement a fail-safe mechanism that triggers an immediate stop of operations in case of an emergency
Safety Standards and Compliance: Adhere to established safety standards (e.g., ISO 10218, ISO/TS 15066) for industrial robots and collaborative robots.
User Training and Awareness: Provide comprehensive training for all personnel working around robotic arms to understand their functions, safety features, and emergency procedures. Promote awareness of the potential risks associated with robotic manipulation.

Citation

bibtex

@misc{yang2025magmafoundationmodelmultimodal,
      title={Magma: A Foundation Model for Multimodal AI Agents}, 
      author={Jianwei Yang and Reuben Tan and Qianhui Wu and Ruijie Zheng and Baolin Peng and Yongyuan Liang and Yu Gu and Mu Cai and Seonghyeon Ye and Joel Jang and Yuquan Deng and Lars Liden and Jianfeng Gao},
      year={2025},
      eprint={2502.13130},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.13130}, 
}

Mungert/Magma-8B-GGUF

Magma-8B GGUF Models

Model Generation Details

Choosing the Right Model Format

BF16 (Brain Float 16) – Use if BF16 acceleration is available