Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a modern Moravec's paradox of VLMs — strong at complex offline reasoning, yet slow and compute-heavy on simple real-time streaming perception. Instead of decoding video into uniformly-sampled frames and pushing a dense grid of patch tokens through a frozen web-pretrained ViT, Mage-VL follows the structure of modern video codecs: it separates a stream into anchor (I) frames and predicted (P) frames, keeps every anchor patch, and retains only the predicted-frame patches where the codec spends bits — the regions carrying real motion and new detail. This codec-aligned sparsity cuts visual tokens by over 75% while preserving spatio-temporal context, yielding up to 3.5× wall-clock inference speedup over uniform frame sampling.
The system pairs two components:
Mage-ViT — a from-scratch Codec-ViT visual encoder that allocates tokens by codec-derived spatio-temporal importance, on a shared 16×16 patch grid with 3D rotary position encoding. It is codec-agnostic: the same interface accepts a traditional codec (H.264/AVC, HEVC/H.265) via motion vectors + residual energy, or a neural codec (DCVC-RT) via its learned rate map — no architecture or retraining change.
Qwen3-4B causal decoder — a Qwen3-4B-Instruct-2507 language backbone (the only pretrained component) that consumes Mage-ViT's variable-length token stream through a lightweight two-layer MLP projector, with a unified interface for images, short/long/ultra-long video, and streaming.
On top of this pair, a System 1 & System 2 dual-process design adds proactive streaming inside a single model: a lightweight cognition gate (System 1) watches each rolling codec window and stays silent on routine content, invoking the full VLM (System 2) only when a response-worthy event completes — no multi-agent pipeline required.
✨ Highlights
Codec-native & from scratch. The entire visual stack is trained from scratch — no billion-scale image-text ViT initialization. The bio-inspired predictive-patch mechanism (I/P frames at 16×16) cuts visual-token consumption by over 75% (~1/8 or less of dense frame sampling), letting the model train on videos 8× longer under the same budget.
Codec-native speedup. Codec tokenization sets a superior accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling at matched accuracy, and the fastest of all compared models on most video benchmarks (single 8×B200 node).
Data-efficient tokenizer. Trained on only ~100M unlabeled images/videos, Mage-ViT matches or beats frontier encoders trained on billions of image-text pairs (SigLIP2 @ 10B, MoonViT @ 2B) — e.g. 99.33% on CIFAR-10 and 85.69% on ImageNet with 256 tokens, showing web-scale pretraining is not essential for a strong VLM front-end.
Native-resolution scaling. Variable-resolution pretraining lets Mage-ViT improve monotonically with the token budget (peaking >96.1% Food-101 / >86.3% ImageNet at 676 tokens) where fixed-resolution encoders saturate or degrade.
Matched-LLM video gains. With the 4B Qwen3 backbone held fixed and only the ViT swapped, Mage-VL improves over Qwen3-VL-4B on every reported video and temporal-grounding benchmark — largest on localization-heavy tasks (+22.5 QVHighlight, +17.1 ActivityNet, +11.0 VSI-Bench, +24.5 VideoEval-Pro).
Strong for its size. On par with Qwen3-VL-4B on static images, and clearly ahead on video understanding and spatial intelligence (+11.0 VSI-Bench, +53.1 CrossPoint, +5.2 EmbSpatial, +22.5 QVHighlight).
Proactive streaming, single model. A frozen-backbone cognition gate delivers low-latency, event-gated commentary; it tops TimVal / F1 / ROC-AUC / PR-AUC on SoccerNet streaming and generalizes to real 2026 World Cup broadcasts.
📥 Model
A single checkpoint, microsoft/Mage-VL, is one unified model that simultaneously provides image & video understanding and the proactive streaming gate — the same weights answer offline image/video questions and drive event-gated commentary. It covers every Mage-VL capability: image understanding, frame-sampled video, traditional H.264/HEVC codec video, neural DCVC-RT codec video, and event-gated streaming. The repository bundles the codec processor, the neural codec package, and the proactive gate weights — no separate understanding, NVC, or streaming checkpoint is required.
We additionally release microsoft/Mage-ViT — the standalone visual encoder from the two-stage, from-scratch ViT pre-training (cluster-discrimination on ~100M unlabeled image/video frames). This is the ViT-pre-trained checkpoint only: it has not gone through the joint VLM training with the language model. Use it as a data-efficient, codec-native visual encoder or as a drop-in ViT for your own multimodal training.
Model
Task
Backbone
Hugging Face
Mage-VL
image & video understanding + proactive streaming gate
Proactive streaming framework — Mage-ViT incrementally encodes the continuous stream into codec-native visual features shared by the event gate and the causal decoder. The gate scores each rolling window and stays silent on routine content; when it opens, the decoder emits an event-conditioned response.
Mage-ViT — a from-scratch Codec-ViT visual encoder. On a 16×16 patch grid it keeps all anchor (I) frame patches and only the motion-salient predicted (P) frame patches, cutting visual tokens by over 75% while a shared 3D RoPE preserves spatio-temporal positions.
Mage-VL — a unified model where projected visual tokens and text tokens share one causal Qwen3 decoder. Still images become a single spatial block; videos become temporally-ordered codec windows. In streaming mode, a lightweight cognition gate predicts p_speak = g(h_t) per rolling window (over a recurrent streaming memory kept by an event-preserving feature extractor) and triggers generation when p_speak ≥ τ; the response is decoded by the frozen base model from a local sliding window of the most recent codec segments, and a text query can be injected at any time.
Training — a progressive five-stage supervised curriculum (no preference/RL post-training) that produces one unified model:
Instruction tuning + short temporal grounding — ~54M image-instruction samples + 3.4M 30–180s video captions.
Temporal-horizon expansion — medium/long video (LLaVA-Video, TimeLens, VideoChat-Flash, Molmo2) with retained image SFT.
Codec-native long-context adaptation — 350K long videos as rolling codec windows (up to 384/768 frames).
Proactive streaming alignment — a cognition gate fine-tuned on ~3.3M streaming samples with the visual encoder and LLM kept frozen (only the gate is trained).
The five stages together produce a single unified model, Mage-VL, that handles image understanding, offline video reasoning, and proactive streaming — no separate variants are shipped.
Two parts of the pipeline apply an AI4AI (AI-for-AI) paradigm: (1) dense recaptioning runs through an agentic closed loop where a GPT-5 rubric scorer grades captions and a Copilot coding agent co-designs the prompt and harness code (e.g. rendering timestamp overlays) under a human validation gate — improving every downstream OCR/doc/chart/perception benchmark and inspiring SkillOpt-Lite; and (2) Stage-3 uses AI-based diagnostics to decide which video categories, resolutions, and frame counts to train on.
📊 Performance
Image understanding & spatial intelligence — click to expand
Performance comparison across models. Mage-VL-4B and Qwen3-VL-4B use the same 4B Qwen3 LLM backbone; Phi-4-Multimodal-Instruct (Phi-4-MM, 5.6B) and Phi-4-Reasoning-Vision (Phi-4-R-V, 15B) are reported for reference. – = not run. Bold = best in row.
Benchmark
Mage-VL-4B
Qwen3-VL-4B
Phi-4-MM-5.6B
Phi-4-R-V-15B
Document understanding
DocVQA-val
95.14
94.69
92.79
76.20
InfoVQA-val
80.33
79.50
71.84
55.41
AI2D w/ Mask
83.16
81.54
81.83
82.87
ChartQA
84.88
83.96
83.76
83.40
OCRBench
81.80
81.60
81.70
73.90
MultiDocVQA-val
87.46
87.21
46.84
58.35
ChartQAPro
32.57
26.79
0.13
25.38
TextVQA-val
77.28
80.55
39.93
76.06
CC-OCR Doc
32.25
39.69
4.99
17.65
General VQA
MMBench-EN-dev
84.02
83.25
65.81
84.19
MMBench-CN-dev
82.04
80.58
75.17
79.47
MMStar
67.32
62.04
61.24
59.63
MME-Perception
1709.54
1703.50
1409.66
1590.21
SeedBench (All)
76.78
75.65
68.28
73.70
CV-Bench
87.79
85.37
57.09
81.31
MME-RealWorld
66.52
63.20
32.45
57.80
Spatial intelligence
CV-Bench-2D
82.13
81.00
56.12
80.11
CV-Bench-3D
94.75
92.30
56.92
82.50
BLINK
65.11
65.10
35.24
57.80
EmbSpatial
82.67
77.50
41.51
72.67
CrossPoint
80.00
26.90
12.20
47.73
CRPE-Relation
76.12
77.70
34.60
74.46
SAT
67.33
69.30
55.33
66.67
Video understanding & temporal grounding — click to expand
Bold = best in row.
Benchmark
Mage-VL-4B
Qwen3-VL-4B
Phi-4-MM-5.6B
Phi-4-R-V-15B
Video QA
MV-Bench
65.1
66.7
44.9
49.2
NextQA
83.1
79.8
54.1
69.0
VideoMME
64.0
59.7
44.7
55.3
LongVideoBench
61.3
57.7
41.14
51.2
LVBench
41.8
39.2
25.31
34.4
MLVU-dev
68.7
61.5
44.18
51.8
VideoEval-Pro
45.2
20.7
14.35
16.8
Temporal grounding
Timelens-Charades
50.7
43.1
4.09
20.6
Timelens-ActivityNet
45.4
28.4
2.03
23.0
Timelens-QVHighlight
57.4
34.9
2.47
11.6
Spatial reasoning
VSI-Bench
64.3
53.3
24.09
25.5
Tracking (J&F)
Ref-DAVIS17
25.83
7.48
3.14
2.15
MeViS-ValidU
22.55
3.16
10.28
1.53
ReasonVOS
17.76
9.66
9.50
9.77
Ref-YT-VOS
25.57
5.28
8.64
3.85
Proactive streaming (SoccerNet) & online video (OVO-Bench) — click to expand
SoccerNet — response timing (StreamMind protocol, codec-native inputs, zero-tolerance canvas matching). Bold = best in column.
Method
TriggerAcc
TimVal
F1
ROC-AUC
PR-AUC
StreamMind
52.18
47.36
–
–
–
JoyAI-VL-Interaction-9B
97.98
19.25
3.55
56.26
1.68
Mage-VL-4B
79.21
55.54
16.35
83.14
9.30
JoyAI's high TriggerAcc comes from predicting silence almost everywhere under SoccerNet's heavy class imbalance, so it collapses on the precision-sensitive metrics; StreamMind is trained in-distribution on SoccerNet, whereas Mage-VL is not.
OVO-Bench — online video understanding (SimpleStream recent-window protocol, 4 frames @ 1 fps; no streaming-specific fine-tuning). Mage-VL sets a new state-of-the-art overall score among streaming architectures. RT-Avg / BT-Avg are the Real-Time Visual Perception / Backward Tracing sub-task averages; Overall is their mean. Bold = best model per column (Human is the reference upper bound).
Beyond the model, the report distills seven empirical findings for efficient multimodal training:
Web-scale pretraining is not essential. A from-scratch backbone on ~100M unlabeled frames matches encoders trained on billions of image-text pairs.
Variable-resolution pretraining scales monotonically. Quality keeps improving with the visual-token budget instead of saturating/degrading like fixed-resolution encoders.
Codec-native tokenization sets a better accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling.
Explicit VideoQA SFT is redundant. Dense video captions + standard image SFT are sufficient for strong zero-shot VideoQA.
Motion–spatial synergy. Dynamic video training substantially improves static 2D/3D spatial reasoning.
AI4AI data pipeline. Agentic closed-loop feedback + prompt/code co-design systematically lift caption quality and downstream scores (inspired SkillOpt-Lite).
Zero-Vision SFT for multimodal RL. Bypassing visual SFT in favor of pure-text reasoning SFT unlocks stronger multimodal RL — a compute-efficient path.
🚀 Quick Start
A single checkpoint, microsoft/Mage-VL, covers every capability below.
Capability
Script
How to run
Image understanding
inference.py
--mode offline --image
Frame-sampled video
inference.py
--mode offline --video --video-backend frames
Traditional H.264/HEVC codec video
inference.py
--mode offline --video --video-backend codec --codec-engine traditional
Download inference.py. Offline mode loads the checkpoint with AutoModelForCausalLM.from_pretrained and supports images, frame sampling, and both codec engines:
bash
1# image2python inference.py --mode offline --image examples/dog.jpg \3 --question "Describe this image in detail."
The image depicts a dog sitting on a patterned rug. The dog appears to be a
medium-sized breed with a thick, fluffy coat. Its fur is primarily white with
patches of black and brown. The dog's ears are perked up, and it has a calm and
attentive expression. [...]
bash
1# video — uniform frame sampling2python inference.py --mode offline --video examples/soccer-broadcast.mp4 \3 --video-backend frames --num-frames 32\4 --question "Describe this video."
The video opens with a man in a black polo shirt, sporting a short haircut,
standing in a stadium. He is holding a yellow microphone with the BBC Sport
logo on it. The background reveals a large crowd of spectators. [...]
bash
1# video — traditional codec (HEVC/H.264)2python inference.py --mode offline --video examples/soccer-broadcast.mp4 \3 --video-backend codec --codec-engine traditional --num-frames 32\4 --question "Describe this video."
The video opens with a BBC Sport broadcast, featuring a presenter in a black
shirt holding a yellow microphone. The background reveals a packed stadium,
with the scoreboard displaying "ENG 1 ARG 2 FT", indicating the final score of
the match. [...]
The video opens with a BBC Sport broadcast, featuring a presenter standing in a
stadium filled with spectators. The presenter, dressed in a black shirt, holds
a yellow BBC Sport microphone and wears a black earpiece. [...]
Online inference
Online mode talks to an OpenAI-compatible SGLang server. First build and launch the server with the Mage-VL SGLang branch (building it needs protobuf-compiler and a Rust toolchain):
Use --model, --max-new-tokens, and --api-key to override their defaults.
Streaming inference
streammind_gate.safetensors in this repository holds the event gate. Streaming inference splits a video into non-overlapping segments, stays silent on routine content, and generates a caption only when a response-worthy event is detected. Run it with inference_streaming.py from the GitHub repository:
1[t=0.0-8.0s] gate=silence (p=0.19)
2[t=8.0-16.0s] gate=response (p=0.55) -> The video features a live sports broadcast from BBC Sport, set in a large stadium filled with spectators. The broadcast focuses on a football match between England and Argentina, with the score displayed as England 1, Argentina 2. [...]
3[t=16.0-24.0s] gate=response (p=0.73) -> The video features a sports broadcast set in a large stadium filled with spectators. Four commentators are gathered around a table with a 'BBC Sport' logo, each holding a yellow microphone. [...]
4[t=24.0-30.0s] gate=silence (p=0.31)
The gate is trained on codec inputs, so --video_backend codec is the intended setting. Use --video_backend frames for direct frame sampling. Additional controls include --num_frames, --cur_fps, --max_segments, --max_new_tokens, --gate_threshold, and --attn_impl.
📝 Citation
bibtex
1@article{yang2026mage,
2 title={Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
3 author={Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and Li, Xiao and Zhang, Peng and others},
4 journal={arXiv preprint arXiv:2607.24904},
5 year={2026}
6}
