MATPO allows planner and worker agents to coexist within a single LLM and be trained via RL, achieving an 18.38% relative improvement over single-agent baselines on GAIA-text, FRAMES, and WebWalker-QA.
News & Updates
[2025-Oct-08] MATPO-Qwen3-14B checkpoints and rollouts released
[2025-Oct-08] Code and training scripts released
[2025-Oct-06] Arxiv Paper released
Overview
MATPO (Multi-Agent Tool-Integrated Policy Optimization) is a novel reinforcement learning framework that enables training multiple specialized agent roles (planner and worker agents) within a single large language model.
The Problem
Current single-agent approaches for multi-turn tool-integrated planning face critical limitations:
Context Length Bottleneck: Tool responses (e.g., web scraping) consume excessive tokens, making long-range planning prohibitive
Noisy Tool Responses: Raw tool responses interfere with the model's attention and planning capabilities
Our Solution
MATPO introduces a multi-agent-in-one-model architecture where:
A planner-agent orchestrates high-level planning and delegates subtasks
Worker-agents handle specific browsing and search tasks with isolated contexts
Both roles are trained within a single LLM using role-specific prompts via reinforcement learning
Key Features
Multi-Agent-in-One-Model: Train planner and worker agents within a single LLM using role-specific system prompts
Principled Credit Assignment: Extends GRPO with theoretically grounded reward distribution across planner and worker rollouts
Easy Integration: Built on top of veRL, compatible with existing RL training frameworks
Robust Training: More stable learning curves compared to single-agent approaches, especially with noisy tool responses
Infrastructure Efficient: No need for deployment of separate models or additional rollout engines
MATPO Architecture
MATPO employs a hierarchical multi-agent framework where a single LLM serves multiple roles:
User Query → Planner Agent → Subtask 1 → Worker Agent → Result 1
→ Subtask 2 → Worker Agent → Result 2
→ ...
→ Final Answer
Single-agent GRPO Framework
MATPO Framework
Comparison between the rollout trajectories between the single-agent GRPO (top) and the multi-agent MATPO (bottom).
Multi-Agent Rollout Process
Planner Agent:
Receives user query with planner-specific system prompt
Generates high-level plan and decomposes it into subtasks
Delegates subtasks to worker agents
Synthesizes worker responses into final answer
Worker Agent:
Receives subtask with worker-specific system prompt
We release the trained Qwen3-14B-base model checkpoints at the 180th training step of both single-agent GRPO and MATPO.
The associated model rollouts across various training steps can be found here.
Key Findings
More Stable Training: MATPO exhibits more stable learning curves and avoids catastrophic performance drops observed in single-agent training
Robustness to Noise: Multi-agent decomposition effectively isolates noisy tool responses, preventing them from interfering with high-level planning
Better Credit Assignment: Principled reward distribution across planner and worker rollouts leads to more effective learning
Practical Implementation Tips
Based on our experiments, we recommend:
Final Summary: Final summaries from worker agents are critical for clean planner-worker interfaces
Query Recap: Recapping original user query in worker prompt significantly improves performance
URL Blocking: Remember to blocking HuggingFace search results to avoid data leakage
Citation
If you find MATPO helpful in your research, please consider citing our paper:
bibtex
1@misc{mo2025multiagenttoolintegratedpolicyoptimization,
2 title={Multi-Agent Tool-Integrated Policy Optimization},
3 author={Zhanfeng Mo and Xingxuan Li and Yuntao Chen and Lidong Bing},
4 year={2025},
5 eprint={2510.04678},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2510.04678},
9}
Acknowledgments
We would like to thank:
VolcEngine for developing and open-sourcing veRL, the RL training framework that powers MATPO
Alibaba Cloud for the Qwen3 model series
Google for the Serper API that enables web search capabilities
The authors of GAIA, WebWalkerQA, FRAMES, and MuSiQue datasets
The open-source community for valuable feedback and contributions
FAQ
Q: What's the difference between MATPO and traditional multi-agent systems?
MATPO uses a single LLM to play multiple agent roles via different system prompts, rather than deploying separate models. This offers:
Lower infrastructure complexity
Better parameter efficiency
Easier deployment and maintenance
Compatible with existing RL frameworks
Q: Can I use MATPO with models other than Qwen3?
Yes! MATPO is model-agnostic. You can use any decoder-only LLM that supports tool calling and multi-turn conversations. We've tested with Qwen3-14B-base, but models like Llama 3, Mistral, or other reasoning-capable LLMs should work.
Q: How many GPUs do I need for training?
For Qwen3-14B-base, we recommend:
Training: 8x A100/A800 GPUs (80GB)
Inference: 1-2x A100/A800 GPUs (40GB/80GB)
Q: How does MATPO handle credit assignment?
MATPO extends GRPO with principled credit assignment:
The planner's final answer determines the accuracy reward
This reward is normalized across all rollouts in a group
Gradients flow proportionally to both planner and worker actions
Worker agents receive the same advantage value as their parent planner rollout
See our paper for more details.
Q: Can I use MATPO for tasks other than web search?
Absolutely! While our paper focuses on web search, MATPO's framework is general. You can extend it to:
Code generation with execution feedback
Scientific reasoning with calculator tools
Data analysis with pandas/SQL tools
Any multi-turn task with verifiable rewards
Q: How stable is MATPO training compared to single-agent RL?
MATPO is significantly more stable. Our experiments show:
Single-agent GRPO often suffers catastrophic drops after step 120
MATPO maintains steady improvement throughout training
Q: Do I need to block HuggingFace URLs during training?
For research integrity, yes - especially if your evaluation benchmarks are hosted on HuggingFace. This prevents models from "cheating" by finding ground-truth answers online.
For production systems with no data leakage concerns, this is optional.
