verl: Volcano Engine Reinforcement Learning for LLMs

• Easy extension of diverse RL algorithms: The hybrid-controller programming model enables flexible representation and efficient execution of complex post-training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code.
• Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc
• Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
• Ready integration with popular HuggingFace models

• State-of-the-art throughput: SOTA LLM training and inference engine integrations and SOTA RL throughput.
• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

News

• [2026/01] verl has been migrated to the verl-project
• [2026/01] verl first meetup was successfully held in Shanghai on 01/10, hosted by Volcengine and NVIDIA, the slides has been uploaded to verl-data.
• [2026/01] The recipe directory has been migrated to a dedicated repository: verl-recipe and added as a submodule. See https://github.com/volcengine/verl/pull/4795. It can be used as it was after git submodule update --init --recursive recipe. Note that transfer_queue, fully_async_policy, one_step_off_policy and vla are kept under verl/experimental since they are planned to be merged into the main library. Use them through verl.experimental.{module}.
• [2025/12] Mind Lab successfully used verl and Megatron-bridge to train GRPO Lora for Trillion-parameter model on 64 H800 - See their techblog.
• [2025/10] verl is presented in the PyTorch Conference 2025.
• [2025/08] verl is presented in the PyTorch Expert Exchange Webinar. Slides available.
• [2025/07] The ReTool recipe is fully open sourced. Blog
• [2025/07] The first verl meetup will be held at ICML Vancouver on July 16th! Please join us if you are at ICML! (onsite only)
• [2025/06] verl with Megatron backend enables large MoE models such as DeepSeek-671B and Qwen3-235B.
• [2025/03] DAPO is the open-sourced SOTA RL algorithm that achieves 50 points on AIME 2024 based on the Qwen2.5-32B pre-trained model, surpassing the previous SOTA achieved by DeepSeek's GRPO (DeepSeek-R1-Zero-Qwen-32B). DAPO's training is fully powered by verl and the reproduction code is available in recipe/dapo now.

Key Features

• FSDP, FSDP2 and Megatron-LM for training.
• vLLM, SGLang and HF Transformers for rollout generation.
• Compatible with Hugging Face Transformers and Modelscope Hub: Qwen-3, Qwen-2.5, Llama3.1, Gemma2, DeepSeek-LLM, etc
• Supervised fine-tuning.
• Reinforcement learning with PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, DrGRPO, KL_Cov & Clip_Cov etc.
  • Support model-based reward and function-based reward (verifiable reward) for math, coding, etc
  • Support vision-language models (VLMs) and multi-modal RL with Qwen2.5-vl, Kimi-VL
  • Multi-turn with tool calling
• LLM alignment recipes such as Self-play preference optimization (SPPO)
• Flash attention 2, sequence packing, sequence parallelism support via DeepSpeed Ulysses, LoRA, Liger-kernel.
• Scales up to 671B models and hundreds of GPUs with expert parallelism
• Multi-gpu LoRA RL support to save memory.
• Experiment tracking with wandb, swanlab, mlflow and tensorboard.
• Hardware Support: Supports NVIDIA, AMD, Ascend

Upcoming Features and Changes

• Q3 Roadmap https://github.com/volcengine/verl/issues/2388
• DeepSeek 671b optimizations with Megatron https://github.com/volcengine/verl/issues/1033
• Multi-turn rollout and tools using optimizations https://github.com/volcengine/verl/issues/1882
• Agent integration
• Async and off-policy architecture https://github.com/volcengine/verl/pull/2231
• List of breaking changes since v0.4 https://github.com/volcengine/verl/discussions/2270

Getting Started

• Installation
• Quickstart
• Programming Guide & Tech Talk (in Chinese)
• PPO in verl
• GRPO in verl

• Prepare Data for Post-Training
• Implement Reward Function for Dataset
• PPO Example Architecture
• Config Explanation

• RL performance on coding, math

• PPO Trainer and Workers
  • PPO Ray Trainer
  • PyTorch FSDP Backend
  • Megatron-LM Backend
• Advanced Usage and Extension
  • Add Models with the FSDP Backend
  • Add Models with the Megatron-LM Backend
  • Multi-turn Rollout Support
  • Search Tool Integration
  • Sandbox Fusion Integration
  • Deployment using Separate GPU Resources
  • Extend to Other RL(HF) algorithms
  • Ray API design tutorial

• When Reasoning Models Break Tokenization: The Hidden Complexity of Multiturn Training
• verl deployment on AWS SageMaker
• verl x SGLang Multi-turn Code Walkthrough
• Optimizing SGLang Memory Usage in verl
• SGLang, verl, OpenBMB and Tsinghua University: Pioneering End-to-End Multi-Turn RLHF
• Reinforcement Learning from Human Feedback on AMD GPUs with verl and ROCm Integration
• veMLP x verl ：玩转强化学习训练
• 使用 verl 进行 GRPO 分布式强化学习训练最佳实践
• HybridFlow verl 原文浅析
• 最高提升 20 倍吞吐量！豆包大模型团队发布全新 RLHF 框架，现已开源！

Performance Tuning Guide

Upgrade to vLLM >= v0.8.2

Use Latest SGLang

Upgrade to FSDP2

actor_rollout_ref.ref.strategy=fsdp2
actor_rollout_ref.actor.strategy=fsdp2
critic.strategy=fsdp2

AMD Support (ROCm Kernel)

Citation and acknowledgement

• HybridFlow: A Flexible and Efficient RLHF Framework
• A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization

Awesome Projects Built with verl

• TinyZero: a reproduction of DeepSeek R1 Zero recipe for reasoning tasks

  

• SkyThought: RL training for Sky-T1-7B by NovaSky AI team.

  

• simpleRL-reason: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild

  

• Easy-R1: Multi-modal RL training framework

  

• OpenManus-RL: LLM Agents RL tuning framework for multiple agent environments.

  

• rllm: async RL training with verl-pipeline

  

• RAGEN: a general-purpose reasoning agent training framework

  

• Search-R1: RL with reasoning and searching (tool-call) interleaved LLMs

  

• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

  

• Skywork-OR1: Skywork open reaonser series

  

• ToRL: Scaling tool-integrated RL

  

• Absolute Zero Reasoner: A no human curated data self-play framework for reasoning

  

• verl-agent: A scalable training framework for long-horizon LLM/VLM agents, along with a new algorithm GiGPO

  

• RL-Factory: An easy and efficient RL post-training framework for Agentic Learning

  

• ReTool: ReTool: reinforcement learning for strategic tool use in LLMs. Code release is in progress...
• verl-tool: An unified and easy-to-extend tool-agent training framework based on verl

  

• PRIME: Process reinforcement through implicit rewards

  

• MemAgent: MemAgent: Reshaping Long-Context LLM with Multi-Conv RL based Memory Agent

  

• POLARIS: A Post-training recipe for scaling RL on Advanced Reasoning models

  

• GUI-R1: GUI-R1: A Generalist R1-style Vision-Language Action Model For GUI Agents

  

• DeepRetrieval: RL Training of Search Agent with Search/Retrieval Outcome

  

• Code-R1: Reproducing R1 for Code with Reliable Rewards

  

• DeepResearcher: Scaling deep research via reinforcement learning in real-world environments

  

• VAGEN: Training VLM agents with multi-turn reinforcement learning

  

• RM-R1: RL training of reasoning reward models

  

• Dr. MAS: Stable end-to-end RL post-training for multi-agent LLM systems

  

• LUFFY: Learning to Reason under Off-Policy Guidance

  

• DeepMath: DeepMath-103K data and series models for math reasoning

  

• PACS: Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR

  

• Entropy Mechanism of RL: The Entropy Mechanism of Reinforcement Learning for Large Language Model Reasoning

  

• LLaSA-TTS-GRPO: TTS fine-tuning with GRPO optimization based on LLASA models

  

• PF-PPO: Policy Filtration for PPO based on the reliability of reward signals for more efficient and robust RLHF.
• RACRO: Build multi-modal reasoning models via decoupling it into query-conditioned captioning and text-only reasoning

  

• Agent Lightning: A flexible and extensible framework that enables seamless agent optimization for any existing agent framework.

  

• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use.

  

• Kimina-Prover-RL: Training pipeline for formal theorem proving, based on a paradigm inspired by DeepSeek-R1.
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization.
• rStar2-Agent: Using reinforcement learning with multi-step tool-calling for math tasks, rStar2-Agent-14B reaches frontier-level math reasoning in just 510 RL training steps

  

• Vision-SR1: Self-Rewarding Vision-Language Model via Reasoning Decomposition

  

• SimpleVLA-RL: SimpleVLA-RL: A Simple yet Effective Vision-Language Action Model for Reinforcement Learning

  

• Table-R1: Table-R1: Inference-Time Scaling for Table Reasoning

  

• Revisual-R1: Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning

  

• ARES: ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping

  

• Meta-Bandit-LLM: Meta-Bandit-LLM: Long-horizon multiturn interactive training for meta-bandit agents

  

• PokeeResearch: PokeeResearch: State-of-the-art 7B DeepResearch Agent that leverages web search and content reading capabilities to answer complex questions using the most up-to-date information available online.

  

• Search Self-play: Pushing the Frontier of Agent Capability without Supervision

  

• OneThinker: All-in-one Reasoning Model for Image and Video

  

• OpenTinker: Democratizing Agentic Reinforcement Learning as a Service

  

• FlowRL: Matching reward distributions via flow balance for diverse exploration and generalizable reasoning

  

• Logic-RL: a reproduction of DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset.

  

• Seed-Coder: RL training of Seed-Coder boosts performance on competitive programming

  

• all-hands/openhands-lm-32b-v0.1: A strong, open coding agent model, trained with multi-turn fine-tuning
• s3 Efficient Yet Effective Search Agent Training via RL

  

• Rec-R1: Bridging Generative Large Language Models and Recommendation Systems via Reinforcement Learning
• Explore RL Data Scaling: Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
• FIRE: Flaming-hot initiation with regular execution sampling for large language models
• DQO: Enhancing multi-Step reasoning abilities of language models through direct Q-function optimization
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• cognition-engineering: Test time scaling drives cognition engineering.

  

• Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning.

  

• AdaRFT: Efficient Reinforcement Finetuning via Adaptive Curriculum Learning

  

• critic-rl: LLM critics for code generation

  

• self-rewarding-reasoning-LLM: self-rewarding and correction with generative reward models

  

• DeepEnlighten: Reproduce R1 with social reasoning tasks and analyze key findings

  

• MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse

  

• PURE: Credit assignment is the key to successful reinforcement fine-tuning using process reward model

  

• cognitive-behaviors: Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

  

• deepscaler: iterative context scaling with GRPO

  

• DAPO: the fully open source SOTA RL algorithm that beats DeepSeek-R1-zero-32B

  

• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

  

• SPEAR: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning (ICLR 2026)

  

• RuleReasoner: RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling (ICLR 2026)

  

• MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning

  

Contribution Guide

About ByteDance Seed Team