verl: Volcano Engine Reinforcement Learning for LLMs

• Easy extension of diverse RL algorithms: The hybrid-controller programming model enables flexible representation and efficient execution of complex post-training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code.
• Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc
• Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
• Ready integration with popular HuggingFace models

• State-of-the-art throughput: SOTA LLM training and inference engine integrations and SOTA RL throughput.
• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

News

• [2025/08] verl is presented in the PyTorch Expert Exchange Webinar. Slides available.
• [2025/07] The ReTool recipe is fully open sourced. Blog
• [2025/07] The first verl meetup will be held at ICML Vancouver on July 16th! Please join us if you are at ICML! (onsite only)
• [2025/06] verl with Megatron backend enables large MoE models such as DeepSeek-671B and Qwen3-235B.
• [2025/03] DAPO is the open-sourced SOTA RL algorithm that achieves 50 points on AIME 2024 based on the Qwen2.5-32B pre-trained model, surpassing the previous SOTA achieved by DeepSeek's GRPO (DeepSeek-R1-Zero-Qwen-32B). DAPO's training is fully powered by verl and the reproduction code is available in recipe/dapo now.

Key Features

• FSDP, FSDP2 and Megatron-LM for training.
• vLLM, SGLang and HF Transformers for rollout generation.
• Compatible with Hugging Face Transformers and Modelscope Hub: Qwen-3, Qwen-2.5, Llama3.1, Gemma2, DeepSeek-LLM, etc
• Supervised fine-tuning.
• Reinforcement learning with PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, DrGRPO, KL_Cov & Clip_Cov etc.
  • Support model-based reward and function-based reward (verifiable reward) for math, coding, etc
  • Support vision-language models (VLMs) and multi-modal RL with Qwen2.5-vl, Kimi-VL
  • Multi-turn with tool calling
• LLM alignment recipes such as Self-play preference optimization (SPPO)
• Flash attention 2, sequence packing, sequence parallelism support via DeepSpeed Ulysses, LoRA, Liger-kernel.
• Scales up to 671B models and hundreds of GPUs with expert parallelism
• Multi-gpu LoRA RL support to save memory.
• Experiment tracking with wandb, swanlab, mlflow and tensorboard.

Upcoming Features and Changes

• Q3 Roadmap https://github.com/volcengine/verl/issues/2388
• DeepSeek 671b optimizations with Megatron https://github.com/volcengine/verl/issues/1033
• Multi-turn rollout and tools using optimizations https://github.com/volcengine/verl/issues/1882
• Agent integration
• Async and off-policy architecture https://github.com/volcengine/verl/pull/2231
• List of breaking changes since v0.4 https://github.com/volcengine/verl/discussions/2270

Getting Started

• Installation
• Quickstart
• Programming Guide & Tech Talk (in Chinese)
• PPO in verl
• GRPO in verl

• Prepare Data for Post-Training
• Implement Reward Function for Dataset
• PPO Example Architecture
• Config Explanation

• RL performance on coding, math

• PPO Trainer and Workers
  • PPO Ray Trainer
  • PyTorch FSDP Backend
  • Megatron-LM Backend
• Advanced Usage and Extension
  • Add Models with the FSDP Backend
  • Add Models with the Megatron-LM Backend
  • Multi-turn Rollout Support
  • Search Tool Integration
  • Sandbox Fusion Integration
  • Deployment using Separate GPU Resources
  • Extend to Other RL(HF) algorithms
  • Ray API design tutorial

• When Reasoning Models Break Tokenization: The Hidden Complexity of Multiturn Training
• verl deployment on AWS SageMaker
• verl x SGLang Multi-turn Code Walkthrough
• Optimizing SGLang Memory Usage in verl
• SGLang, verl, OpenBMB and Tsinghua University: Pioneering End-to-End Multi-Turn RLHF
• Reinforcement Learning from Human Feedback on AMD GPUs with verl and ROCm Integration
• veMLP x verl ：玩转强化学习训练
• 使用 verl 进行 GRPO 分布式强化学习训练最佳实践
• HybridFlow verl 原文浅析
• 最高提升 20 倍吞吐量！豆包大模型团队发布全新 RLHF 框架，现已开源！

Performance Tuning Guide

Upgrade to vLLM >= v0.8.2

Use Latest SGLang

Upgrade to FSDP2

actor_rollout_ref.ref.strategy=fsdp2
actor_rollout_ref.actor.strategy=fsdp2
critic.strategy=fsdp2 
reward_model.strategy=fsdp2 

AMD Support (ROCm Kernel)

Citation and acknowledgement

• HybridFlow: A Flexible and Efficient RLHF Framework
• A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization

Awesome work using verl

• TinyZero: a reproduction of DeepSeek R1 Zero recipe for reasoning tasks

  

• SkyThought: RL training for Sky-T1-7B by NovaSky AI team.

  

• simpleRL-reason: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild

  

• Easy-R1: Multi-modal RL training framework

  

• OpenManus-RL: LLM Agents RL tunning framework for multiple agent environments.

  

• rllm: async RL training with verl-pipeline

  

• RAGEN: a general-purpose reasoning agent training framework

  

• Search-R1: RL with reasoning and searching (tool-call) interleaved LLMs

  

• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

  

• Skywork-OR1: Skywork open reaonser series

  

• ToRL: Scaling tool-integrated RL

  

• Absolute Zero Reasoner: A no human curated data self-play framework for reasoning

  

• verl-agent: A scalable training framework for long-horizon LLM/VLM agents, along with a new algorithm GiGPO

  

• RL-Factory: An easy and efficient RL post-training framework for Agentic Learning

  

• ReTool: ReTool: reinforcement learning for strategic tool use in LLMs. Code release is in progress...
• verl-tool: An unified and easy-to-extend tool-agent training framework based on verl

  

• PRIME: Process reinforcement through implicit rewards

  

• MemAgent: MemAgent: Reshaping Long-Context LLM with Multi-Conv RL based Memory Agent

  

• POLARIS: A Post-training recipe for scaling RL on Advanced Reasoning models

  

• GUI-R1: GUI-R1: A Generalist R1-style Vision-Language Action Model For GUI Agents

  

• DeepRetrieval: RL Training of Search Agent with Search/Retrieval Outcome

  

• Code-R1: Reproducing R1 for Code with Reliable Rewards

  

• DeepResearcher: Scaling deep research via reinforcement learning in real-world environments

  

• VAGEN: Training VLM agents with multi-turn reinforcement learning

  

• RM-R1: RL training of reasoning reward models

  

• LUFFY: Learning to Reason under Off-Policy Guidance

  

• DeepMath: DeepMath-103K data and series models for math reasoning

  

• PACS: Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR

  

• Entropy Mechanism of RL: The Entropy Mechanism of Reinforcement Learning for Large Language Model Reasoning

  

• LLaSA-TTS-GRPO: TTS fine-tuning with GRPO optimization based on LLASA models

  

• PF-PPO: Policy Filtration for PPO based on the reliability of reward signals for more efficient and robust RLHF.
• RACRO: Build multi-modal reasoning models via decoupling it into query-conditioned captioning and text-only reasoning

  

• Agent Lightning: A flexible and extensible framework that enables seamless agent optimization for any existing agent framework.

  

• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use.

  

• Kimina-Prover-RL: Training pipeline for formal theorem proving, based on a paradigm inspired by DeepSeek-R1.
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization.
• rStar2-Agent: Using reinforcement learning with multi-step tool-calling for math tasks, rStar2-Agent-14B reaches frontier-level math reasoning in just 510 RL training steps

  

• Vision-SR1: Self-Rewarding Vision-Language Model via Reasoning Decomposition

  

• SimpleVLA-RL: SimpleVLA-RL: A Simple yet Effective Vision-Language Action Model for Reinforcement Learning

  

• Table-R1: Table-R1: Inference-Time Scaling for Table Reasoning

  

Contribution Guide

About ByteDance Seed Team