They don’t just see an image, they can integrate visual information directly into the reasoning chain.
Key insights:
The capability of DeepEyes to think with images is learned via end-to-end reinforcement learning. It is directly guided by outcome reward signals, requires no cold-start or supervised fine-tuning, and does not rely on specialized external model.
Although there is no direct supervision applied for intermediate steps, both the grounding IoU and tool-calling accuracy was increased during the RL training stage.
The end-to-end RL training yields significant performance gain on high resolution benchmarks, and shows strong generalization for visual grounding, hallucination mitigation, and math problem solving tasks.
We observed an emergence of thinking pattern during RL training process, such as visual search for small objects, visual comparisons across different regions, using image_zoom_in_tools for answer verification, etc.
Quick Start
Environment Setup
bash
1# Follow the VeRL official installation procedure2pip install -e .34# Additional dependencies required by DeepEyes5bash scripts/install_deepeyes.sh
We recommend using no less than 32 GPUs (4 nodes x 8 GPUs) for 7B training, and no less than 64 GPUs (8 nodes x 8 GPUs) for 32B training. For each node, we recommend using no less than 1200GB CPU RAM, as the high resolution images in V* and ArxivQA datasets can consume large amount of memory.
Step 1: Start a vllm serving of Qwen-2.5-72B-Instruct for llm-as-a-judge verification.
Step 2: Build a ray cluster for all of the training nodes. Prepare data before starting training. Our training dataset can be downloaded from huggingface.
Step 3: Use one of the following scripts to start training.
bash
1# your wandb access key here...2wandb login
34# the IP and port for your Qwen-2.5-72B-Instruct vllm serving5exportLLM_AS_A_JUDGE_BASE="http://your.vllm.machine.ip:18901/v1"67# umber of training nodes8exportWORLD_SIZE=8910# config for 7B11bash examples/agent/final_merged_v1v8_thinklite.sh
1213# config for 32B14bash examples/agent/final_merged_v1v8_thinklite_32b.sh
The training scripts use both wandb and RL Logging Board (great work) to visualize the training dynamics.
Programming Guide
General Introduction for Codes
General Introduction
The code in this repository is a general agentic RL training framework based on VeRL. Apart from DeepEyes, it is possible to perform any form of general agentic RL (multi-turn RL) training using our code implementation.
The code is designed to fulfill the following needs:
High efficient Agent RL training: Agent rollout is asynchronous among all data parallel groups.
Allowing dynamic multi-modal input in agent observations: This is the key for the RL training of "thinking with images" ability.
Allowing hybrid training for agent data with different tools and non-agentic data: Tool usage is not hard-coded in rollout loop, instead, each sample can specify its own tool usage constraint via env_name field.
Support for algorithm: PPO, GRPO, and reinforce++ are supported. We modified the advantage estimation, the policy loss masks, as well as the mrope for Qwen-VL models, to make it compatible with the interleaved structure of agentic multi-turn RL training.
Compatible for latest VeRL updates: agentic RL training is implemented as a plugin for VeRL, making it easy to merge with the latest VeRL updates. Once you turn off the plugin switch, the functionality will be no different to the original version of VeRL.
Training on Customized Datasets
Use your own data
Add an additional field env_name to your data parquet files. The env_name of each sample should specify the which tool is allowed to use when performing agent rollout. For non-agent training data, leave the env_name to None or empty string.
For DeepEyes style training, for example, env_name should be specified as visual_toolbox_v2.
The rest part is no different to the original VeRL dataset format. Refer to VeRL official documentation for details.
Training with Customized Tools
Implement your own tools
Implement your tool function in a new class that inherents ToolBase class in verl/workers/agent/tool_envs.py as its base class.
The subclass MUST include name variable, whose value corresponds to the env_name field in training data parquet files.
Implement the execute and reset functions. Here is an simple example:
Example code:
python
1classCustomTool(ToolBase):2 name ="custom_tool_v0"34def__init__(self, _name, _desc, _params,**kwargs):5super().__init__(name=self.name)67defexecute(self, action_string:str,**kwargs)->tuple:8"""
9 Execute the tool functionality based on the LLM generated text.
10 This function is called EACH TIME after vllm.generate
1112 Args:
13 action_string: The string generated by LLM via vllm.generate.
1415 Returns:
16 observation: The structured observation with the processed image.
17 reward: setting a non-zero value if you want to assign a reward to the LAST GENERATED TOKEN in the intermediate steps.
18 done: Whether the episode is terminated.
19 info: Additional info.
20 """21pass2223defreset(self, raw_prompt, multi_modal_data, origin_multi_modal_data,**kwargs):24"""
25 This function is called ONLY ONCE when initializing the tools
2627 Args:
28 raw_prompt: setting config param `data.return_raw_chat=True` to get raw prompt input.
29 multi_modal_data: refer to vllm documentation for details https://docs.vllm.ai/en/stable/features/multimodal_inputs.html
30 origin_multi_modal_data: VLM vision processor can modify the original images, typically by resizing, when they are too small or too large, use this param if you want to get access to the unmodified vision input.
31 """32pass
