Interleaved Vision-Text-Action Pretraining for General Robot Control
We introduce EO-1 model, an open-source unified embodied foundation model comprising 3B parameters, trained on the carefully curated interleaved embodied dataset EO-Data1.5M, Web Multimodal Data, and Robot Control Data (AgiBotWorld, Open X-Embodiment, RoboMIND, SO100-Community, etc.). The EO-1 model adopt a single unified decoder-only transformer that integrates discrete auto-regressive decoding with continuous flow matching denoising for multimodal embodied reasoning and robot control, enabling seamless perception, planning, reasoning, and acting in single model. This work highlights the following features:
- ⚡ Unified Architecture: A single decoder-only transformer integrating text, image, video, and actions.
- 📚 EO-1.5M Dataset: 1.5M high-quality interleaved samples (Physical, Reasoning, Spatial, Control).
- 🌀 Interleaved Pretraining: Seamless synergy between language and action with autoregressive + flow matching.
- 🤖 Reasoning-Enhanced Generalization: Superior generalization capabilities with multimodal embodied reasoning and real robot control.
0. Model Architecture
EO-1 model is a Vision-Language-Action (VLA) model that adopts a single unified decoder-only transformer, equipping with discrete language-modeling head for multimodal embodied reasoning and continuous flow-matching head for robot action generation. The language instruction, image observations, robot state, and noisy action are encoded into an interleaved token sequence of tokens to be processed by the shared transformer backbone, whose weights are initialized from Qwen2.5-VL. The model is trained on interleaved vision-text-action data with a combination of flow-matching objective and next-token-prediction objective and capable of seamless embodied reasoning and acting.
Input:
Input Type:
- Vision: Image Frames, Video
- State: Robot Proprioception
- Language Instruction: Text, Pointing, Bounding Box, etc.
- Input Format:
- Vision: Variable number of uint8 image frames or long video sequence
- State: Floating Point
- Language Instruction: String
Output:
Output Type(s): Actions, Language
Output Format: Continuous-value vectors, Discrete Text
1. Inference with pre-trained model
EO-1 is built entirely on 🤗 HuggingFace Transformers and Lerobot, making deployment straightforward and accessible. If your environment supports transformers and lerobot, you can load the model and run inference directly with just a few lines of code (requires ~6.5GB GPU memory). EO-1 unifies high-level embodied reasoning with low-level robot control, producing either natural language outputs or actionable robot commands.
1from transformers import AutoModel, AutoProcessor
2# load the model and processor
3processor = AutoProcessor.from_pretrained("IPEC-COMMUNITY/eo1-qwen25_vl-bridge", trust_remote_code=True)
4model = AutoModel.from_pretrained(
5 "IPEC-COMMUNITY/eo1-qwen25_vl-bridge",
6 trust_remote_code=True,
7 torch_dtype=torch.bfloat16
8).eval().cuda()
9
10# prepare the model input
11batch = {
12 "observation.images.image_0": [img], # PIL.Image
13 "observation.state": [state],
14 "task": ["You are a helpful physical agent equipped with both reasoning and robotic control. \
15 You see the Tic-Tac-Toe board, think strategically, act logically, and block threats."]
16}
17
18ov_output = processor.select_action(
19 new_model,
20 batch,
21)
22
23print(ov_output)
2. Benchmark
WidowX Benchmark
| Model | Put Spoon on Towel | Put Carrot on Plate | Stack Blocks | Put Eggplant in Basket | Overall |
|---|
| $\pi_0$ | 0.838 | 0.525 | 0.525 | 0.879 | 0.692 |
| $\pi_0$-fast | 0.291 | 0.219 | 0.108 | 0.666 | 0.321 |
| EO-1 | 0.636 | 0.545 | 0.818 | 0.909 | 0.727 |
📚 3. Citation
If you find this project useful, please consider citing:
1@article{eo1,
2 title={EO-1: Interleaved Vision-Text-Action Pretraining for General Robot Control},
3 author={Delin Qu and Haoming Song and Qizhi Chen and Zhaoqing Chen and Xianqiang Gao and Xinyi Ye and Qi Lv and Modi Shi and Guanghui Ren and Cheng Ruan and Maoqing Yao and Haoran Yang and Jiacheng Bao and Bin Zhao and Dong Wang},
4 journal={arXiv preprint},
5 year={2025},
6 url={https://arxiv.org/abs/2508.21112}
7}
