Orca: The World is in Your Mind
Orca Team, Beijing Academy of Artificial Intelligence
A general world foundation model centered on Next-State-Prediction.
💬 If you have any questions, feel free to contact us via WeChat.
🔥 Overview
Orca is an initial instantiation of a general world foundation model. It learns a unified world latent space from multimodal world signals and exposes the learned latent through multimodal readout interfaces.
Rather than optimizing isolated next-token, next-frame, or next-action prediction objectives, Orca is centered on Next-State-Prediction: a unified state-transition modeling route toward understanding, predicting, and acting upon the world. In this version, Orca focuses on two fundamental input signals: visual signals for dense observations of world evolution, and language signals for event descriptions, task intentions, causal explanations, and semantic constraints.
- Unconscious learning: dense natural transitions from continuous videos.
- Conscious learning: sparse meaningful transitions under language-described events and VQA supervision.
- Frozen-backbone readouts: lightweight decoders for text, images, and actions.
- Scaling analysis: stronger world modeling, stronger downstream readouts.
🗞️ News
- 2026-07-14: 🚀 Orca-4B checkpoint was released on HuggingFace.
2026-06-29: 🎉 Orca Technical Report was released.
📆 Todo
⭐️ Architecture
Orca follows an Encoder-Decoder architecture. Given multimodal world signals, the Encoder learns a world latent through unconscious and conscious learning. After pre-training, the Encoder is frozen, and only lightweight modality-specific decoders are trained to read out the latent into downstream modalities.
Figure 1. Orca learns a unified world latent through unconscious and conscious learning.
Figure 2. Lightweight readouts adapt frozen world latents to language, vision, and action.
Orca models world-state transitions under both implicit dynamics and explicit conditions. Implicit dynamics capture latent or unobserved factors such as physical laws, object properties, scene dynamics, and environmental forces, while explicit conditions describe observed signals such as human instructions, event descriptions, task intentions, or causal premises.
📚 Data
For pre-training, Orca constructs a large-scale world-learning inventory from visual signals and language signals. The data mixture includes video data for observation-only state transitions, event data for event-conditioned state transitions, and VQA data for response generation.
- 125K hours of video data covering egocentric interaction, exocentric manipulation, robot execution, and natural dynamics.
- 160M event annotations with fine- and coarse-grained captions for event-level transition learning.
- General VQA data for aligning world latents with language understanding and response generation.
Figure 3. Orca data pipeline from multimodal world signals to world latent learning.
🔍 Evaluation
Orca is evaluated through three representative downstream readouts: text generation, image prediction, and action generation.
Text Generation
Text generation evaluates understanding on TemporalBench, MVBench, SWITCH, and 3DSRBench.
| Model | Size (B) | MVBench ↑ | TemporalBench ↑ | 3DSRBench ↑ | SWITCH ↑ | Avg. ↑ |
|---|
| Emu3 | 8 | 35.2 | 9.5 | 39.1 | 38.0 | 30.4 |
| Emu3.5 | 34 | 39.5 | 9.5 | 31.3 | 38.9 | 29.8 |
| MiniCPM-V-4.6 | 2 | 41.4 | 21.2 | 47.7 | 41.2 | 37.9 |
| Qwen3.5 | 4 | 67.1 | 25.2 | 48.1 | 46.5 | 46.7 |
| Orca | 0.8 | 53.6 | 22.6 | 43.4 | 43.7 | 40.8 |
| Orca | 4 | 65.3 | 34.2 | 52.1 | 55.6 | 51.8 |
Image Prediction
Image prediction evaluates future-state prediction on PRICE-V0.1 real-world interactions.
The PRICE evaluation toolkit is available in
evaluation/image_gen/PRICE/, and the benchmark data is hosted at
BAAI/PRICE.
| Model | Size (B) | Gemini 3.1 Pro ↑ | GPT 5.4 ↑ | Doubao-Seed-2.0 ↑ | Gemma 4-31B ↑ | Avg. ↑ |
|---|
| OmniGen2 | 3+4 | 24.6 | 46.8 | 41.4 | 45.5 | 39.6±10.2 |
| FLUX.1-Kontext | 12 | 21.6 | 46.9 | 42.7 | 52.5 | 40.9±13.5 |
| FLUX.2 [klein] | 4+4 | 29.7 | 64.6 | 60.0 | 70.2 | 56.1±18.1 |
| Orca | 0.8+2 | 17.0 | 48.5 | 46.0 | 26.5 | 34.5±15.3 |
| Orca | 4+2 | 44.0 | 67.9 | 61.0 | 66.3 | 59.8±10.9 |
Action Generation
Action generation evaluates five real-robot manipulation tasks under environment and object OOD settings.
| Model | Rule ↑ | M25 ↑ | M50 ↑ | SR ↑ | MaxP-F ↑ | FNS ↑ | RBS ↑ | SQS ↑ |
|---|
| V-JEPA 2.1 | 17.0 | 27 | 7 | 0 | 17.4 | 10.1 | 20.5 | 0.0 |
| Qwen3.5 | 10.5 | 18 | 5 | 0 | 13.1 | 7.6 | 11.9 | 0.0 |
| pi0.5 | 29.4 | 54 | 14 | 5 | 26.5 | 15.3 | 26.7 | 3.0 |
| Orca | 32.4 | 55 | 14 | 6 | 27.9 | 15.1 | 30.3 | 2.9 |
M25/M50: trajectories reaching 25%/50% milestones; SR: success rate; MaxP-F: max process in failed trials; FNS: failure near-success score; RBS: robustness score; SQS: success quality score.
Scaling Behavior
Figure 4. Downstream readout performance improves as Orca pre-training scales.
Experiments indicate that stronger world latents from pre-training lead to stronger downstream readouts. As pre-training scales up, Orca improves across text, image, and action readouts while keeping the backbone frozen during readout post-training.
🛠️ Usage
The current release provides the
🤗 BAAI/Orca-4B checkpoint and evaluation code for image and text generation.
Clone the repository and install the shared dataset downloader:
1git clone https://github.com/orca-wm/Orca.git
2cd Orca/evaluation
3python -m pip install -r requirements-data.txt
Download evaluation datasets into evaluation/data/:
1python download_datasets.py price
2python download_datasets.py switch
3python download_datasets.py mvbench
4python download_datasets.py temporalbench
5python download_datasets.py 3dsrbench
See the task-specific instructions for model setup and evaluation:
📑 Citation
If you find Orca useful for your research, please consider citing our technical report.
1@article{orca2026,
2 title={Orca: The World is in Your Mind},
3 author={Yihao Wang and Yuheng Ji and Mingyu Cao and Yanqing Shen and Runze Xiao and Huaihai Lyu and Senwei Xie and Euan Liu and Klara Tian and Tianfeng Long and Yichi Zhang and Zhengliang Cai and Ruike Chen and Jifan Zhao and Ruochuan Shi and Zihan Tang and Jing Lyu and Wenxing Tan and Ningbo Zhang and Yangtao Hu and Yuming Gao and Xiansheng Chen and Junkai Zhao and Congsheng Xu and Boan Zhu and Ziqi Wang and Yupu Feng and Qiongqiong Zhang and Yingli Zhao and Yulong Ao and Shaoxuan Xie and You Liu and Guocai Yao and Leiduo Zhang and Xiaodan Liu and Yunyan Zhang and Yance Jiao and Xinyan Yang and Jiaxing Wei and Xu Liu and Tengfei Pan and Shaokai Nie and Chunlei Men and Sen Cui and Xiaojie Jin and Hongyang Li and Jianlan Luo and Yao Mu and Yunchao Wei and Jun Yan and Hang Zhao and Xiaolong Zheng and Jiaming Li and Yonghua Lin and Tiejun Huang and Zhongyuan Wang and Pengwei Wang},
4 journal={arXiv preprint arXiv:2606.30534},
5 year={2026}
6}
