Pelican-VLA 0.5: Attending Before Acting Benefits Generalization
Zero-shot action-pathway attention. Before any task-specific fine-tuning, Pelican-VLA 0.5
concentrates attention on the instruction-relevant object and its contact region, while other
open-source VLA models attend diffusely over the robot arm, background, and irrelevant objects.
Model Summary
Pelican-VLA 0.5 is a unified Vision-Language-Action (VLA) model that integrates
vision-language understanding, future-frame generation, and action prediction within a single
shared
Qwen3-VL 4B backbone. It is developed by
the
WFM System Group at the Beijing Innovation Center of Humanoid Robotics (X-Humanoid).
This repository hosts the cross-embodiment pre-trained checkpoint. It is trained on ~2,400
hours of heterogeneous manipulation data and is the model used for the zero-shot attention and
zero-shot RoboTwin demonstrations in the paper. For the RoboTwin-tuned policy, see
X-Humanoid/Pelican-VLA05-Robotwin.
The central design is a compact set of learnable bottleneck tokens (BotTokens) inserted
between perception and action. Rather than letting the action pathway attend directly to dense
visual tokens, the BotTokens form a fixed-capacity interface that compresses and routes only
manipulation-relevant information into action generation. This information bottleneck induces
attention-level generalization: without object annotations, segmentation masks, attention
supervision, or task-specific fine-tuning, the action pathway already focuses on the
instruction-relevant object and its contact region — and this behavior transfers to unseen scenes,
unseen objects, and unseen robot embodiments.
Architecture
A single Qwen3-VL 4B Transformer processes one token sequence with four consecutive segments:
- Vision-Language Prefix — multi-view images and the language instruction (scene semantics).
- Temporal / World Middle Tokens — visual history encoded by a frozen NVIDIA Cosmos
image tokenizer; supplies pixel-level dynamics and carries future-frame prediction.
- BotTokens — 32 learnable bottleneck tokens that compress and organize
manipulation-relevant information.
- Action Suffix — robot proprioceptive state plus noisy action tokens used by the
flow-matching action head to produce action trajectories.
Crucially, the action suffix cannot bypass the BotTokens to read dense visual tokens
directly. Three mechanisms make the bottleneck hold: a curriculum bottleneck attention mask,
an orthogonality regularizer across BotTokens, and BotToken-gated future-frame generation.
| Property | Value |
|---|
| Backbone | Qwen3-VL 4B (Instruct) |
| Total parameters | ~5B |
| Action head | Flow matching (10 denoising steps) |
| Bottleneck tokens | 32 |
| Action chunk size | 50 |
| Max state / action dim | 32 (shorter vectors are padded) |
| Image resolution | 224 × 224 |
| Default dtype | bfloat16 |
| Visual history / future-frame branch | frozen NVIDIA Cosmos-Tokenizer-CI8x8 |
Intended Use
- Research on generalizable vision-language-action policies and manipulation-centric attention.
- Zero-shot / few-shot deployment and analysis on robot manipulation tasks.
- A starting point for downstream fine-tuning on your own robot data.
This is an
intermediate research model, not a production controller. See
Limitations.
Requirements
To run this checkpoint you also need:
- Qwen3-VL-4B-Instruct weights (backbone initialization) — Qwen, Apache-2.0.
- NVIDIA Cosmos-Tokenizer-CI8x8 (frozen visual-history / future-frame branch) — nvidia/Cosmos-0.1-Tokenizer-CI8x8, NVIDIA Open Model License.
- The inference release from Open-X-Humanoid/Pelican-VLA05 (
pelican_vla0.5_infer/).
How to Use
Download the checkpoint and install the inference package:
1huggingface-cli download X-Humanoid/Pelican-VLA05 --local-dir ./pretrained_model
2
3git clone https://github.com/Open-X-Humanoid/Pelican-VLA05.git
4cd Pelican-VLA05/pelican_vla0.5_infer
5pip install -r requirements.txt # or: pip install -e .
6
7export QWEN3_VL_PATH=/path/to/Qwen3-VL-4B-Instruct
8# Optional (for offline Cosmos tokenizer):
9# export COSMOS_TOKENIZER_PATH=/path/to/Cosmos-0.1-Tokenizer-CI8x8
For the inference API and a runnable example, see the inference release
README.
Checkpoint layout
from_pretrained accepts a directory containing either a single model.safetensors or the split
files backbone.safetensors + heads.safetensors (+ optional action_head.safetensors /
gen_head.safetensors), together with config.json and stats.json. Loading is strict by
default, so mismatched or stale parameter names fail immediately.
Attention visualization
The release ships a
visualization tool
that loads this checkpoint and dumps action→image attention as
.npz per camera, so you can
verify whether attention focuses on the task-relevant object, the gripper, and the actionable
region. It runs on the shipped datasets and on
your own data, provided it is packaged as a
LeRobot 3.0 dataset.
Performance
Zero-shot on RoboTwin 2.0 (RoboTwin data was not used during pre-training). Deployed directly
on held-out tasks with unseen objects, unseen scene layouts, and a new robot embodiment, the
pre-trained policy reaches toward the correct instruction-relevant object and produces coherent,
goal-directed motions.
After fine-tuning on RoboTwin 2.0 (see the
RoboTwin checkpoint),
Pelican-VLA 0.5 achieves the best average success rate among open-source VLA baselines, with only a
0.4-point gap between clean and randomized settings:
| Method | Clean | Randomized | Average |
|---|
| π₀ | 80.0 | 79.5 | 79.8 |
| π₀.₅ | 86.8 | 87.0 | 86.9 |
| X-VLA | 72.9 | 72.8 | 72.9 |
| StarVLA-OFT | 88.2 | 88.3 | 88.3 |
| ABot-M0 | 86.1 | 85.1 | 85.6 |
| LingBot-VLA | 88.6 | 86.7 | 87.7 |
| Qwen-VLA | 86.1 | 87.2 | 86.7 |
| JoyAI-RA | 90.5 | 89.3 | 89.9 |
| Hy-VLA | 90.9 | 90.1 | 90.5 |
| Pelican-VLA 0.5 | 91.4 | 91.0 | 91.2 |
Limitations
Pelican-VLA 0.5 is an intermediate stage toward practical zero-shot manipulation. A
representation-to-action gap remains: the model begins to identify what to act upon, but
reliable execution (stable grasping, contact timing, precise placement) is not solved. This is
attributed mainly to:
- Limited action experience — the current checkpoint sees only ~2,400 hours of heterogeneous
data with some language-action noise.
- Joint-position actions — more embodiment-specific than end-effector pose representations,
which limits cross-embodiment transfer.
A stronger version trained on approximately 7,000 hours, with improved action parameterization
and stricter data curation, is planned to supersede this checkpoint.
License
Released under the
Apache-2.0 License. This license covers the source code and this checkpoint
distribution. Component model assets keep their own licenses:
Qwen3-VL weights are Apache-2.0;
NVIDIA Cosmos tokenizer weights use the
NVIDIA Open Model License. Third-party components
(LeRobot, openpi, Hugging Face Transformers, NVIDIA Cosmos) are attributed in the release's
NOTICE
and
THIRD_PARTY_LICENSES.md.
Citation
1@article{ding2026pelican,
2 title = {Pelican-VLA 0.5: Attending Before Acting Benefits Generalization},
3 author = {Ding, Zeyuan and Liu, Wenhai and Xu, Yang and Hu, Jiayu and Chen, Yinda and
4 Zhang, Yi and Dai, Yong and Tang, Jian and Ju, Xiaozhu},
5 journal = {arXiv preprint arXiv:2607.06655},
6 year = {2026},
7 url = {https://arxiv.org/abs/2607.06655}
8}
Acknowledgement
We thank the developers of
LeRobot,
openpi,
Qwen3-VL, and
NVIDIA Cosmos for their contributions to the open-source
community.
