GEAR-VLA is a vision-language-action framework designed to connect embodied perception, geometric reasoning, and robot control. This repository releases GEAR-VLA-9B, the vision-language model (VLM) checkpoint obtained after the Embodied Pretraining stage.
GEAR-VLA introduces three complementary designs:
🎯 Coarse-to-Fine Action Learning. The framework combines latent action representations from manipulation videos with discretized robot actions. This design first captures high-level interaction dynamics and subsequently learns precise embodiment-specific control.
🧭 Semantic-Aligned 3D Geometry Integration. A trainable VGGT branch extracts multi-view 3D geometry while preserving the semantic knowledge of the frozen 2D visual pathway. The two feature streams are aligned and fused before entering the language model.
🤖 Embodiment Canonicalization. Embodiment-aware state inputs and a unified relative end-effector action space confine robot-specific variation to the low-level interface. This design supports transfer across robot platforms without introducing robot-specific semantic prompts into the shared representation.
The released checkpoint provides the VLM foundation used by the complete GEAR-VLA system. It does not include the downstream DiT action expert, robot-state projector, or a complete continuous-control policy.
📚 Embodied Pretraining
Embodied Pretraining uses approximately 4.189 million general vision-language and embodied-perception samples, together with 23,027.6 hours of robot and egocentric manipulation data. All tasks are formulated as autoregressive token prediction within a unified training objective.
For robot trajectories with action annotations, the model predicts both FAST-style action tokens and latent action identifiers. For videos without action annotations, it learns from latent action identifiers. Pretraining was conducted for 350,000 iterations on 240 NVIDIA H200 GPUs, with a batch size of eight per GPU.
General vision-language and embodied perception data
Robotic and human manipulation data
🏗️ Model Architecture
GEAR-VLA-9B uses a dual-branch visual architecture. The original 2D visual encoder remains frozen to preserve pretrained semantic representations, while VGGT serves as a trainable 3D spatial encoder for multi-view geometric reasoning. The current implementation uses the VGGT Aggregator and does not use its camera, depth, point, or tracking heads.
The 2D features H2D and 3D features H3D are concatenated along the feature dimension and passed through an expanded MLP projector. The projector weights associated with the 2D pathway are inherited from the pretrained VLM, whereas the newly introduced 3D pathway is zero-initialized. This initialization preserves the original visual-language behavior at the start of training while allowing geometry-aware features to be learned progressively.
GEAR-VLA-9B architecture
📊 VLM Capability Evaluation
We evaluated the current GEAR-VLA-9B Embodied Pretraining checkpoint under the same evaluation protocols and obtained consistent results.
The evaluation covers EgoPlan2, Where2Place, RefSpatial-bench, ShareRobot-affordance, ShareRobot-trajectory, BLINK, EmbSpatial, ERQA, SAT, and CVBench. These benchmarks primarily assess embodied reasoning, spatial understanding, grounding, affordance prediction, and trajectory reasoning.
GEAR-VLA-9B VLM capability overview
GEAR-VLA-9B benchmark performance
GEAR-VLA-9B detailed benchmark results
These scores characterize the embodied capabilities of the VLM checkpoint. They should not be interpreted as robot-task success rates for the complete continuous-control policy.
🏆 Complete GEAR-VLA System Results
The complete GEAR-VLA system, which includes downstream action-policy components that are not part of this release, achieves the following results reported in the paper:
Evaluation setting
Result
LIBERO
98.7%
LIBERO-Plus
88.7%
RoboTwin, clean
91.1%
RoboTwin, randomized
89.9%
AgileX
85.9%
LDT-01
81.0%
Mask-guided universal object grasping pipeline
The same VLM is used for target grounding and mask-guided reasoning. Given the first-frame target mask and current multi-view observation, the DiT-based action expert predicts the robot action for manipulating the specified object.
Across 6,360 trials involving 212 previously unseen objects, the complete system achieves a grasping success rate of 90.1%.
The real-world universal object grasping data view
The training split contains 35 target objects, with 100 demonstrations collected for each object. During data collection,each scene contains the target object and 4–20 background objects sampled from the training-object pool, including objects with similar object categories. The test split contains 212 target objects that do not overlap with the training objects, covering the same object categories to evaluate generalization over object geometry rather than memorization of specific object instances.
📦 Released Checkpoint
This repository contains the VLM parameters produced by the Embodied Pretraining stage. The checkpoint is intended for embodied visual-language understanding, geometric reasoning, and initialization of downstream GEAR-VLA training.
The release does not contain the DiT action expert, robot-state projector, embodiment-specific action heads, or downstream continuous-control policy. Reproducing the complete-system robot results therefore requires the remaining policy components and downstream training procedure described in the paper.
⚠️ Limitations
The VGGT branch and multi-stage training procedure increase computational and memory requirements. The current geometry integration also relies on multi-view inputs. Single-camera settings, mobile manipulation, and scenes with severe occlusion may require additional adaptation.
🚀 Quick Start
Using 🤗 Transformers to Chat
The checkpoint uses custom Transformers code and the VGGT backbone. Install the required packages before loading the model.
1import torch
2from huggingface_hub import snapshot_download
3from PIL import Image
4from transformers import AutoModel, AutoProcessor
567REPO_ID ="yuanzhang/GEAR-VLA"89# Download the complete checkpoint first because the custom processor loads10# its configuration and tokenizer from a local snapshot.11model_dir = snapshot_download(REPO_ID)1213processor = AutoProcessor.from_pretrained(14 model_dir,15 trust_remote_code=True,16)17model = AutoModel.from_pretrained(18 model_dir,19 torch_dtype=torch.bfloat16,20 low_cpu_mem_usage=True,21 trust_remote_code=True,22 device_map="auto",23).eval()2425# Supply one or more synchronized views of the same scene.26views =[27 Image.open("./view_1.jpg").convert("RGB"),28 Image.open("./view_2.jpg").convert("RGB"),29]3031# The processor creates aligned inputs for both visual branches:32# pixel_values: [V, 3, 448, 448]33# vggt_images: [1, V, 3, 518, 518]34image_inputs = processor.preprocess_images(views)35pixel_values = image_inputs["pixel_values"].to(36 device="cuda",37 dtype=torch.bfloat16,38)39vggt_images = image_inputs["vggt_images"].to(40 device="cuda",41 dtype=torch.bfloat16,42)4344question =(45"\n".join(["<image>"]*len(views))46+"\nDescribe the scene and reason about the spatial relationships "47"between the visible objects."48)49generation_config ={50"do_sample":False,51"max_new_tokens":512,52}5354response = model.chat(55 processor.tokenizer,56 pixel_values,57 question,58 generation_config,59 num_patches_list=[1]*len(views),60 vggt_images=vggt_images,61)62print(response)
For the architecture, training procedure, and evaluation protocol, see the paper.
