To address the degradation of visual-language (VL) representations during VLA supervised fine-tuning (SFT), we introduce Visual Representation Alignment. During SFT, we pull a VLA’s visual tokens toward a frozen teacher’s patch features using cosine similarity through a lightweight frozen projector. This keeps perception anchored while the model learns to act — improving OOD generalization with almost no added cost.
method_1
Model Details
Model Description
The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. This work systematically studies representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, the authors probe VLA's hidden representations and analyze attention maps, and design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. They introduce a simple yet effective method, Visual Representation Alignment, that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios.
The paper also introduces the VL-Think Task Suite, a diagnostic suite assessing the transfer of VL understanding and knowledge from VLMs to VLAs independently of low-level control. This suite focuses on whether models retain the ability to interpret visual symbols, compositional cues, and categorical relations rather than pure manipulation skills.
Developed by: Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov
Model type: Vision-Language-Action (VLA) model (LoRA adapter)
This model is intended for research in Vision-Language-Action (VLA) models, particularly for understanding and improving out-of-distribution (OOD) generalization in robotic and agent control tasks through visual representation alignment. Researchers can use this adapter and methodology to fine-tune base VLA models and explore the impact of representation degradation.
Out-of-Scope Use
As a research artifact, this model is not intended for deployment in real-world, safety-critical applications without further rigorous testing, validation, and adaptation. It is focused on studying and mitigating specific representation issues in VLAs, rather than serving as a production-ready agent.
How to Get Started with the Model
Installation
Use the environment setup commands below to get started:
bash
1# Create and activate conda environment2conda create -n blindvla python=3.10 -y
3conda activate blindvla
45# Install PyTorch. Below is a sample command to do this, but you should check the following link6# to find installation instructions that are specific to your compute platform:7# https://pytorch.org/get-started/locally/8pip install torch torchvision torchaudio
910# Clone and install the BlindVLA repo11git clone https://github.com/CognitiveAISystems/BlindVLA.git
12cd BlindVLA
13pip install -e ./openvla
1415# Install Flash Attention 2 for training (https://github.com/Dao-AILab/flash-attention)16# =>> If you run into difficulty, try `pip cache remove flash_attn` first17pip3 install packaging ninja
18ninja --version;echo$?# Verify Ninja --> should return exit code "0"19pip install"flash-attn==2.5.5" --no-build-isolation
20pip installdiffusers==0.33.0
2122pip install -e ./ManiSkill
23pip install -e ./SimplerEnv
24pip install -U "typeguard>=3"
The pretrained OpenVLA model is warmed up using 140 episodes collected with Octo-Small and a motion planner for 2k steps.
You can download the training dataset (1.4k episodes) here and the warm-up checkpoint here.
Sample Usage (LoRA Fine-tuning with Visual Representation Alignment)
Below is a minimal example from the GitHub README of how you can integrate Visual Representation Alignment into your VLA’s training pipeline. Just plug in these few lines right after your forward pass — no architecture changes are needed.
python
1# ....2# out = vla.forward(..., output_hidden_states=True)3# pixel_values = preprocessor(image, ...)4# ....5#67n_vis = out.projector_features.shape[1]8pos, pos_end =1,910# 1. Extract VLA's visual features from specific layer and project to visual teacher dimention11vla_features = out.hidden_states[align_layer][:, pos:pos_end]12vla_features = alignment_projector(vla_feats)1314# 2. Get teacher patch features15with torch.no_grad():16 teacher_features = teacher_vision_backbone(pixel_values)1718# 3. Compute cosine alignment loss19emb_t = F.normalize(teacher_features, dim=-1)20emb_s = F.normalize(vla_features, dim=-1)2122cossim =(emb_t * emb_s).sum(dim=-1)23align_loss =(-cossim).mean()2425loss += cfg.align_coeff * align_loss
You can run LoRA fine-tuning with Visual Representation Alignment using this script:
The model is warmed up using 140 episodes collected with Octo-Small and a motion planner for 2k steps. A larger training dataset (1.4k episodes) is available here.
Training Procedure
Training Hyperparameters
The model is fine-tuned using LoRA with the following hyperparameters, as described in the provided finetune.py script:
LoRA rank: 32
Batch size: 8
Max steps: 60000
Evaluation steps: 200
Save steps: 0, 5000, 10000, 20000, 30000, 40000, 50000, 60000
Gradient accumulation steps: 1
Learning rate: 5e-4
Image augmentation: True
Evaluation
Testing Data, Factors & Metrics
The model is evaluated using the VL-Think Task Suite, a diagnostic suite assessing the transfer of VL understanding and knowledge from VLMs to VLAs independently of low-level control. The suite includes various tasks, focusing on the ability to interpret visual symbols, compositional cues, and categorical relations.
Examples of tasks include:
Evaluation is performed using batched environments for efficient parallel processing.
Results
The paper demonstrates that Visual Representation Alignment mitigates degradation of visual representations and yields improved generalization to out-of-distribution (OOD) scenarios. For detailed results, refer to the paper.
Citation
If you find our code useful, please cite our paper:
BibTeX
1@misc{kachaev2025dontblindvlaaligning,
2 title={Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization},
3 author={Nikita Kachaev and Mikhail Kolosov and Daniil Zelezetsky and Alexey K. Kovalev and Aleksandr I. Panov},
4 year={2025},
5 eprint={2510.25616},
6 archivePrefix={arXiv},
7 primaryClass={cs.LG},
8 url={https://arxiv.org/abs/2510.25616},
9}
Acknowledgement
BlindVLA is built with reference on: RL4VLA, Simpler, REPA, OpenVLA. Many thanks for their awesome work!
