OpenVLA-OFT without memory, MIKASA-Robo, 5 tasks (mu-VLA ablation)
This is the
memoryless control for the mu-VLA checkpoints: plain
OpenVLA-OFT, with no memory tokens and no recurrent
state, fine-tuned on the same five MIKASA-Robo-VLA environments, with the same episodic
dataloader, for the same number of steps. It exists so that any gain reported by a mu-VLA
checkpoint can be attributed to the memory module rather than to the data, the task mixture or the
fine-tuning recipe.
The model is a per-step policy: every action is predicted from the current observation alone.
On the memory-heavy MIKASA-Robo tasks, where the cue is shown early in the episode and is no longer
visible when the action has to be taken, this is the behaviour a memory mechanism has to beat.
This checkpoint corresponds to experiment 2 in the mu-VLA experiment matrix, at training step
150000.
Sibling checkpoints, same five environments, with recurrent memory:
Training setup
Base model: openvla/openvla-7b, fine-tuned with the OpenVLA-OFT recipe (L1 regression action
head, proprioception projector, two input camera views, no FiLM, no diffusion). No memory module
is present in the checkpoint.
| Setting | Value |
|---|
| Checkpoint step | 150000 |
| Memory | none (use_memory off) |
| Dataset | MIKASARoboVLAEpisodicDataset, mikasa_five |
| Training environments | ShellGamePush-VLA-v0, InterceptMedium-VLA-v0, RememberColor5-VLA-v0, TakeItBack-VLA-v0, RememberShapeAndColor3x3-VLA-v0 |
| GPUs | 8 x NVIDIA A100 80GB |
| Batch size | 4 per GPU |
| LoRA rank | 32 |
| Learning rate | 5e-4, constant for 100000 steps, then decayed |
| Image augmentation | on |
The learning rate schedule is the difference to keep in mind when reading this model as a control:
the memory checkpoints use a cosine schedule with 2000 warmup steps and a minimum ratio of 0.1,
while this run uses the constant-then-decay schedule of the original OpenVLA-OFT recipe. Everything
else, including the dataloader, the mixture and the number of steps, is identical.
The episodic dataloader feeds batch_size independent streams, each replaying whole episodes in
order, and marks every step with is_first and is_last. It is used here as well, so that the
control sees exactly the same frame ordering as the memory arms; without a memory state, that
ordering only affects which frames land in a batch. Action normalisation statistics (q01/q99)
are computed jointly over all five environments and stored under the mikasa_combined key in
dataset_statistics.json.
Evaluation
Protocol: each environment is evaluated independently, 100 episodes, starting seed 4242424242,
metric success_once.
Both inference modes are reported. Receding horizon queries the model at every environment
step and executes only the first of the 8 predicted actions; open loop executes the whole
predicted chunk before querying again. Receding horizon costs 8 times more forward passes and is
the mode used for the numbers quoted in the memory checkpoints' cards.
Only the first five rows are training environments. Everything else is zero-shot transfer to
environments this model never saw during fine-tuning.
| Environment | Receding horizon | Open loop |
|---|
ShellGamePush-VLA-v0 (train) | 0.33 | 0.90 |
InterceptMedium-VLA-v0 (train) | 0.53 | 0.39 |
RememberColor5-VLA-v0 (train) | 0.23 | 0.09 |
TakeItBack-VLA-v0 (train) | 0.98 | 0.87 |
RememberShapeAndColor3x3-VLA-v0 (train) | 0.11 | 0.13 |
ShellGameTouch-VLA-v0 | 0.00 | 0.00 |
ShellGamePick-VLA-v0 | 0.01 | 0.00 |
InterceptSlow-VLA-v0 | 0.06 | 0.04 |
InterceptFast-VLA-v0 | 0.24 | 0.33 |
InterceptGrabSlow-VLA-v0 | 0.00 | 0.00 |
InterceptGrabMedium-VLA-v0 | 0.00 | 0.00 |
InterceptGrabFast-VLA-v0 | 0.00 | 0.00 |
RotateLenientPos-VLA-v0 | 0.08 | 0.04 |
RotateLenientPosNeg-VLA-v0 | 0.07 | 0.08 |
RotateStrictPos-VLA-v0 | 0.03 | 0.04 |
RotateStrictPosNeg-VLA-v0 | 0.05 | 0.03 |
RememberColor3-VLA-v0 | 0.25 | 0.19 |
RememberColor9-VLA-v0 | 0.07 | 0.11 |
RememberShape3-VLA-v0 | 0.17 | 0.08 |
RememberShape5-VLA-v0 | 0.15 | 0.11 |
RememberShape9-VLA-v0 | 0.11 | 0.11 |
RememberShapeAndColor3x2-VLA-v0 | 0.12 | 0.09 |
RememberShapeAndColor5x3-VLA-v0 | 0.08 | 0.06 |
Files
| File | Purpose |
|---|
model-0000*-of-00004.safetensors, model.safetensors.index.json | merged vision-language backbone |
lora_adapter/ | LoRA adapter as saved by PEFT, before merging |
action_head--150000_checkpoint.pt | L1 regression action head |
proprio_projector--150000_checkpoint.pt | proprioception projector |
dataset_statistics.json | action normalisation statistics, key mikasa_combined |
configuration_prismatic.py, modeling_prismatic.py, processing_prismatic.py | custom model code |
There is no memory_module--150000_checkpoint.pt and no memory_meta.json here: that is the
point of this checkpoint. Optimizer and scheduler state are not published for this run.
Usage
1huggingface-cli download mu-vla/mu-vla-openvla-oft-mikasa-robo-5-tasks-no-memory \
2 --local-dir ./openvla-oft-mikasa-no-memory
The checkpoint needs the mu-VLA code, which depends on a forked
transformers, and on
MIKASA-Robo for the environments. Point the
evaluation script at the downloaded directory; with no
memory_meta.json present it runs as plain
OpenVLA-OFT. See
CognitiveAISystems/muVLA for setup
and the exact command line.
Attribution
mu-VLA is a fork of OpenVLA-OFT by Moo Jin Kim, Chelsea Finn and Percy Liang
(
arXiv:2502.19645), which in turn builds on OpenVLA. The
recurrent memory module, the episodic dataloaders and the MIKASA-Robo integration are the
contribution of this work. Released under the MIT license, following upstream.