BaseVLA — ManiSkill Franka (Qwen2.5-VL-3B), vision-only, with augmentation
Plain behaviour cloning baseline: no V-JEPA, no latent alignment, no
decorrelation. Just L1(action) on ManiSkill's merged Franka set, trained
without proprioception and with domain augmentation on.
Checkpoint at step 40,000. The run was stopped early at ~44k steps (of a
planned 100k) because the loss had flattened — 0.0487 at 30k to 0.0474 at 42k.
Data
maniskill-franka-merged from Ngseo/stage1-data,
single camera (observation.images.anchor, 256×256 h264), 30 fps, action
action.ee_delta_pose (7-D).
The raw set needs filtering. 17.1% of its action.ee_delta_pose rows are
NaN, and they are not spread evenly — whole tasks were never populated:
task
frames
NaN (ee_delta)
NaN (joint_delta)
NaN (joint_pos)
PushT-v1
168,934
100%
100%
100%
PokeCube-v1
68,545
69.7%
32.5%
100%
PlaceSphere-v1
118,223
12.1%
2.6%
0.8%
PegInsertionSide-v1
150,055
6.2%
0.7%
0.7%
other 7 tasks
946,166
0.4–2.1%
The ~1% floor is the last frame of each episode (no next action) and is normal.
PushT has no usable action under any of the three control modes.
Training on it unfiltered silently produces a broken model: NaNs get zeroed by
nan_to_num, so the policy learns "output nothing" on 17% of frames, and the
normalisation statistics come out as NaN.
This run therefore drops episodes whose action column is NaN in more than 5% of
rows (--max_nan_frac 0.05):
9,216 / 12,257 episodes kept → 1,218,188 frames (83.3% of the original)
Architecture
Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
└─ Head B → z_b ─┴─ concat → ResNetActionHead → 30×7
L = L1(action) ← that is the whole objective
--no_state is used, so the state token degenerates to a constant learned bias
(state_proj(0)) and no proprioception reaches the model. ManiSkill's merged
set has no observation.state column to begin with.
Training curve
step
loss
1,000
0.0962
10,000
0.0601
20,000
0.0524
30,000
0.0487
40,000
0.0476
Not evaluated in simulation or on a robot. The number above is a
training-set L1 on normalised actions. No ManiSkill success rate is reported.
Contents
checkpoint.pt holds policy (full VLM incl. LoRA), latent_head,
free_latent_head, action_head, optimizer, agg_stats (normalisation
statistics) and args (the exact CLI configuration, including the NaN filter
threshold).
python
1import torch
2ck = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)3ck["step"]# 400004ck["args"]# full training config5ck["agg_stats"]# action normalisation used at train time
