Stage 3 — UR5 teleop VLA (Qwen2.5-VL-3B), act-heavy without augmentation
Training run complete: 60,000 steps. Byte-identical setup to
Ngseo/stage3-ur5-actheavy
except augmentation is off — raw decoded frames go to both the VLM and
V-JEPA2, with no ColorJitter/SharpnessJitter and no DomainRandomization.
The config diff against its partner is one line: no_aug: true.
The setup this was trained for
Each of the 7 tasks in Ngseo/ur5_teleop_multitask
is only ever shown from one of the 4 cameras, so viewpoint alone almost
determines the task. Object colour is deliberately crossed between the two task
families so that "colour ⇒ camera" is not a valid shortcut on its own:
camera
tasks
camera_0
Point at the red cup · Pick up the blue die → basket
camera_1
Pull a tissue out of the box · Close the laptop · Stand the shoe upright
camera_2
Point at the blue cup · Pick up the red die → basket
Architecture
Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
└─ Head B → z_b ─┴─ concat(8192) → ResNetActionHead → 30×7
L = 1.0·L1(action) + 0.02·InfoNCE(z_a, z_task_future) + 0.002·SIGReg([z_domain_future ; z_b])
Frozen target encoder: Ngseo/stage1
disentangled V-JEPA2 ViT-L, on future frames (8 frames, stride 4 → 1.07 s ahead).
AdamW, lr 1e-4, 1000-step warmup + cosine to 0, batch 32, bf16
Augmentation
none
Augmentation ablation — final metrics
epoch (10k steps each)
no-aug (this)
with aug
1
0.0602
0.0836
2
0.0399
0.0649
3
0.0256
0.0442
4
0.0195
0.0351
5
0.0164
0.0305
6 (final)
0.0163
0.0300
InfoNCE (chance 3.466)
1.54
1.79
cos(z_a, z_target)
0.311
0.292
cos(z_b, z_domain)
−0.000
−0.000
Removing augmentation cuts the training action L1 by ~46% and lets InfoNCE fall
further. Read this the right way round: augmentation exists to trade
training fit for generalisation, so a lower training loss here is the expected
consequence of an easier problem, not evidence of a better policy. Whether the
augmented model is actually better only shows up out of distribution — asking
for a task from a camera it never saw it from — and that measurement is not in
this repo.
Reference points on the same normalised scale, none of which use vision or
language: dataset mean 0.834, copying the input state across all 30 steps
0.155.
Not evaluated on a robot. Everything above is a training-set loss.
The 7-D joint state is an input here, and a probe on that state alone recovers
which of the 7 tasks is running with 94% accuracy (chance 14%), so a
viewpoint-OOD failure could not be attributed to the viewpoint shortcut alone.
epoch_6.pt holds model_state_dict (full VLM incl. LoRA), latent_head_state_dict
(Head A), free_latent_head_state_dict (Head B), stage2_action_head_state_dict,
optimiser state, and the run config. config.yaml is the exact training config.
