Action Chunking Transformer with the CVAE objective disabled (use_vae=false) — no latent, no KL
term, plain L1 regression onto action chunks — trained on all three camera feeds.
34.20M parameters against 51.57M for the CVAE variant.
Final eval_loss on the held-out 30 episodes: 0.2067 — the highest of six variants, though the whole
spread is within noise.
One observation worth recording
This is the only variant whose eval curve sat clearly above the others early on: 0.2330 at 10k and
0.2326 at 20k, while every other run was already near 0.21. It converged by 100k, but it is the sole
configuration with no regularization of any kind — no latent, no KL, no noise token — and it is
also the one given the most capacity, three ResNet-18 backbones and 903 encoder tokens.
A plausible reading is that the CVAE's contribution here is capacity control rather than
mode-encoding (the β sweep having ruled out the latter), and that it only becomes visible once the
model has enough capacity to misuse. Two runs is not evidence. Offered as a hypothesis for the
rollouts to test, not a conclusion.
✅ Camera keys are correct — and all THREE are required
Physical camera
Observation key
wrist (gripper module)
observation.images.wrist
front (desk-level Brio 101)
observation.images.front
top (overhead, boom arm)
observation.images.top
This policy will not run without the overhead camera connected. Its two-camera sibling
(..._chunk50_bc) takes wrist+front only.
Keys verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the
transposed mapping from the 8bin models.
5 episodes from each of the 6 object x container blocks. By episode_index: red cube (25 mm) 0-39,
yellow cylinder 40-79, white cube (45 mm) 80-119; cardboard box in the first 20 of each block, white
3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin
is always the target.
Inputs / outputs
Shape
observation.images.wrist
(3, 480, 640)
observation.images.front
(3, 480, 640)
observation.images.top
(3, 480, 640)
observation.state
(6,)
action
(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed
903 encoder tokens (301 per camera at 640x480, plus latent and state), against 602 for the two-camera
variants. Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights — load via
ACTPolicy.from_pretrained.
🚨 All six variants are indistinguishable on held-out loss
Variant
cameras
latent
eval_loss @100k
cvae
2
CVAE β=10
0.2041
bc
2
none
0.1988
cvae_kl1
2
CVAE β=1
0.2014
cvae_kl0p1
2
CVAE β=0.1
0.2018
cvae_3cam
3
CVAE β=10
0.2006
bc_3cam
3
none
0.2067
The full spread is 0.0079. A single run's eval_loss bounces by ~0.008 between consecutive
checkpoints. The between-model differences do not exceed the within-model noise.
And the metric is biased for this data: the demonstrations contain roughly three distinct grasp
approaches per object, so held-out L1 rewards a policy that averages them over one that commits to
a single valid approach. A worse number here may be a better policy on the robot.
AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone
ResNet-18 per camera, ImageNet-pretrained, not frozen
n_obs_steps
1 — single frame, no history
Hardware
1x H200, 1 h 48 m wall, 5.7 GB VRAM
eval_loss is flat from step 60,000 to 100,000, so no overfitting penalty; this is the
100,000-step checkpoint.
Known limitations
n_obs_steps=1, so no velocity information.
Scene-dependent: the cardboard box and white bin as used during recording, and the camera poses as mounted. A moved camera or a different container is a domain shift.
One room, one lighting setup, one operator.
⚠️ A camera frame-rate warning was observed on battery power during roughly the first 50 episodes of collection. Reported but unconfirmed — a pixel-level check found 4-9% stalled frames uniformly with no early-vs-late pattern.
⚠️ The white cube is low contrast against the light table, most visibly in the overhead view — which this policy uses. Episodes 80-119 are the likeliest to underperform.
