Action Chunking Transformer with the standard CVAE objective (use_vae=true, kl_weight=10.0),
trained on all three camera feeds, with LeRobot 0.6.0.
51.57M parameters.
Task: pick up one of three objects (red 25 mm cube, yellow cylinder, white 45 mm cube) and place it in
whichever container is on the table.
Final eval_loss on the held-out 30 episodes: 0.2006 — the second-lowest of six variants, and within
noise of all of them.
The latent is collapsed, and the penalty is not why
A β sweep at kl_weight = 10, 1 and 0.1 all finished with train loss 0.056-0.057 — identical to the
BC run's 0.059, which has no KL term at all. At β=0.1 a KL of 1 nat would add 0.1 to the loss and
nothing of the sort appears, putting the KL under roughly 0.02 nats regardless of β.
So collapse is not caused by the penalty. The likely reason is that observation.state already
discloses which grasp approach is underway for most frames: the demonstrations are multimodal at the
task level but close to deterministic per-observation, which is what the encoder conditions on.
Treat this as a near-deterministic chunk regressor. Do not expect behavioural diversity from the
latent.
✅ Camera keys are correct — and all THREE are required
Physical camera
Observation key
wrist (gripper module)
observation.images.wrist
front (desk-level Brio 101)
observation.images.front
top (overhead, boom arm)
observation.images.top
This policy will not run without the overhead camera connected. Its two-camera sibling
(..._chunk50_cvae) takes wrist+front only.
Keys verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the
transposed mapping from the 8bin models.
5 episodes from each of the 6 object x container blocks. By episode_index: red cube (25 mm) 0-39,
yellow cylinder 40-79, white cube (45 mm) 80-119; cardboard box in the first 20 of each block, white
3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin
is always the target.
Inputs / outputs
Shape
observation.images.wrist
(3, 480, 640)
observation.images.front
(3, 480, 640)
observation.images.top
(3, 480, 640)
observation.state
(6,)
action
(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed
903 encoder tokens (301 per camera at 640x480, plus latent and state), against 602 for the two-camera
variants. Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights — load via
ACTPolicy.from_pretrained.
🚨 All six variants are indistinguishable on held-out loss
Variant
cameras
latent
eval_loss @100k
cvae
2
CVAE β=10
0.2041
bc
2
none
0.1988
cvae_kl1
2
CVAE β=1
0.2014
cvae_kl0p1
2
CVAE β=0.1
0.2018
cvae_3cam
3
CVAE β=10
0.2006
bc_3cam
3
none
0.2067
The full spread is 0.0079. A single run's eval_loss bounces by ~0.008 between consecutive
checkpoints. The between-model differences do not exceed the within-model noise.
And the metric is biased for this data: the demonstrations contain roughly three distinct grasp
approaches per object, so held-out L1 rewards a policy that averages them over one that commits to
a single valid approach. A worse number here may be a better policy on the robot.
AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone
ResNet-18 per camera, ImageNet-pretrained, not frozen
n_obs_steps
1 — single frame, no history
Hardware
1x H200, 2 h 00 m wall, 5.7 GB VRAM
eval_loss is flat from step 60,000 to 100,000, so no overfitting penalty; this is the
100,000-step checkpoint.
Known limitations
n_obs_steps=1, so no velocity information.
Scene-dependent: the cardboard box and white bin as used during recording, and the camera poses as mounted. A moved camera or a different container is a domain shift.
One room, one lighting setup, one operator.
⚠️ A camera frame-rate warning was observed on battery power during roughly the first 50 episodes of collection. Reported but unconfirmed — a pixel-level check found 4-9% stalled frames uniformly with no early-vs-late pattern.
⚠️ The white cube is low contrast against the light table, most visibly in the overhead view — which this policy uses. Episodes 80-119 are the likeliest to underperform.
