Action Chunking Transformer, standard CVAE objective (use_vae=true, kl_weight=10.0), trained with
LeRobot 0.6.0.
Task: pick up one of three objects (red 25 mm cube, yellow cylinder, white 45 mm cube) and place it in
whichever container is on the table (cardboard box or white 3D-printed bin).
Paired baseline: act_so101_cubcyl_poshold_chunk50_bc
— identical in every respect except use_vae=false. Trained to answer whether the CVAE objective
matters on this data.
Final eval_loss (held-out 30 episodes): 0.2041. Note this includes the kl_weight x KL term and
the BC model's does not, so the two numbers are not directly comparable.
The latent is collapsed, and lowering the penalty did not change that
Measured across a β sweep at kl_weight = 10, 1 and 0.1: all three finished with train loss
0.056-0.057, identical to the BC run's 0.059, which has no KL term at all. At β=0.1 a KL of 1 nat
would add 0.1 to the loss; nothing of the sort appears, putting the KL below roughly 0.02 nats
regardless of β.
So the collapse is not caused by the penalty. The likely explanation is that observation.state
already discloses which grasp approach is underway for most frames — the demonstrations are
multimodal at the task level but close to deterministic per-observation, which is what the encoder
actually conditions on.
Practical consequence: this model behaves close to a deterministic chunk regressor. Do not expect
behavioural diversity from the latent.
✅ Camera keys are correct — wire by name
Physical camera
Observation key
wrist (gripper module)
observation.images.wrist
front (desk-level Brio 101)
observation.images.front
Verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the
transposed mapping from the 8bin models — that swap belongs to a different dataset.
The top camera was recorded but is not an input to this policy.
Held-out episodes — evaluate on these
30 episodes were excluded from training: 5 from each of the 6 object x container blocks.
Objects and containers, by episode_index: red cube 0-39, yellow cylinder 40-79, white cube 80-119;
cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one
container was on the table at a time, so the visible bin is always the target.
Inputs / outputs
Shape
observation.images.wrist
(3, 480, 640)
observation.images.front
(3, 480, 640)
observation.state
(6,)
action
(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed
Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights. Load through
ACTPolicy.from_pretrained or actions come out in the wrong units.
🚨 Evaluation caveat that applies to both models
These two policies are indistinguishable on held-out loss. Final eval_loss: 0.2041 (CVAE) vs
0.1988 (BC). But a single model's eval bounces by ~0.008 between consecutive checkpoints, so the
0.005 spread between models is smaller than the noise within either one.
Worse, the metric is biased for this data. The demonstrations contain roughly three distinct grasp
approaches per object, and held-out L1 systematically rewards a policy that averages them — a
policy committing crisply to approach 2 is penalised on every held-out episode that used approach 1
or 3. BC's marginally lower number is exactly what an averaging model would produce.
Only scored rollouts can rank these. Do not conclude from the loss.
AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone
ResNet-18, ImageNet-pretrained, not frozen
n_obs_steps
1 — single frame, no history
Hardware
1x H200, Northeastern Explorer
No overfitting was observed: eval_loss is flat from step 60,000 to 100,000, so any checkpoint in
that range is equivalent. This is the 100,000-step checkpoint.
Known limitations
n_obs_steps=1, so no velocity information.
Scene-dependent: the cardboard box and white bin as used during recording. A different container is a domain shift.
One room, one lighting setup, one operator.
⚠️ The operator observed a camera frame-rate warning on battery power during roughly the first 50 episodes of collection. Recorded as reported; a pixel-level check found 4-9% stalled frames uniformly across the dataset with no early-vs-late pattern, so this is unconfirmed.
⚠️ The white cube is low contrast against the light table in the overhead view. Not an input here (top camera unused), but it is the object most likely to underperform.
