Action Chunking Transformer with the CVAE objective disabled (use_vae=false) — no latent, no KL
term, plain L1 regression onto action chunks. 34.20M parameters against 51.57M for the CVAE
variant.
Final eval_loss (held-out 30 episodes): 0.1988 — the lowest of four variants trained on this
split. Read the evaluation caveat below before drawing any conclusion from that.
Why this baseline exists
The ACT paper's ablation reports that removing the CVAE objective costs almost nothing on scripted
data but drops human-demonstration performance from 35.3% to 2%. Our demonstrations are human and
our latent measured as collapsed, so this run tests whether the collapse matters.
On loss, the answer so far is that it does not — this model is at least as good as every CVAE variant.
Whether that survives contact with the robot is the open question, and the metric is biased in this
model's favour (see below).
✅ Camera keys are correct — wire by name
Physical camera
Observation key
wrist (gripper module)
observation.images.wrist
front (desk-level Brio 101)
observation.images.front
Verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the
transposed mapping from the 8bin models — that swap belongs to a different dataset.
The top camera was recorded but is not an input to this policy.
Held-out episodes — evaluate on these
30 episodes were excluded from training: 5 from each of the 6 object x container blocks.
Objects and containers, by episode_index: red cube 0-39, yellow cylinder 40-79, white cube 80-119;
cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one
container was on the table at a time, so the visible bin is always the target.
Inputs / outputs
Shape
observation.images.wrist
(3, 480, 640)
observation.images.front
(3, 480, 640)
observation.state
(6,)
action
(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed
Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights. Load through
ACTPolicy.from_pretrained or actions come out in the wrong units.
🚨 Evaluation caveat that applies to both models
These two policies are indistinguishable on held-out loss. Final eval_loss: 0.2041 (CVAE) vs
0.1988 (BC). But a single model's eval bounces by ~0.008 between consecutive checkpoints, so the
0.005 spread between models is smaller than the noise within either one.
Worse, the metric is biased for this data. The demonstrations contain roughly three distinct grasp
approaches per object, and held-out L1 systematically rewards a policy that averages them — a
policy committing crisply to approach 2 is penalised on every held-out episode that used approach 1
or 3. BC's marginally lower number is exactly what an averaging model would produce.
Only scored rollouts can rank these. Do not conclude from the loss.
AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone
ResNet-18, ImageNet-pretrained, not frozen
n_obs_steps
1 — single frame, no history
Hardware
1x H200, Northeastern Explorer
No overfitting was observed: eval_loss is flat from step 60,000 to 100,000, so any checkpoint in
that range is equivalent. This is the 100,000-step checkpoint.
Known limitations
n_obs_steps=1, so no velocity information.
Scene-dependent: the cardboard box and white bin as used during recording. A different container is a domain shift.
One room, one lighting setup, one operator.
⚠️ The operator observed a camera frame-rate warning on battery power during roughly the first 50 episodes of collection. Recorded as reported; a pixel-level check found 4-9% stalled frames uniformly across the dataset with no early-vs-late pattern, so this is unconfirmed.
⚠️ The white cube is low contrast against the light table in the overhead view. Not an input here (top camera unused), but it is the object most likely to underperform.
