ACT · SO-101 · 8-bin pick-and-place · wrist + top · chunk 50
Action Chunking Transformer trained with
LeRobot 0.6.0
on
BrutalCaesar/phi_so101_8bin_v1.
Two cameras:
wrist (gripper module) +
top (EMEET C960, overhead on a boom arm).
One of six runs in a chunk-size × camera-pair sweep (
{50, 75, 100} ×
{wrist+top, wrist+front}).
Sibling with the same chunk size, different second camera:
act_so101_8bin_wrist_front_chunk50.
🚨 Both camera keys are transposed — read this before inference
The dataset was recorded with the wrist and top camera keys swapped, and this model uses both
of them. So the key names are exactly backwards from what you want:
| Feed this physical camera | Under this observation key |
|---|
| wrist (gripper module) | observation.images.top |
| top (overhead, boom arm) | observation.images.wrist |
Wire these by name and you get a fully inverted setup: the policy receives an overhead shot where
it expects a gripper close-up and vice versa. It will not error. It will load, run, and produce
plausible-looking garbage.
Verified against pixels (episode 0, frame 200): observation.images.top holds a gripper close-up of
the target object; observation.images.wrist holds the overhead view of the whole arm and table.
The key names are wrong in the data; the physical rig is labelled correctly.
Camera framing matters too — the mounts must sit where they sat during recording. A camera that
drifted between recording and evaluation silently invalidates the policy.
Inputs / outputs
| Shape | Notes |
|---|
observation.images.top | (3, 480, 640) | physical wrist camera, no resize anywhere in the pipeline |
observation.images.wrist | (3, 480, 640) | physical top / overhead camera |
observation.state | (6,) | SO-101 joint positions |
| action | (50, 6) | chunk of 50 future actions, all executed (n_action_steps=50) |
Normalization ships with the checkpoint as LeRobot 0.6.0 processor files
(policy_preprocessor*, policy_postprocessor*) — not baked into the policy. Load through
ACTPolicy.from_pretrained so they are picked up; hand-rolling a forward pass without them
produces actions in the wrong units.
Training
| |
|---|
| Steps | 100,000 (= 800K samples, ≈14.6 epochs over 54,800 frames) |
| Batch size | 8 |
| Final train loss | 0.056 |
| Optimizer | AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4 |
| Backbone | ResNet-18, ImageNet-pretrained, not frozen |
dim_model / chunk_size / kl_weight | 512 / 50 / 10.0 |
n_obs_steps | 1 (single frame, no history) |
| Seed | 1000 |
| Hardware | 1× Tesla V100-32GB, Northeastern Explorer cluster, 4 h 41 m |
Everything except chunk_size and the camera pair is LeRobot default, deliberately — the
sweep is meant to be comparable across people training the same task on the same defaults.
Train / held-out split
Trained on 89 episodes covering 6 of the 8 bins. Left bin 3 and right bin 2 are held out
entirely — no episode touching them appears in training. Those two bins are the generalization
test; the other six measure fit.
Evaluation
Not yet evaluated on hardware. No success rate is reported here because none has been measured.
The planned protocol is 20 scored rollouts per bin, reported per bin and never averaged — an
average over 8 bins hides exactly the failure this split is designed to expose.
This model's train loss (0.056) is within 0.001 of its wrist+front sibling (0.055). Do not read
that as the two camera setups being equivalent — it means training loss cannot distinguish them,
which is why the rollouts exist.
Usage
1from lerobot.policies.act.modeling_act import ACTPolicy
2
3policy = ACTPolicy.from_pretrained("BrutalCaesar/act_so101_8bin_wrist_top_chunk50")
Or point a LeRobot eval script at it with --policy.path=BrutalCaesar/act_so101_8bin_wrist_top_chunk50,
remembering the transposed camera keys above when you write --robot.cameras.
Known limitations
- Single observation frame, so no velocity information.
- The CVAE latent is effectively collapsed at
kl_weight=10.0 on this dataset (KL ≈ 0 throughout training), so this behaves close to a deterministic chunk regressor. Expected for ACT, not a defect — but do not read the latent as a source of behavioral diversity.
- Trained on one lighting setup in one room. No domain randomization beyond LeRobot's default image augmentation.
- The overhead camera sits ~105 cm above the workspace at 90° dFOV; a boom arm that sags or gets bumped changes the observation more than the wrist camera does.
