SO-ARM101 pick-and-place — ACT, single task (cube)
An
ACT (Action Chunking Transformer) policy trained from
scratch on one SO-ARM101 tabletop task: pick up a cube and place it in the bin.
Read this first. This checkpoint
does not clear our offline acceptance bar — on every one of
its six joints, its best step still predicts motion with larger error amplitude than a
"hold current pose" baseline on at least one axis (
worst_ratio 1.681, needs < 1). It is published
as a
reference and comparison artifact, not a deployable policy. See
Offline evaluation.
Task
Single task, no language conditioning. LeRobot's ACT implementation takes no language input —
which is exactly why a single task was chosen for it: on a multi-task mix ACT cannot see the
instruction, so identical frames from different tasks would only teach it a cross-task average.
| Task | Episodes | Frames |
|---|
| pick up a cube and place it in the bin | 300 | 106,085 |
Chosen from a nine-task set because it has the joint-highest episode count, and a cube is a rigid
body with clean geometry and stable appearance — the most standard pick-and-place benchmark of the
nine. The two stacking tasks need precise alignment and are clearly harder, so they were excluded.
Data
- Source: the
pick_up_a_cube_and_place_in_the_bin split of
zhuzhuangtian/so101-pick-place-tasks
on ModelScope (Apache-2.0), LeRobot v3.0 format.
- Robot: SO-ARM101 follower arm, 6 DoF.
- Cameras: two RGB streams,
observation.images.top and observation.images.wrist, 480×640 each, 30 fps.
observation.state and action are both float32(6): shoulder_pan, shoulder_lift,
elbow_flex, wrist_flex, wrist_roll, gripper (joint positions, absolute targets).
Training
| |
|---|
| Base | none — trained from scratch; vision backbone is torchvision ImageNet ResNet-18 |
| Params | ≈ 51.6 M |
| Architecture | 4 encoder / 1 decoder layers, dim_model 512, VAE objective enabled |
| Steps | 50,000 (≈ 11.3 epochs at effective batch 24) |
| Batch | 8 per device × 3 devices = 24 effective |
| Optimizer | AdamW, lr 1e-5 constant, wd 1e-4, betas (0.9, 0.999) |
| Schedule | none — ACT's LeRobot preset returns no scheduler (no warmup, no decay) |
| Action chunk | 100 (chunk_size = n_action_steps = 100) |
| Augmentation | brightness/contrast 0.7–1.3, hue ±0.05, saturation 0.5–1.5, sharpness 0.5–1.5, affine ±5° / translate 0.05; up to 5 of 6 sampled per frame |
| Normalization | state/action/visual all MEAN_STD from real dataset statistics |
Final training loss 0.126. Note that ACT's loss is not comparable to that of the π0/SmolVLA
siblings below — constant LR versus cosine schedules, and a different objective.
Published checkpoint = step 40,000, the best of 10 saved checkpoints on the metrics below.
Offline evaluation
No simulator exists for this rig, so evaluation is offline only, against a
copy-the-current-joint-angles baseline:
ratio = MAE(prediction, action) / MAE(current_state, action) — must be < 1.delta_corr = corr(prediction − state, action − state) — must be > 0.delta_std — ratio of predicted to true motion magnitude; ≈ 1 is healthy.
Best checkpoint (step 40,000), 200 frames sampled uniformly across the task:
| value | verdict |
|---|
| worst_ratio | 1.681 (wrist_roll) | ✗ fails the < 1 bar |
| best joint ratio | 0.475 (gripper) | ok |
| worst delta_corr | 0.618 | direction is learned |
| delta_std | 1.086 – 1.478, all six joints > 1 | over-shoots |
Failure mode: amplitude over-shoot, not wrong direction. delta_corr is respectable
(0.618–0.870), so the policy has learned where to move; but delta_std > 1 on every joint means
it predicts increments 9–48 % larger than ground truth, and that inflation pushes total error past
the do-nothing baseline. All 10 checkpoints fail the bar (worst_ratio 1.905 → 1.688), and the
curve is flat after step 20,000 — this is not under-training, so simply training longer is not
the fix. Plausible directions instead: temporal ensembling of chunks at inference, a lower action
chunk size, or loss/normalization changes.
The weakest axes are the two wrist DoF (wrist_roll, wrist_flex) — the same weak spot the π0
and SmolVLA models show, on data from the same source. Across three different architectures, that
points at the data rather than at any one model.
Real-robot status
Not validated on hardware, and not recommended for hardware. The over-shoot failure mode is the
kind that gets worse in closed loop, where errors accumulate. If you run it anyway, reduce speed and
keep a hand on the e-stop.
Usage
1from lerobot.policies.act.modeling_act import ACTPolicy
2
3policy = ACTPolicy.from_pretrained("Harrysunshine/so101-act-cube")
4# batch: observation.state (6,), observation.images.top (3,480,640),
5# observation.images.wrist (3,480,640). No language input.
6action = policy.select_action(batch)
The two *normalizer_processor.safetensors files are required. Without them LeRobot silently
falls back to identity normalization: loss looks fine, but every action comes out at the wrong scale.
Load the repo as a whole, do not cherry-pick model.safetensors.
Only inference assets are published here. Optimizer/scheduler/RNG state is not included, so this repo
cannot be used to resume training.
Limitations
- Does not pass the offline acceptance bar (
worst_ratio 1.681 ≥ 1). Reference artifact, not a
deployable policy.
- No language conditioning — one task only, cannot be re-targeted by prompt.
- Rig-specific top camera. The training data comes from a different physical setup than ours: the
wrist view matches pixel-for-pixel (same SO-ARM101 follower hardware), but the top camera
extrinsics differ. Dropping this checkpoint onto another SO-ARM101 will not work without camera
adaptation or a short fine-tune on target-rig data.
- Absolute joint-position action space; sending these as deltas will diverge in closed loop.
- Fixed 480×640 dual-camera observation.
- Single cube on the training tabletop; no generalization claims beyond it.
Comparison note
This model is a single-task run; the two siblings below are 9-task mixes on a superset of the
same source data. Different training sets, so the metric magnitudes cannot be ranked against each
other. What is comparable is the failure mode and the location of the weak axes: ACT over-shoots
(delta_std > 1) where π0 and SmolVLA under-shoot (0.71–0.99), and all three are weakest at the
wrist.
Harrysunshine/so101-pi0-9task — π0 full fine-tune, 9 tasks, passes the bar.Harrysunshine/so101-smolvla-9task — SmolVLA, 9 tasks, passes the bar.
License
Apache-2.0, following LeRobot and the source dataset.
中文说明
SO-ARM101 桌面抓放 ACT 单任务（方块放进筐）权重，从零训练（视觉塔＝torchvision ImageNet ResNet-18），约 51.6M 参数。
先看这条：本 checkpoint 没过我们的离线硬底线（worst_ratio 1.681，要求 < 1），
作为对照与参考产物发布，不是可部署策略。
- 为什么是单任务：lerobot 的 ACT 不接语言输入。9 任务混训时它看不到任务描述，
同样画面+关节角下只能学跨任务平均动作。九任务里挑方块＝集数并列最多、刚体几何清晰外观稳定，
两个 Stack 要精确对准明显更难，排除。
- 配方＝ACT 官方默认：LR 1e-5 恒定（ACT 的 preset 不返回 scheduler，无 warmup 无衰减），
50k 步、有效 batch 24（epoch 11.31）、action chunk 100。
- 三个模型的 loss 不可横向比：ACT 恒定 LR，π0/SmolVLA 都是余弦，只有各自收敛后的验收指标可比。
- 发布第 40,000 步，10 存点里最优。
- 失效模式＝幅度过冲，与 π0/SmolVLA 方向相反：
delta_corr 0.618–0.870 其实不低（方向学对了），
但 delta_std 六关节全 > 1（1.086–1.478）＝预测增量比真值大 9–48%，误差被放大到超过照抄基线。
10 个点全不过门、20k 步后曲线就平了 ⇒ 不是训练不足，再训不解决；可试的方向是推理端 chunk 时序集成、
减小 chunk、或改损失/归一化口径。
- 腕部两自由度最差，与 π0/SmolVLA 一致 ⇒ 跨三个架构同一短板，是数据的难点。
- 真机未验证、也不建议上真机：过冲这种失效模式在闭环里会累积。真要跑就降速 + 手放急停。
- 两个
*normalizer_processor.safetensors 必须一起加载，缺了静默退回恒等归一化、动作尺度整体错。
- 边界：top 相机外参与我们自己的臂不同，换机器需相机适配或少量目标机器数据再微调。
