lift_act_sim_v2 — the no-foam cell
An action chunking transformer trained by behaviour cloning to lift a 2 × 6 × 2 cm red block 10 cm
off the table of a
simulated Dobot X-Trainer dual-arm cell —
omn4i/Lift-v0 in
Physical41/omn4i_envs.
81.0% success on held-out seeds, against the scripted expert's 99.0%.
This is published as a baseline to beat, not a good model. It has a 23-point gap between the
arms and its cause is not yet known. See "The open problem" below.
Which cell this is for — read this before using either version
| lift_act_sim_v1 | lift_act_sim_v2 (this one) |
|---|
| Cell | 50 mm foam slab on the table | no foam — the cell as it is now |
| Trained at | pre-foam-removal tree | commit ebb10c8 (main HEAD) |
| Corpus | 2847 episodes | 1987 episodes |
| Overall | 93.8% | 81.0% |
| Left / right arm | 93.8% / 93.8% | 93.0% / 69.8% |
v1 is for a cell that no longer exists. The foam was removed from main in 326dc18, which
changed every object height by 50 mm and gave the arms back 50 mm of reach. v1's 93.8% is a real
measurement, but of a configuration you cannot reproduce from main today. Use v2 with current
main.
The weights here were verified to load and run under main's own omn4i_envs/baselines/act.py at
ebb10c8. Every file defining the lift task — lift.py, env.py, scene.xml, lift.xml, both
arm XMLs, contract.py, classical.py, benchmark.py, transforms.py — is byte-identical
between main and the tree this was trained on.
Results
500 held-out episodes (benchmark.TEST_SEEDS, range(10000, 10500)), by
python tools/benchmark.py act. None of these seeds was trained on.
| act (this model) | classical (its teacher) |
|---|
| Success | 81.0% (405/500) | 99.0% (495/500) |
| 95% Wilson interval | 77.3% – 84.2% | 97.7% – 99.6% |
| Per-step latency, median → p99 | 7.52 → 7.61 ms | 0.01 → 0.01 ms |
Against a 33 ms environment step the policy uses 7.5 ms. It replans every step.
By arm and band — where it goes wrong
| Band | left | right |
|---|
| under 0.40 m | 95% (52/55) | 95% (42/44) |
| 0.40 to 0.50 m | 92% (78/85) | 74% (62/84) |
| 0.50 to 0.62 m | 94% (80/85) | 54% (62/115) |
| 0.62 m and out | 88% (15/17) | 93% (14/15) |
| overall | 93.0% | 69.8% |
The expert is symmetric on the same seeds (98.8% left, 99.2% right), so nothing about the task
favours one side.
The open problem
The right arm is 23 points behind the left and we do not know why.
What it is not:
- Not data scarcity. The corpus is balanced by arm — 978 left-arm episodes against 1009 right.
The worst cell, right 0.50–0.62 m at 54%, has more training episodes (461) than its left-hand
counterpart (407) which scores 94%.
- Not the two demonstration defects fixed in
main PR #13 (the idle arm swinging 90° every
episode, and the wrist roll scattering across its equivalents). Both are verified absent from
this corpus — idle drift 0.3°, wrist roll within ±90° on both arms. Fixing them coincided with
the gap closing on the foam cell, but the gap returned when only the cell changed, so they were
not the whole cause.
- Not the expert.
classical scores 99.2% on right-arm seeds.
The open lead is the right arm's joint 6 sign, which is unverified upstream. It turns about the
tool axis, which the end-effector site sits on, so no trajectory data can settle it — but it does
set which way the jaws straddle the block. That is consistent with a policy that reaches correctly
and closes on nothing, and with the expert being unaffected, since its inverse kinematics is
self-consistent inside the same model.
Architecture and training
18.7M parameters. One shared ResNet18 across the three cameras, feeding a 4-layer
encoder/decoder at width 256, 8 heads. Predicts 32 actions from one observation, L1 loss, temporal
ensembling at inference. The backbone starts from ImageNet weights and is fine-tuned at 1e-5
against 1e-4 for the rest — 11.2M of the 18.7M parameters, not frozen.
No CVAE (the demonstrations come from a deterministic planner) and no language conditioning (one
task, so the prompt is a constant).
1900 demonstrations / 117,874 frames from Physical41/lift_classical_sim_v1, 87 episodes held out
stratified by reach band. 40,000 steps at batch 64, 59.4 minutes on one RTX 4090. Final held-out
chunk error 0.01123 rad.
Note that held-out chunk error has been a poor predictor of rollout success on this task: a run
at 0.01138 scored 93.4% and one at 0.00928 scored 73.2%. Judge by the benchmark, not by the loss.
Loading
1from omn4i_envs.baselines import act # import before mujoco
2policy = act.ACTPolicy(checkpoint="act.pt")
3
4observation, info = env.reset(seed=10000)
5policy.reset(observation, info)
6action = policy(observation) # (14,) float32, clamped into the action space
Actions are absolute joint position targets in the controller's convention, which differs per
arm — the right is read at the negation of the left, and neither neutral is the zero vector.
Gripper entries are normalized 1.0 open, 0.0 closed.
This is simulation only
Recorded real joint angles do not reproduce the real robot's kinematics in this simulation, and the
joint sign and zero-offset convention between the vendor stack and the MJCF is only partly
resolved — the right arm's joint 6 sign remains open, which may well be the same thing causing the
asymmetry above. Do not cite these numbers as evidence about the physical robot.
