A π0.5 (pi05) policy finetuned on the RoboSynChallenge click_bell task: a bimanual
CobotMagic robot must press a desk bell placed at a random position on the table.
Trained from pi05_base on 1,000 synthetic demonstrations
(RoboSynChallenge/cobotmagic_Sim_click_bell) for 20,000 steps.
The gap between those two rows is not a model limitation. The default sampling range
contains a large region that the right arm physically cannot reach, and this policy only
ever uses the right arm (see Known limitations). Restricted to
positions the arm can actually reach, the policy solves the task essentially perfectly
without domain randomization, and loses only ~7 points with it.
Efficiency
Metric
clear
random
Steps to success (median)
60
60
Inference calls per episode (median)
6
6
Inference latency (median)
0.387 s
0.397 s
Successful episodes are tightly clustered at 60 steps / 6 inference calls — the policy
either solves it in one smooth approach or not at all. There is no "retry until it works"
behaviour.
Training-step comparison
Same seeds, same configuration, three wrist+head camera streams recorded:
step 10000
step 19999
clear
13/20 = 65%
13/20 = 65%
random
10/20 = 50%
14/20 = 70%
Doubling training steps did not change clear at all, and improved random. A
per-episode breakdown shows exactly where the gain came from: failures of the type
"end-effector hovers 2–10 mm above the bell without pressing" dropped from 4 to 2, and
"pressed but not deep enough" dropped from 2 to 0. In other words the extra training
bought press precision under visual perturbation and nothing else.
Caveat: at n=20 these differences are not statistically separable — see
Reproducibility.
Usage
This is an Orbax checkpoint for openpi. It is not a transformers model and will not
load with AutoModel.
Images must be HWC uint8, not CHW. Norm stats are read automatically from
assets/<asset_id>/ inside the checkpoint directory, where asset_id comes from the
train config (RoboSynChallenge/cobotmagic_Sim_click_bell).
train_state/ (optimizer state, 31 GB) is not included — this checkpoint is for
inference and evaluation, not for resuming training.
Evaluate in RoboSynChallenge
bash
1# place under policy/pi05/checkpoints/<train_config>/<model_name>/19999/2bash policy/pi05/eval.sh click_bell random \3 pi05_base_robosynchallenge_full pi05_click_bell_baseline 0\4 --checkpoint_id 19999 --max_episodes 30 --headless true\5 --eval_video_obs_keys cam_high,cam_left_wrist,cam_right_wrist
The checkpoint directory must be named with the bare step number (19999) — the
adapter does int(checkpoint_id).
Task definition
Success is decided purely by the physical displacement of the button
(robosynchallenge/tasks/click_bell/click_bell.py):
python
1press_depth =-button_qpos[:,0]2success = press_depth >=0.0048# button joint travel is [-0.005, 0]3self._button_pressed |= success # latched for the episode
The threshold is 4.8 mm out of 5.0 mm of total travel (96%) — a light touch does not
count. Success is latched: one qualifying frame marks the episode successful.
clear vs random
Both settings randomize the bell position identically over
x∈[0.40, 0.85], y∈[−0.30, +0.30]. The difference is 10 additional perturbations present
only in random:
Perturbation
Applied
Light position / colour / intensity (10–30, a 3× range)
every 10 steps
Material of table, robot, floor, button (50% chance of random texture)
every 10 steps
Head camera intrinsics (±50 px focal) and extrinsics (±2 cm, ±10°)
The right-arm equivalent is marked # DONE. Worse, the collection loop
(scripts/run_env.py:_generate_function) silently resets and re-randomizes whenever
action generation fails, leaving no record:
python
1valid = generate_and_execute_action_list(...)2ifnot valid:3 _, _ = env.reset(options={"save_data":False})# try a different scene4break
So left-side scenes are systematically dropped from the dataset rather than sampled and
failed. The policy never sees a left-arm demonstration and cannot invent one. More
training will not fix this — the data has to be regenerated after the function is fixed.
2. Part of the default evaluation range is unreachable
An IK sweep of the right arm's press pose over a grid of bell positions gives a clean
diagonal boundary (O reachable, . not):
y=-0.30 -0.00 +0.30
x=0.40 O O O O O O O O O O O O O O O O O O O O .
x=0.50 O O O O O O O O O O O O O O O O O O O . .
x=0.60 O O O O O O O O O O O O O O O O . . . . .
x=0.70 O O O O O O O O O O O O O . . . . . . . .
x=0.80 O O O O O O O . . . . . . . . . . . . . .
x=0.84 . . . . . . . . . . . . . . . . . . . . .
Roughly 35% of the default sampling area is out of the right arm's workspace. Note
this is a property of the config shipped for data collection, not an official benchmark
specification — RoboSynChallenge's official ranking is on held-out physical robots, and
the repository ships no evaluation config of its own.
3. Failure breakdown
Of 16 failures over 40 instrumented episodes at step 10000:
Cause
Count
Whose problem
Hovers 2–10 mm above the bell, never contacts
9
the policy
Right arm cannot reach (all had y > +0.18)
5
task configuration
Pressed 4.3–4.7 mm, threshold is 4.8 mm
2
success criterion
Successes and failures separate almost perfectly by a single number — the closest
approach of the right end-effector to the bell:
Closest approach
24 successes
2.0 – 2.8 cm
16 failures
2.7 – 24.4 cm
Press depth is bimodal, not continuous: successes all bottom out at 5.00 mm, and 14 of 16
failures never leave the 0.95 mm resting value. The task is all-or-nothing.
4. Reproducibility
Because the threshold sits at 96% of total travel, marginal episodes flip between runs
from GPU floating-point non-determinism alone. With the same seed we observed episodes
going fail → success and success → fail across repeats, and random scored 60% / 45%
/ 50% / 70% across four 20-episode runs of the same configuration.
clear reproduced exactly (13/20 three times). Use ≥50 episodes when comparing
checkpoints, or the noise will exceed the effect.
Reproducing the evaluation
The upstream policy/pi05/ adapter is out of date relative to its own openpi copy and
to scripts/eval_policy.py; evaluation cannot run without these four fixes (all present
in policy/pi0/, which is current):
File
Problem
Fix
pi_model.py
passes robotwin_repo_id, which this openpi does not accept
drop it — openpi resolves norm stats from data_config.asset_id
pi_model.py
__init__ does not accept pytorch_device, but deploy_policy.py passes it
add the parameter and forward it
pi_model.py
builds an aloha-style observation ({"state", "images"}, CHW)
build flat observation/* keys in HWC, as EmbodiChainInputs expects
deploy_policy.py
eval() returns 2 values, eval_policy.py unpacks 3
also return truncated
One further trap: in scripts/eval_policy.py the episode loop's finally calls
env.close(), which terminates the process before Python prints the traceback, and
the exit code is 0. Every one of the bugs above therefore presented as "the run finished
normally but the robot never moved." Printing the exception inside the loop before
env.close() runs is what made them findable.
Citation
bibtex
1@misc{click_bell_pi05,
2 title = {click_bell: a pi0.5 policy for the RoboSynChallenge bell-pressing task},
3 author = {puheliang},
4 year = {2026},
5 url = {https://huggingface.co/puheliang/click_bell}
6}
Base model: pi05_base (Physical Intelligence).
Benchmark and training data: RoboSynChallenge (EDEM-AI).
