XLeRobot pi0.5 - bottle pick-and-place (sim)
pi0.5 LoRA fine-tune of pi05_base for the XLeRobot's left arm picking a Life Extension
Two-Per-Day bottle off a table and standing it on a randomized green pad, trained entirely in
ManiSkill. Two stages: a fixed-scene run (step-*) and a domain-randomized fine-tune on top of
it (dr2-step-*), which is the one to use.
- Configs:
pi05_xlerobot_orange (stage 1) and pi05_xlerobot_orange_dr (stage 2),
registered into openpi by experiments/vla_orange/openpi_xlerobot/install_openpi.py
- Model:
Pi0Config(pi05=True), gemma_2b_lora + gemma_300m_lora, action_horizon 10,
EMA off
- Training: batch 32, FSDP across 2x RTX 3090. Stage 1 stopped at 33k steps; stage 2 ran
12k steps from stage 1's weights (fresh optimizer state — the dataset and its norm stats
differ), peak LR 5e-5 cosine to 5e-6.
- Data: scripted-IK demos collected in ManiSkill, head + wrist cameras at 224x224.
Stage 1: 1024 episodes, fixed scene. Stage 2: 512 episodes with table height 0.760-0.860 m,
per-env lighting, background walls/floor/clutter, camera pose+fov jitter, and a rest pose
moved off the spawn region's sightline. Scripted success 88.9%.
- Prompt:
put the bottle on the green pad
Results
eval_sim.py, 32 episodes each, closed-loop through the same websocket policy server used for
deployment. The scripted policy that generated the data scores 88.9%.
| checkpoint | scene | episode cap | success | grasp | lift |
|---|
step-33000 (stage 1) | fixed | 280 | 71.9% | - | - |
step-33000 (stage 1) | domain-randomized | 280 | 0% | - | - |
dr2-step-7000 | domain-randomized | 280 | 34.4% | 90.6% | 71.9% |
dr2-step-11999 | domain-randomized | 280 | 56.2% | 84.4% | 84.4% |
dr2-step-11999 | domain-randomized | 400 | 71.9% | 84.4% | 78.1% |
dr2-step-11999 | DR, table pinned to 0.851 m | 280 | 50.0% | 90.6% | 75.0% |
dr2-step-11999 | DR, table pinned to 0.851 m | 400 | 68.8% | 93.8% | 75.0% |
The episode cap matters more than it looks. The scripted demos average 254 steps and the
env registers max_episode_steps=280, leaving 26 steps of slack; the learned policy is slower
than the script, so raising the cap to 400 converts most "failures" into successes — the
lifted-but-not-placed rate falls from 28% to 9%. Budget at least 400 steps at deployment
(more on hardware, where deploy_client.py's per-step clamp slows the arm further).
Motion profile. The commanded per-step joint deltas are bursty rather than uniformly fast:
median 0.010 rad but p99 0.25 rad, with 3–6% of steps exceeding deploy_client's ~0.067 rad
clamp. The bursts are not chunk-boundary artifacts — jump statistics are flat across the
position within the action chunk. An RTC (real-time chunking, arXiv:2506.07339) guided sampler
is available for these checkpoints (experiments/vla_orange/openpi_xlerobot/pi0_rtc.py,
installed into openpi by install_openpi.py); serve as usual and send optional
prev_actions / inference_delay / rtc_guidance_weight request keys to activate it.
In zero-latency sim it neither helps nor hurts (the splice discontinuity it repairs only
exists under real inference latency); it costs ~65% extra inference time (~270 ms vs ~165 ms
on an RTX 3090). Note the short action horizon (10 steps) bounds how much latency RTC can
hide: at 30 Hz control the chunk spans 333 ms, so plan for a lower control rate or a longer
retrained horizon if end-to-end latency approaches that.
Checkpoints
<name>/ is an orbax checkpoint directory exactly as openpi writes it:
params/ - weights, enough to serve the policytrain_state/ - optimizer state; needed to --resume training from this point
1uv run scripts/serve_policy.py policy:checkpoint \
2 --policy.config=pi05_xlerobot_orange_dr \
3 --policy.dir=<path>/dr2-step-11999
dr2-step-11999 is the final stage-2 checkpoint: num_train_steps=12000 counts steps
0..11999, so 11999 is the last one written — there is no dr2-step-12000. Serve dr2-* under
pi05_xlerobot_orange_dr and step-* under pi05_xlerobot_orange; the two have different
norm stats and swapping them silently denormalizes actions with the wrong statistics.
Evaluating a DR checkpoint: do not use --no-dr
eval_sim.py --no-dr (and safety_probe.py, which hardcodes the same thing) turns off two
things at once: the visual randomization AND the arm's start posture, because
PickOrangeEnv._rest_for_height returns the legacy Q_REST verbatim when randomization is
off. Every stage-2 episode starts from a different rest pose, so a dr2-* checkpoint measured
that way is being started from a posture it has never seen — grasp rate collapses from 91% to
9% on that alone. The flag is correct for the stage-1 step-* checkpoints, which is what it
was written for. To pin one scene parameter for a stage-2 checkpoint, keep
domain_randomization=True and constrain that parameter instead (e.g.
table_z_range=(0.851, 0.851)).
Observation and action spaces
| key | shape | meaning |
|---|
observation/image | 224x224x3 uint8 | head camera |
observation/wrist_image | 224x224x3 uint8 | acting (left) arm's wrist camera |
observation/state | 6 float32 | [Rotation, Pitch, Elbow, Wrist_Pitch, Wrist_Roll, Jaw], absolute radians |
actions | 10x6 float32 | same 6-D vector, absolute radians, action horizon 10 |
The robot is an
XLeRobot 0.4.0: two SO-101 arms on
a trolley whose top is flush with the table at 0.775 m. Only the left arm acts; the right arm
is frozen in frame, and the base and head are frozen.
Known limits
- Sim-only. No real robot data went into training, and no hardware run has completed the
task. On the committed real captures the policy does respond to real frames (its action
separates from the blank-image action by 0.28), but that response is about 26% of the
steering range it shows on sim frames — real images are read, less sharply than sim ones.
- Commanded motion is faster than deployment allows. Per-step joint jumps reach 0.40 rad
against
deploy_client.py's ~0.067 rad clamp, so the clamp engages constantly and the arm
lags the intended trajectory.
- A bare scene is out of distribution. The randomization covers the nominal value of every
continuous parameter, but the structural elements are always on: back wall, floor overlay,
exactly 4 clutter pieces, shadows. An empty fixed scene scores well below the randomized one.
- Height safety is clear: at the measured bench height (0.851 m) none of 1120 commanded actions
put the jaw tip below the tabletop, so the floor guard never engages.
Licensing
These are LoRA fine-tunes of
pi05_base, which is built on PaliGemma and a Gemma action
expert, so the
weights are governed by the Gemma Terms of Use
(openpi ships
LICENSE_GEMMA.txt alongside its Apache-2.0 code for exactly this reason).
openpi itself is Apache-2.0.
