π₀.₅ — SO-ARM101 Right Arm Pick-and-Place (v1, action-expert-only fine-tune)

TL;DR: A first iteration on a small (50 episode) dataset that demonstrates the full pick → grasp → transport → place behavior, with a notable emergent recovery from an out-of-distribution box orientation. It is not yet robust across the full workspace.

Demonstration videos

Clean pick-and-place

Emergent recovery from vertical → horizontal box orientation

Task

Model details

Training data

• observation.images.wrist — wrist-mounted camera on the right gripper (close-up egocentric view of the object).
• observation.images.top — overhead workspace camera.

Data-collection methodology

1. Spatial variation — the box was placed at different positions within the same reachable patch of the table across episodes (no fixed grid; varied by hand). Position variation was the dominant axis.
2. Orientation variation — the rectangular box was placed in both vertical (long axis up) and horizontal (long axis flat) orientations.
3. Asymmetric texture flip — the box has a black half and a white half; roughly half the demos had the white half on top, the other half had black on top, to discourage the model from anchoring on the texture pattern.
4. Deliberate failure-and-correction episodes (~15%) — about 7 of the 50 episodes contain an intentional near-miss or mis-grasp followed by an in-episode correction. This adds recovery behavior to the training distribution rather than only showing clean successes. (Inspired by the well-known finding that pure success-only imitation learning is brittle under distributional shift.)
5. Consistent home pose — every episode starts from approximately the same arm configuration (shoulder_pan ≈ -20°, shoulder_lift ≈ -105°, elbow_flex ≈ 98°, wrist_flex ≈ 76°, wrist_roll ≈ 3°, gripper closed).

Training hyperparameters

Evaluation observations

What it does well

• Complete pick-and-place trajectory. Reach → pre-grasp → close gripper → lift → transport → release → return-to-home, all from a single policy call chain. No scripted phases.
• Reasonable grasp accuracy within a "sweet spot" of the workspace — typically a band slightly right of center where teleoperation demos were most concentrated.
• One genuinely surprising emergent behavior (observed in a single run, documented for transparency, not claimed as a reproducible capability): the policy was presented with the box standing vertically, attempted to grasp, knocked the box over so it landed horizontally on the table, re-approached, grasped it in the new orientation, and successfully transported it to the cardboard box. The vertical → knocked → horizontal recovery was not present in the training data — vertical and horizontal box poses both appear in demos but as separate episodes, not within a single trajectory. This is consistent with the action expert having internalized the ~15 % failure-and-correction subdistribution as a general "re-approach if grasp fails" behavior rather than a per-instance template.

What it does poorly

• Lateral generalization is limited. Outside the "sweet spot," the gripper tends to land 3-10 cm off the box in the lateral direction. Empirically, the model only weakly conditions on the camera images for lateral object localization and falls back toward the dataset's mean grasp position. This is a well-known small-dataset failure mode for VLAs: with 50 episodes, the model learned a robust temporal policy (the motion shape of a pick-and-place) but did not get enough lateral visual variation to robustly ground "where the object actually is" from the camera feed.
• Calibration sensitivity. State observations enter the policy directly (and dominate the visual signal at small dataset sizes). A shoulder_pan motor zeroing drift of ~2.5° produces a ~1.5 cm physical lateral shift that compounds with the visual-grounding bias above. Operators should verify the right follower's calibration matches the recording session before evaluating.
• Gripper depth/closing precision. The model often stops the gripper ~3-5 cm short of full reach unless the action clamp is widened beyond the strict dataset extremes. (Concretely: the model frequently predicts elbow_flex ≈ 98-100° and shoulder_lift ≈ -113° while the dataset caps at 96.13° and -111.69° respectively. Clipping to the exact dataset range loses the last few cm of reach.)

Practical inference notes

• Use the trained task prompt verbatim: "Pick up the cube and place it into the bowl." Changing object names in the prompt does not improve behavior at this dataset size.
• Relax per-joint action clipping by ~10 % beyond the dataset's action.min / action.max to let the model's modest overshoot through. The per-step max_relative_target clamp in LeRobot still prevents jerky motion.
• num_inference_steps=10 (default for π₀.₅ flow-matching) is sufficient; higher values do not noticeably improve grounding here.
• Start the arm at the recorded home pose every rollout; the model is more sensitive to start-state drift than to camera framing drift.

Intended use

• Pedagogical / research reference for the LeRobot + π₀.₅ + SO-ARM101 fine-tuning workflow on a single arm.
• Sanity-test starting point for further fine-tuning on a larger, more laterally varied dataset (anticipated v2).

• Production / deployed manipulation.
• Tasks involving objects materially different in size, color, or shape from the demonstrated Feetech-style box.
• Workspaces with significantly different lighting, table material, or camera geometry than the recording session.

How to use

Run inference (LeRobot rollout)

Run remotely (Modal cloud GPU)

Reproduction

1. Record a 50-episode SO-ARM101 right-arm pick-and-place dataset matching the methodology above (varied position, orientation, texture flip, ~15 % failure+correction), at 15 fps with wrist and top camera keys.
2. Push as <your_user>/so101_right_arm_pick_cube_v1.
3. Run the lerobot-train command shown under Training hyperparameters, substituting your dataset and policy repo IDs.
4. ~1.5-2.5 h wallclock on a single A100 40 GB.

Limitations and bias

• Single-arm, single-task, single-object. No claim of generality.
• Small dataset (50 episodes). The pi0.5 paper and LeRobot community consensus is that pi0.5 typically needs 100-400 episodes for reliable visual grounding on a new task. Behavior here is therefore at the boundary of what a 50-episode fine-tune can produce.
• Geometric tie to one robot and camera setup. The policy implicitly encodes the recording-time arm and camera positions. Significant changes to camera mount geometry, arm base position, or motor calibration are expected to degrade performance.
• Task prompt is not meaningfully grounded. All 50 demos used the same language string, so the policy effectively does not condition on language — it conditions on state + images and the prompt is constant.

Acknowledgements

• Built on top of Physical Intelligence's open-sourced π₀.₅ via the LeRobot port.
• Trained and deployed using LeRobot and Modal.
• SO-ARM101 hardware design from The Robot Studio's SO-ARM project.

Citation