π0.5 · Bimanual YAM · ultrawide + top camera · "Put all oranges in the bowl"
Full fine-tune of
lerobot/pi05_base on
brandonyang/yam-ultrawide-teleop — 75 episodes / 108,726 frames of bimanual YAM VR
teleoperation with
UMI-style ultrawide wrist cameras (312×416 @ 30 fps) plus the top camera (480×270). Task string (exact):
Put all oranges in the bowl.
Sibling models, same recipe (12,000 steps, global batch 64, seed 1000, fp32):
pi05-BimanualYAM-oranges (RealSense 3-cam) ·
pi05-BimanualYAM-oranges-uw-notop (wrist-only, in-the-wild variant).
Interface
- Action: 30-step chunks of 14-D absolute joint positions
[left_joint_0..5, left_gripper, right_joint_0..5, right_gripper] at 30 Hz; grippers [0,1].
- Observation: 3 RGB cameras + 14-D state (quantile-normalized, discretized into the π0.5 prompt) + task string.
- Camera mapping ships in the saved processors:
observation.images.{top,left,right} → {base_0_rgb, left_wrist_0_rgb, right_wrist_0_rgb} — feed raw
dataset-style keys. Corrected normalization stats embedded (the hub dataset's stock q01/q99 carry
lerobot#4156 bias up to ~50% of the quantile range; recomputed globally here).
Training
Identical recipe to the siblings (see the RealSense card for the full table): fp32 weights + fp32 AdamW, cosine 2.5e-5→2.5e-6, warmup 600,
quantile normalization, narrowed color jitter + affine, gradient checkpointing, 8×RTX PRO 6000. 12,000 steps ≈ 7.06 epochs.
Offline eval (final checkpoint, step 12000)
Open-loop 30-step chunk error on training episodes [3, 17, 41, 62, 74], stride 60. All-in-sample; gates brokenness, does not predict task success.
| Metric | this model | uw-notop (wrist-only) | RealSense 3-cam | MolmoAct2 run 2 |
|---|
| Joint L1 (rad) | 0.0093 | 0.0092 | 0.0071 | 0.0515 |
| Gripper L1 | 0.0040 | 0.0039 | 0.0029 | 0.0036 |
| Shuffled-camera L1 ratio | 8.1× | 8.1× | 5.9× | 1.53× |
| Non-finite | 0 | 0 | 0 | 0 |
Offline probes (grasp-event timing, vision/state ablations) show this model and the wrist-only sibling are near-identical — the top camera adds little
measurable value on this dataset. Small boundary overshoots (~1.5k scalars, worst ≈0.02 rad) occur where demonstrations sit exactly on joint/gripper
limits; rely on client-side clamping.
Deployment notes
- Cameras named exactly
top, left, right. Task string exactly Put all oranges in the bowl. --robot.max_gripper_delta=0.05.
- Local
lerobot-rollout: pass --rename_map='{"observation.images.top":"observation.images.base_0_rgb","observation.images.left":"observation.images.left_wrist_0_rgb","observation.images.right":"observation.images.right_wrist_0_rgb"}'.
- Remote policy server: requires the pi05 serving patch (see the RealSense sibling card).
config.json ships pretrained_path: null intentionally.
- fp32 inference ≈16GB weights → ≥24GB GPU; bf16 autocast at inference is safe.
Training curves:
wandb aq-robotics/molmoact2-yam-oranges, run
pi05-uw-top-12k.
