cup_pnp — yellow-cup pick-and-place policies
Four behavior-cloning policies for the task "pick up a yellow cup and place it
on a baking tray," trained from egocentric human video reconstructed into
robot demonstrations. This is a 2×2 ablation:
| architecture × observation | current (raw tray) | sim-tray |
|---|
π0.5 (LoRA on pi05_base) | pi05_current_tray/ | pi05_sim_tray/ |
| ABC-DiT (from scratch) | abc_dit_current_tray.pt | abc_dit_sim_tray.pt |
- Action / state space: 28-D — right arm only (3-slide + 3-hinge wrist + 22
finger joints), absolute joint targets.
- Observation: single rendered robot-view RGB (224×224).
- sim-tray variant re-renders the tray from its simulation mesh (inpainted
over the original footage); the current variant keeps the tray as raw
footage. Actions are identical across the two variants — only the observation
pixels differ.
Results — held-out action-prediction MSE (normalized, 10-step horizon; lower is better)
| policy | current (raw tray) | sim-tray |
|---|
| π0.5 | 0.021 | 0.021 |
| ABC-DiT | 0.345 | 0.318 |
π0.5 (pretrained VLA + LoRA) is ~16× more accurate than ABC-DiT (trained from
scratch with a randomly-initialized visual backbone). The sim-tray processing is
neutral for π0.5 and a small improvement for ABC-DiT.
Contents
pi05_current_tray/, pi05_sim_tray/ — π0.5 checkpoints (orbax params/ +
assets/ norm-stats). Load with openpi
policy_config.create_trained_policy. Config: pi05_cup_v1 / pi05_cup_v2.abc_dit_current_tray.pt, abc_dit_sim_tray.pt — ABC-DiT checkpoints
({model, step, norm_stats} + optimizer). Load with
ABC. state/action dim = 28, camera-keys = top.
Trained on 80 verified reconstructed demonstrations. See the project report for
data provenance and method details.
