A π0.5 (pi05) vision-language-action policy fine-tuned on an SO-101 arm for a two-task cup manipulation setup: pick a cup up from the table and place it onto a blue circle.

Only the action expert was trained. The PaliGemma VLM (2.923 B params) is frozen (train_expert_only=true) — see configuration_pi05.py: "Freeze entire VLM, train only action expert and projections".

Base model: lerobot/pi05_base — the last pre-relative-actions revision, matching LeRobot 0.4.x
Dataset: aailabkaist/so101_recovery_2task (749 episodes · 151,255 frames · 30 fps · 8 operators)
Checkpoint: step 10,000 of a 10,000-step run (batch 64 → 640,000 samples ≈ 4.2 epochs)
Framework: LeRobot
Robot: SO-101, 6-DoF, two RGB cameras (front, wrist, 480×640)
Siblings: expert 5K · full 5K · full 10K — see the two-task collection

Task and data

Two instructions, both used at training time verbatim:

pick up the cup near the blue circle
place the cup on the blue circle

Pick-episode cup start positions (394 episodes, per-operator colors, anonymized). The cups form a ring around the blue circle rather than covering the workspace uniformly.

Place-episode transport vectors, start → placed (350 of 355 place episodes — 5 excluded where the cup is fully occluded by the gripper in the start frame). Endpoints converge tightly on the blue circle; the release target is nearly constant across the dataset.

Training


Trainable parameters	0.693 B (19.2 %)
Frozen	PaliGemma VLM, 2.923 B
Steps	10,000 (run stopped early at 10K; config said 20K)
Batch size	64
Precision	bfloat16, `gradient_checkpointing=true`
Optimizer	AdamW · lr 2.5e-5 · wd 0.01 · betas (0.9, 0.95) · grad-clip 1.0
Schedule	cosine decay with warmup — auto-scaled by LeRobot (20K steps < 30K decay): warmup 1,000→666, decay 30,000→20,000; peak 2.5e-5, LR at this checkpoint ≈1.4e-5 (configured floor 2.5e-6 never reached — run stopped at 10K)
Chunk	`chunk_size=50`, `n_action_steps=50`, `n_obs_steps=1`
Seed	1000
Hardware	1× NVIDIA RTX PRO 6000 Blackwell (96 GB)
VRAM / speed	28.2 GB · 3.93 s/step
Final loss at this checkpoint	0.087

Interactive training curves for this run — loss · LR · grad-norm · epochs (this variant's W&B run · project overview):

Both runs used identical data, batch size and dtype — the only difference is what was trainable, so the two curves are directly comparable. expert-only is essentially flat after warmup (~step 1K): 0.129 → 0.087 over the remaining ~3.8 epochs; full fine-tuning keeps descending. The run was stopped at 10K rather than the configured 20K on the strength of these curves.

Usage

python

from lerobot.policies.pi05.modeling_pi05 import PI05Policy

policy = PI05Policy.from_pretrained("aailabkaist/pi05_recovery_2task_expert_10k")

Or serve it for async inference:

bash

python -m lerobot.async_inference.policy_server --host=0.0.0.0 --port=8080 --fps=30
python -m lerobot.async_inference.robot_client \
    --server_address=<host>:8080 \
    --policy_type=pi05 --pretrained_name_or_path=aailabkaist/pi05_recovery_2task_expert_10k \
    --task="pick up the cup near the blue circle" \
    --actions_per_chunk=50 --chunk_size_threshold=0.2 \
    --aggregate_fn_name=weighted_average --fps=30

A single-task variant of this setup — one instruction covering the whole pick-and-place motion, with a plain white cup — lives in the SO-101 · cup → blue circle (single-task) collection.