LoRA fine-tune of pi0.5 (pi05_base) for a simulated household manipulation task: wipe a dusty container clean with a sponge, then transfer a potato into it. Trained with the ManiGuard pipeline (NU IDEAS Lab) on top of a pristine openpi trainer.

This is the JointController track: collection → render → SFT → eval are all joint-space (no end-effector / IK anywhere).

Tasks & prompts

The instruction is pulled per-episode from the dataset's task field (prompt_from_task=True). Three variants:

"Wipe the dusty stockpot clean with the sponge, then transfer the potato from the chopping board into the stockpot."
"Wipe the dusty mixing bowl clean with the sponge, then transfer the potato from the platter into the mixing bowl."
"Wipe the dusty saucepan clean with the sponge, then transfer the potato from the platter into the saucepan."

Model

Base: pi0.5 — gs://openpi-assets/checkpoints/pi05_base/params (warm-start).
LoRA: PaliGemma gemma_2b_lora + action expert gemma_300m_lora; base frozen, only the LoRA adapters trained (openpi freeze_filter).
action_dim=32 (padded), action_horizon=16, pi05=True, discrete_state_input default (True).
Controller — JointController: 8-D joint state + 8-D absolute-joint action. The 7 arm joints are consumed as per-step deltas (gripper kept absolute) and reconstructed to absolute joint targets at inference, fed straight to a JointController — no eef→joint IK.
Cameras (LIBERO 2-cam): base_0_rgb ← image_left (third-person overview), left_wrist_0_rgb ← wrist_image; the third slot (right_wrist_0_rgb) is zero-filled and masked off.

Training data

IDEAS-Lab-Northwestern/sim-dusty-transfer-30-joint-3cam (LeRobot v2.1, private): 30 episodes / 20,265 frames, sim teleop rendered joint + 3-cam (image_left / image_right / wrist_image); consumed 2-cam (image_right dropped). Episode split across the 3 task variants: 12 / 10 / 8.

Training setup


Hardware	4× RTX A5000 (24 GB), FSDP (`fsdp_devices=4`)
Precision	float32 (see note)
Global batch	12 (3 / GPU)
Steps	3,380 (~2 epochs)
LR schedule	cosine, warmup 340, peak 3e-5, decay → 3e-6
Optimizer	AdamW (b1 0.9, b2 0.95), global-norm grad clip 1.0, EMA off
Wall-clock	~6.75 h; final train loss ~0.014 (from ~0.069), no divergence

Note: why float32 (and when bf16 is fine)

On 24 GB cards the full model does not fit replicated, so training uses FSDP parameter sharding. In bf16, FSDP's gather/scatter was numerically unstable on this hardware and diverged to NaN within ~25–100 steps — the same config trains fine in bf16 on 80 GB A100/H100 in pure data-parallel (no sharding). Setting the model dtype="float32" removed the instability. If you fine-tune on larger-memory GPUs (data-parallel, no FSDP), bf16 is fine and faster.

Intended use & limitations

Research artifact for sim joint-control evaluation in the ManiGuard pipeline. Success-rate eval is a separate effort and is not reported here.
Sim-only; not validated on physical hardware.
Small dataset (30 episodes, single object/scene family) — expect limited generalization beyond the trained variants.

License & provenance

Base weights are Gemma-derived (pi0.5); use is governed by the Gemma terms. LoRA deltas released for research. Trained via openpi's JAX trainer, configured by ManiGuard's maniguard.openpi_sft module (openpi itself unmodified).

IDEAS-Lab-Northwestern/pi05-base-dusty-transfer-joint-2cam-lora

pi0.5 LoRA — Dusty-Transfer (JointController, 2-cam)