pi0-FAST Cleaning Fine-tune (LoRA)

Pipeline

1. YouTube egocentric cleaning videos → frame extraction
2. HaMeR (3D hand keypoints + MANO) → hand trajectory extraction
3. Franka IK retargeting (smoothed + velocity-clipped)
4. VLM labeling (action/object/contact)
5. LeRobot HDF5 dataset (361 episodes, 72k timesteps)
6. pi0-FAST LoRA fine-tuning

Training Config

• Base model: lerobot/pi0fast-base (2.9B params)
• LoRA: rank=32, alpha=64, targets=q/k/v/o_proj (13.3M trainable, 0.45%)
• Batch size: 4 × 8 grad_accum = 32 effective
• LR: 1e-4 cosine with 200 step warmup
• Dataset: 361 episodes, 72,311 timesteps
• Action: 7 Franka joints + 1 gripper (padded to 32)
• State: 3 EE position + 1 gripper (padded to 32)

Training Progress (Epoch 1, interrupted)

• Loss: 40.5 → 6.4 (16,600/18,077 steps, ~92% of epoch 1)
• Time: ~91 minutes on A100-40GB
• Peak GPU: 29.6GB

Dataset

• 361 episodes from 2 cleaning videos
• 224x224 images + smoothed Franka joint trajectories
• Language instructions from VLM labeling

Resume Training