Base: pi0-fast-piper-h-plate-pick / checkpoints/main/4000 (Phase 1 — 311 teleop + 103 ego episodes on generic manipulation tasks)
Fine-tune: 121 plate-pick episodes collected on the customer's robot (ppv4 + ppv5 setup — updated visual alignment and arm position)
Architecture: PaliGemma 2B VLM backbone + SigLIP ViT-So400m vision encoder + FAST DCT/BPE action tokenizer; ~50 M trainable LoRA parameters
Action: 7-DoF — [j1..j6 radians, gripper in [0,1]] for the right arm only
Chunk size: 10 steps at 20 Hz = 500 ms of motion per inference call

Why we retrained (ft-v2 vs ft-v1)

ft-v1 (ckpt-500, 08062026_single_plate_v10, 52 episodes) failed in deployment because the customer's robot setup changed between data collection and deployment:

What changed	Impact
Camera mount / visual alignment	SigLIP embeddings shifted — model "sees" a different scene
Robot arm home position	State distribution mismatch — model starts in unknown territory

ft-v2 retrains from main/4000 on 121 episodes from the new setup (PranayTest/piper-ppv4-ppv5), fully covering the updated visual and kinematic environment.

Training details

Item	Value
Base checkpoint	`main/4000`
Dataset	`PranayTest/piper-ppv4-ppv5`, episodes 006–126
Train episodes	111
Val episodes	10 (indices 9, 30, 72, 76, 84, 85, 88, 90, 97, 115)
Total frames	48,043 at 20 Hz
Batch size	32
Peak LR	5e-6 (cosine, 200-step warmup)
GPU	A100-SXM4-80GB
Steps run	2,500 (stopped early — val rose after step 2000)
Best step	2000 · val loss 0.878
Val evals	Every 500 steps, 50 batches

Loss curve

Step	Train loss	Val loss	Note
0	3.713	—
100	2.511	—
200	1.296	—
300	1.008	—
400	0.896	—
500	0.825	1.009
600	0.811	—
700	0.766	—
800	0.754	—
900	0.724	—
1000	0.728	0.911
1100	0.723	—
1200	0.698	—
1300	0.698	—
1400	0.683	—
1500	0.671	0.904
1600	0.674	—
1700	0.642	—
1800	0.643	—
1900	0.666	—
2000	0.632	0.878	best — deployed
2100	0.640	—
2200	0.632	—
2300	0.611	—
2400	0.610	—
2500	0.597	0.891	val rose → early stop

Early stop rule: val rose at the next eval after best → stopped at 2500, deployed step 2000.

Dataset details (ppv4ppv5)

Raw format: per-episode directories with front.mp4, right.mp4, meta.json, telemetry.jsonl. Converted to LeRobot v2.1 by scripts/ppv4ppv5_to_lerobot.py.

Field	Value
Source	`PranayTest/piper-ppv4-ppv5` (HF, private)
Episodes used	006–126 (000–005 skipped: test runs / no plate)
Joint units in raw	Degrees — converted x pi/180
Telemetry rate	~17.83 Hz — resampled to 20 Hz via linear interpolation
Video	30 fps — subsampled to 20 Hz
Camera resolution	640x480 landscape (both front + right wrist)
Task prompt	`"pick up the plate"`
Home pose mean (rad)	`[1.252, 2.198, -1.865, 1.192, 1.146, -1.113, 0.985]`
Home pose spread	< 4 deg on any joint across all 121 episodes

Home pose

The robot must start from this pose before each inference run:

[1.252, 2.198, -1.865, 1.192, 1.146, -1.113, 0.985]
  j1     j2     j3      j4     j5     j6    gripper

All in radians. Gripper 0.985 = open (1.0 = fully open, 0.0 = closed). Starting significantly outside this range degrades performance — all 121 training episodes began here.

Inference

The model is served as a FastAPI server. See scripts/serve_api.py.

python

import requests, base64, io, numpy as np
from PIL import Image

URL = "http://<host>:<port>/predict?token=<VAST_TOKEN>"
API_TOKEN = "<API_TOKEN>"

def encode(arr):
    buf = io.BytesIO()
    Image.fromarray(arr).save(buf, format="PNG")
    return base64.b64encode(buf.getvalue()).decode()

r = requests.post(URL,
    json={
        "front_image_b64": encode(front_rgb),   # any resolution HxWx3 uint8 RGB
        "right_image_b64": encode(right_rgb),   # server resizes to 640x480
        "state": list(map(float, state_7)),      # current joints + gripper
        "prompt": "pick up the plate"
    },
    headers={"Authorization": f"Bearer {API_TOKEN}"},
    timeout=30)
actions = np.array(r.json()["actions"])  # shape (10, 7) float32

Execution: apply each row at 20 Hz (50 ms per step). Pipelining: keep 4–8 requests in flight — inference is ~0.8 s but each chunk covers 0.5 s, so naive single-call loop gives < 15% duty cycle.

Repository structure

checkpoints/
  ppv4ppv5_run1/
    2000/         <- deployed (val loss 0.878)
      params/     <- Orbax checkpoint (~8.2 GB bfloat16)
      assets/     <- norm_stats.json
    500/          <- val 1.009
    1000/         <- val 0.911
    1500/         <- val 0.904
    2500/         <- val 0.891
scripts/
  serve_api.py              <- FastAPI inference server (logging, auth, image resize)
  ppv4ppv5_to_lerobot.py    <- dataset converter: raw ppv4ppv5 -> LeRobot v2.1
  config.py                 <- openpi TrainConfig (piper_h_ppv4ppv5_ft)
  data_loader.py            <- openpi data loader (patched)
  piper_policy.py           <- PiperInputs / PiperOutputs transforms
  train.py                  <- openpi training script (patched)
loss_curve.png              <- training loss plot

Phase history

Phase	What	Outcome
Phase 1	Train `main/4000` from scratch — 311 teleop + 103 ego episodes, generic Piper-H manipulation	`main/4000` base
Phase 2	Fine-tune from `main/4000` — 52 customer episodes (`08062026_single_plate_v10`)	`ft-v1 / ckpt-500`
Phase 3	Deploy ft-v1 on Vast.ai	Live at old URL — model failed (env changed)
Phase 4	Retrain from `main/4000` — 121 episodes from new setup (ppv4+ppv5, updated camera + arm position)	ft-v2 / step-2000 · val 0.878

ft-v1 failed because the robot setup changed (camera alignment + arm home pose) between collection and deployment. ft-v2 was collected and trained entirely on the updated setup.

Contact

Model trained by Kavin (kavinrajk.r.2005@gmail.com). Dataset collected by the customer on AgileX Piper-H hardware.

Kavin60606/pi0-fast-piper-h-plate-pick-ft-v2

π0-FAST · Piper-H Plate-Pick · ft-v2

What this is