Psi0 Fine-Tune v3 — G1 Sneaker-in-Box, Bimanual + Lateral + Recovery
v3 community fine-tune of Psi0 (paper) on a Unitree G1 humanoid with Inspire RH56DFTP hands.
Same task as the
v1 and
v2 fine-tunes —
"Pick up the sneaker and place it inside the box." — but trained on the
325-episode v3 dataset (
psi0-g1-sneaker-325ep-v3) which extends the v2 bimanual base with
+60 lateral and
+60 recovery episodes. v3 directly addresses two of the four priorities surfaced by v2's open-loop eval: (1) RIGHT-side scene diversity (lateral coverage on both sides), (2) recovery / re-grasp episodes (carry-over from v1 Phase 14). Trained at the
CloudWalk Robotics Lab (CW-RL), 2026-05.
The third v2 priority (held-out OOD probe in the inner training loop) is a code-side change, not addressed here. The fourth (more episodes — done) is partially addressed: v3 = 1.6× v2 episodes, 1.8× frames.
Status
Post-train (2026-05-09). Training completed in 14 h 46 m 14 s on 8× B200 (CW-RL W&B run
mhh02t1w, 160 000 steps, 183 effective epochs). Headline data-level result:
eval/err_l1_arm_joints = 0.081 rad — 27% better than v2's 0.111 rad;
eval/err_l1_hand_joints = 0.0134 rad — flat vs v2 (0.0135). Aggregate
eval/loss = 0.267 is ~3.3× v2's 0.081, consistent with the broader v3 distribution (lateral / recovery classes are intrinsically harder per-frame than v2's standard-placement-only). Open-loop ID/OOD eval remains a follow-up — see the
Open-loop evaluation section.
Quick facts
| |
|---|
| Base | Psi0 (Qwen3-VL-2B + flow-based MM-DiT action expert) |
| Pretrained | pre.fast.1by1.2601091803.ckpt.ego200k.he30k + postpre.1by1.pad36.2601131206.ckpt.he30k |
| Training dataset | cloudwalk-research/psi0-g1-sneaker-325ep-v3 (325 ep, 111 443 frames @ 30 Hz, 640×480 ego cam) |
| Episode composition | 94 v1-RIGHT + 111 v2-LEFT + 60 lateral + 60 recovery |
| Canonical source | cloudwalk-research/psi0-g1-sneaker-205ep-v2-source (the v2 bimanual base) |
| Robot target | Unitree G1 + Inspire RH56DFTP (NOT Dex3-1) |
| Action space | 36-D (6 hand + 1 pad / side, 14 arm, 4 torso, 4 locomotion) |
| State space | 32-D (padded to 36-D for the model) |
| Action chunk | 30 |
| Hardware | 8× NVIDIA B200 (sm_100 / Blackwell), system PyTorch 2.10 (NVIDIA build) |
| Mixed precision | bf16 |
| Optimizer | AdamW, lr=1e-4 cosine + min_lr 5e-7, warmup 1000, betas=(0.95, 0.999), wd=1e-6, grad-clip 1.0 |
| Steps / batch | 160 000 / global batch 128 (16 per GPU × 8 GPUs) |
| Effective epochs | ~184 (160 000 × 128 / 111 443) |
| Checkpointing | every 10 000 steps; max_checkpoints_to_keep=20 so all 16 saves (ckpt_10000 … ckpt_160000) survive |
| RTC | enabled, max_delay = 8 |
| VLM | frozen (only action header trained) |
| Wall-clock | 14 h 46 m 14 s (8× B200, prediction was ~14.7 h ✅) |
| Final eval/loss | 0.267 (predicted 0.06–0.12 ✗ — see "On the eval/loss surprise" below) |
| W&B run | mhh02t1w (CW-RL internal) |
Step-budget rationale (160k vs v2's 40k)
v2 found the OOD optimum at ~30 k (~63 effective epochs) and overfit past it. v3 jumps to 160 k (~184 effective epochs) on substantially more diverse data. The bet: lateral + recovery coverage broadens the training distribution enough that the OOD optimum no longer pegs at ~30 k. The 16-checkpoint ladder (every 10 k from 10 k to 160 k) is exactly the diagnostic input needed to locate the new optimum post-hoc — without re-training. If the v3 OOD curve still peaks before ~80 k, the dataset wasn't enough; if it peaks past 120 k, retain even more checkpoints in v4 and beyond.
Repository contents
This repo carries 3 diagnostic checkpoints (storage-cap-aware decision — see note below):
checkpoints/ckpt_30000/model.safetensors (~5.9 GB) — early plateau, post-warmupcheckpoints/ckpt_90000/model.safetensors (~5.9 GB) — mid-training referencecheckpoints/ckpt_160000/model.safetensors (~5.9 GB) — finalrun_config.json (tyro-serialized launch config)argv.txt (full command line)README.md (this file)
Optimizer / scheduler / sampler / random-state files are intentionally omitted — training-resume only.
Storage-cap-aware retention. v3 was trained with --train.max_checkpoints_to_keep=20 so all 16 every-10k saves (ckpt_10000 … ckpt_160000) existed on the cluster NFS at end-of-training. The first HF upload attempt with all 16 ckpts (~95 GB raw / ~31 GB net after dedup) hit the cloudwalk-research private-storage cap. Reduced HF retention to the 3-checkpoint diagnostic sweep above. The intermediate ckpts (10k / 20k / 40k / 50k / 60k / 70k / 80k / 100k / 110k / 120k / 130k / 140k / 150k) for finer-grained Phase 11.5 follow-up evaluation may still be available internally at CW-RL on request.
Evaluation
In-process (training-time eval/loss, in-distribution)
Validation runs on a fresh-batch sample of the training repo (Psi0 standard practice — val_repo_ids defaults to train_repo_ids). Treat training-time eval/loss as a smoothed train-loss probe, not a generalization signal. The real generalization signal lives in the open-loop section below; both v3 and v4 reserve that as a follow-up session, with the 16-checkpoint ladder retained as the diagnostic input.
| Metric | Step 0 | Step 160k (v3) | v2 reference (3yifcdgs, 40k) | v1 reference (hjy0caz0, 40k) |
|---|
eval/loss | ~38 | 0.267 | 0.081 | 0.0675 |
eval/err_l1_arm_joints | ~3.6 | 0.081 rad | 0.111 rad | 0.058 rad |
eval/err_l1_hand_joints | ~1.4 | 0.0134 rad | 0.0135 rad | 0.012 rad |
eval/err_l1_height | ~5.0e-3 | 1.0e-5 m | 1.0e-5 m | 1.0e-5 m |
eval/err_l1_target_yaw | n/a | 6.9e-4 | 0 | 0 |
eval/err_l1_{vx, vy, vyaw, torso_rpy} | 0 | 0 | 0 | 0 |
On the headline numbers
Arm down 27%, hand flat. v3's eval/err_l1_arm_joints (0.081 rad) is the lowest of the three rounds at the same dimension and substantially closer to v1's single-side baseline (0.058 rad), despite v3's training distribution being the most diverse of the three. Hand error is unchanged from v2 — bimanual coverage absorbed cleanly across the 4× bigger step budget without per-active-side regression. These two facts together are the data-level evidence that the v3 dataset interventions (lateral + recovery) paid off: the lateral coverage broadened the arm-trajectory distribution the model fits cleanly, and the recovery class didn't collapse the hand-side fidelity it inherits from v2.
On the eval/loss surprise
eval/loss = 0.267 is ~3.3× higher than v2's 0.081, but the per-group L1 errors above are equal-or-better. Reading: eval/loss is an averaged-over-distribution metric, so the headline value reflects the harder average frame in v3 (lateral has wider arm trajectory variance; recovery contains discontinuities at re-grasp moments, which are harder to fit per-frame than smooth pickups). The per-group L1 numbers strip out that distribution-mix effect. Cross-check: at v2's 40 k mark in this v3 curve, eval/loss was already in v2's family — the difference is structural to the v3 distribution, not to v3's late-training overfitting.
Caveat to flag in any cross-version comparison going forward: v3 and v4's aggregate eval/loss are not directly comparable to v2's at face value — use the per-group L1 errors when comparing across rounds, or carve a comparable held-out subset.
Per-checkpoint ladder retained
This repo carries 3 diagnostic checkpoints on HF (ckpt_30000 / ckpt_90000 / ckpt_160000), per the storage-cap-aware decision above. The wider 16-checkpoint local ladder may also be available on the cluster NFS for follow-up eval — see "Repository contents" above.
Open-loop evaluation (ID + OOD)
Pending — to be done in a follow-up session, mirror of v2 Phase 11.5. The v2-era wiring (openloop_eval_psi0.py + --eval-dataset-id + --episode-indices) carries over without modification. Two preconditions for v3:
- A held-out v3 eval split (analogous to
psi0-g1-sneaker-6ep-v2-eval) — not built yet. Either reuse the v2 eval set as a strict OOD probe (its episodes are not in v3 by orig_episode_index), or carve a fresh held-out set from the lateral / recovery additions (better — exercises v3-specific scene classes).
- Decision on which ckpts to evaluate. Recommended sweep: 30 k, 60 k, 90 k, 120 k, 160 k (5-point pass over the 16 retained). If the curve is non-monotonic, fill in 1-2 more between adjacent steps.
The 16-checkpoint ladder this repo retains is the diagnostic input.
How to download
1import os
2from huggingface_hub import hf_hub_download
3
4repo = "cloudwalk-research/psi0-g1-sneaker-325ep-v3-finetune"
5local_dir = ".runs/finetune/sneaker-real-v3"
6for f in [
7 "run_config.json",
8 "argv.txt",
9 "checkpoints/ckpt_160000/model.safetensors", # one of {ckpt_30000, ckpt_90000, ckpt_160000}
10]:
11 hf_hub_download(repo_id=repo, filename=f, repo_type="model",
12 local_dir=local_dir)
How to evaluate (closed-loop sim)
1bash sim/scripts/run_closed_loop.sh \
2 "Pick up the sneaker and place it inside the box." \
3 .runs/finetune/sneaker-real-v3 160000 \
4 --fixed_base --live_viewer
For v3, watch (on top of v2's bimanual signal): does the policy attempt to re-grasp when the first attempt fails? That is the v3-specific dataset class (recovery, 60 ep) that should now manifest in behavior.
Known caveats
- Generalization is not yet validated. This card reports the recipe and the (expected) data-level signals. Whether v3 closes the v2 OOD/ID gap requires the open-loop eval below; until that runs, treat v3's improvements over v2 as a hypothesis with structural support, not a confirmed result.
- Hand commands are still lockstep across the 6 active Inspire DoF (per side). The teleop dataset was collected with a Pico 4 Ultra trigger driving all 6 DoF on a single grip-strength scalar; the model can only output what it saw. v3 keeps this property; per-finger dexterous control would require a different teleop modality.
- Task is bimanual but stationary. Action dims for locomotion (vx/vy/vyaw/target_yaw) and torso RPY are constant in the dataset; the model produces those constants at inference. Do not expect locomotion behavior.
- In-process
eval/loss is in-distribution. Psi0 LerobotDataConfig defaults val_repo_ids to train_repo_ids. The real generalization signal lives in the open-loop section above.
- Inspire deploy bridge is required (resolved in v1, inherited in v2/v3).
real/teleop/master_whole_body.py:206 of the Psi0 deploy stack hard-codes Dex3_1_Controller; the Inspire RH56DFTP listens on rt/inspire/cmd. The Inspire_VLA_Controller bridge implemented during v1 Phase 14 is the prerequisite.
- Recovery scenes are short (60 of 325 ep) and lateral scenes likewise. Frame-weighted exposure of these new scene classes is roughly proportional to their episode count (mean episode length is similar across collections), so ~18% by frame for lateral and ~18% for recovery. If real-robot behavior shows v3 reverting to v2 patterns under stress, the new scene classes may need up-sampling at training time in v4 / v5.
Lineage
References
- Psi0 — Generalist VLA model, base policy fine-tuned here. Paper, GitHub.
- Cluster runbooks (v2 and v3) — Container reuse strategy, 160 k step budget, 16-ckpt retention; tracked internally at CW-RL.
- v2 fine-tune —
cloudwalk-research/psi0-g1-sneaker-199ep-v2-finetune; the round v3 follows up on.
- v1 fine-tune —
cloudwalk-research/psi0-g1-sneaker-94ep-v1-finetune; first publicly reported external reproduction of the Psi0 pipeline.
- Pre-deploy checklist — Psi0 issue #39.
Attribution
Developed by
cloudwalk-research in the
CloudWalk Robotics Lab (CW-RL). Part of the
Psi0 collection. Fine-tuned from the
Psi0 base model; integrates with the
LeRobot framework.
Citation
1@misc{cwrl_psi0_sneaker_v3_2026,
2 title = {Psi0 Fine-Tune v3 --- G1 Sneaker-in-Box, Bimanual + Lateral + Recovery (Inspire RH56DFTP)},
3 author = {{CloudWalk Robotics Lab}},
4 year = {2026},
5 howpublished = {Hugging Face model repository},
6 url = {https://huggingface.co/cloudwalk-research/psi0-g1-sneaker-325ep-v3-finetune}
7}
8
9@article{psi0_2026,
10 title = {Psi0: A Generalist Vision-Language-Action Model for Humanoid Robots},
11 author = {Wei, Songlin and others},
12 year = {2026},
13 eprint = {2603.12263},
14 archivePrefix = {arXiv},
15 url = {https://arxiv.org/abs/2603.12263}
16}
17
18@article{black2024pi0,
19 title = {{$\pi_0$}: A Vision-Language-Action Flow Model for General Robot Control},
20 author = {Black, Kevin and Brown, Noah and Driess, Danny and others},
21 journal = {arXiv preprint arXiv:2410.24164},
22 year = {2024}
23}
24
25@software{lerobot2024,
26 title = {{LeRobot}: State-of-the-art Machine Learning for Real-World Robotics in {PyTorch}},
27 author = {{Hugging Face}},
28 url = {https://github.com/huggingface/lerobot},
29 year = {2024}
30}
License
Released under Apache-2.0. This is a research preview: not intended for safety-critical use; closed-loop deployment on a physical humanoid requires human oversight.
