Psi0 Fine-Tune v5 — G1 Sneaker-in-Box + HE Adapter Mix (385 sneaker + 80 HE pick-place)
v5 community fine-tune of Psi0 (paper) on a Unitree G1 humanoid with Inspire RH56DFTP hands — first multi-task run.
Same primary task as the
v1,
v2,
v3 and
v4 fine-tunes —
"Pick up the sneaker and place it inside the box." — but trained on a
465-episode multi-task mix combining the full v4 sneaker dataset (
psi0-g1-sneaker-385ep-v4) with two HE adapter pick-place subsets from
psi0-g1-he-pickplace-9x40ep —
stack_two_cubes_g1 (40 ep) and
place_a_cube_in_a_bag (40 ep). v5 is the first CW-RL run that asks whether multi-task imitation
helps or hurts the in-distribution sneaker eval when the shared VLA backbone is exposed to two semantically-related but distinct tabletop manipulations. Trained at the
CloudWalk Robotics Lab (CW-RL), 2026-05.
Status
Trained 2026-05-15 / -16. 80 000 steps on 8× B200 in 7 h 29 m, ~69 effective epochs over 465 ep / 148 847 frames. Final eval/loss 0.16598 on the v4 sneaker val slice (vs v4's 0.209 on the same slice — 21 % lower at half the step budget). Per-axis L1 errors moved in the opposite direction — see Evaluation §. Open-loop eval and real-robot eval reserved as Pending.
Quick facts
| |
|---|
| Base | Psi0 (Qwen3-VL-2B + flow-based MM-DiT action expert) |
| Pretrained | pre.fast.1by1.2601091803.ckpt.ego200k.he30k + postpre.1by1.pad36.2601131206.ckpt.he30k |
| Training mix | composed-local psi0-g1-sneaker-he-mix-465ep-v5 (385 ep v4 sneaker + 40 ep stack_two_cubes_g1 + 40 ep place_a_cube_in_a_bag) — components linked in front-matter; the composed dataset is not pushed to HF |
| Episode composition | 385 v4-sneaker (60 negative + 94 v1-RIGHT + 111 v2-LEFT + 60 lateral + 60 recovery) + 40 stack_two_cubes_g1 + 40 place_a_cube_in_a_bag = 465 ep / 148 847 frames @ 30 Hz |
| Predecessor (v4) | cloudwalk-research/psi0-g1-sneaker-385ep-v4-finetune (single-task sneaker, 160 k steps) |
| Validation dataset | pinned to psi0-g1-sneaker-385ep-v4 (comparable to v4 baseline) |
| Robot target | Unitree G1 + Inspire RH56DFTP (NOT Dex3-1) |
| Action space | 36-D (6 hand + 1 pad / side, 14 arm, 4 torso, 4 locomotion) |
| State space | 32-D (padded to 36-D for the model) |
| Action chunk | 30 |
| Hardware | 8× NVIDIA B200 (sm_100 / Blackwell), system PyTorch 2.10 (NVIDIA build) |
| Mixed precision | bf16 |
| Optimizer | AdamW, lr=1e-4 cosine + min_lr 5e-7, warmup 1000, betas=(0.95, 0.999), wd=1e-6, grad-clip 1.0 |
| Steps / batch | 80 000 / global batch 128 (16 per GPU × 8 GPUs) |
| Effective epochs | ~69 (80 000 × 128 / 148 847) |
| Checkpointing | every 20 000 steps; default max_checkpoints_to_keep=5 covered all 4 saves (ckpt_20000 / ckpt_40000 / ckpt_60000 / ckpt_80000) |
| RTC | enabled, max_delay = 8 |
| VLM | frozen (only action header trained) |
| Wall-clock | 7 h 29 m on 8× B200 |
| Final eval/loss | 0.16598 (vs v4's 0.209 on same val slice — 21 % lower at half the step budget) |
| W&B run | offline-wwt0zx6p (CW-RL internal; synced post-hoc) |
HE adapter mix — the v5-specific signal
v5 layers 80 episodes of HE adapter pick-place (40 stack_two_cubes_g1 + 40 place_a_cube_in_a_bag) onto the 385-episode v4 sneaker base. The HE adapter dataset was originally prepared as a Hand-Eye coordination benchmark for the G1 / Inspire stack, and both subsets exercise the same pick-place primitive class as the sneaker task but on visually and morphologically different objects (small cubes / fabric bag vs sneaker / cardboard box). The training instruction was kept task-specific per episode (HE episodes carry their own language prompts, sneaker episodes carry the canonical "Pick up the sneaker and place it inside the box.").
The hypothesis. Shared VLA backbones (Qwen3-VL-2B + flow-based action expert) should benefit from broader manipulation supervision — more pick-place geometry, more language variety — without degrading the in-domain sneaker target, if the model has enough capacity and the task gradient interference is mild. The 80 / 385 mix ratio (~17% by episode count, ~20% by frame count) was chosen as a conservative first-step probe: large enough to register a multi-task signal, small enough that the sneaker task remains the dominant source of gradient.
Frame-weighted exposure. HE pick-place episodes average ~7-8 s vs ~9 s for v4 sneaker positives, so the 80 / 385 episode share corresponds to ~29 720 / 148 847 ≈ 20 % by frame count — slightly over-weighted vs the episode share but still well below the sneaker majority. If real-robot behavior on v5 shows degraded sneaker manipulation precision but functional HE pick-place, the load-bearing question becomes: is the eval/loss gain on sneaker (21 % lower than v4) explained by easier HE-frame minibatches diluting the per-batch loss, rather than by genuine sneaker quality improvement? The Evaluation § captures the v4-vs-v5 per-axis L1 comparison that addresses this.
Step-budget rationale (80 k vs v4's 160 k)
Halved relative to v4 — the training-loss curve flattens well before 80 k on the multi-task mix (final-step train loss ≈ 0.05–0.08 from step ~60 k onward), so doubling to 160 k would have added wall-clock without measurable gain on the in-process eval/loss metric. The 4-checkpoint ladder (every 20 k) keeps the diagnostic resolution needed to locate the post-eval optimum once open-loop eval lands, while the smaller step budget reflects the empirical observation that v5 reached its plateau roughly half as fast as v4 in step terms.
Repository contents
This repo carries 3 diagnostic checkpoints (storage-cap-aware decision, carry-over from v4 — skip 20 k as too early to be a useful diagnostic given the loss curve flattens well before 40 k):
checkpoints/ckpt_40000/model.safetensors (~5.9 GB) — earliest useful diagnosticcheckpoints/ckpt_60000/model.safetensors (~5.9 GB) — mid-late referencecheckpoints/ckpt_80000/model.safetensors (~5.9 GB) — finalrun_config.json (tyro-serialized launch config)argv.txt (full command line)README.md (this file)
Optimizer / scheduler / sampler / random-state files are intentionally omitted — training-resume only. The 20 k checkpoint is retained on CW-RL NFS and is available on request.
Evaluation
In-process (training-time eval/loss, in-distribution sneaker)
| Metric | v2 (199ep, 40k) | v3 (325ep, 160k) | v4 (385ep, 160k) | v5 (465ep mix, 80k) |
|---|
eval/loss | 0.0816 | 0.267 | 0.209 | 0.16598 |
eval/err_l1_arm_joints | 0.114 | 0.0834 | 0.0821 | 0.16103 |
eval/err_l1_hand_joints | 0.0117 | 0.0123 | 0.0124 | 0.01914 |
eval/err_l1_height | small | small | 1e-5 | 1e-5 |
eval/err_l1_target_yaw | small | small | 6.3e-4 | 6.6e-4 |
| Wall-clock | 3 h 42 m | 14 h 46 m | 14 h 35 m | 7 h 29 m |
| Effective epochs | 84 | 184 | 172 | 69 |
The v5-specific finding: eval/loss and downstream per-axis L1 move in opposite directions. v5's eval/loss is 21 % lower than v4 (0.166 vs 0.209), but the per-axis L1 errors on the same v4 sneaker val slice went up — arm_joints ~2× (0.161 vs 0.082), hand_joints ~1.5× (0.019 vs 0.012). The mechanism is plausibly:
eval/loss is the flow-matching noise-prediction objective, averaged across all val frames. Multi-task training gives the model more diverse trajectories to learn from, easing the noise-prediction problem (the model gets better at "what direction in action space does each noise level point to"). This is reflected in the lower scalar eval/loss.- Per-axis L1 errors measure final action prediction accuracy on held-out sneaker frames specifically. These can degrade independently of
eval/loss if the multi-task gradient signal pulls the action expert away from sneaker-precise action sequences toward a more general "pick-place" mean.
This is the load-bearing v5 finding: lower flow-matching loss does not imply better per-axis action prediction on the in-distribution target, especially in a multi-task setting. The open-loop / real-robot eval below will resolve whether the L1 regression translates into observable closed-loop degradation, or whether sneaker manipulation remains functional despite the L1 number.
Open-loop evaluation (ID + OOD)
Pending — to be done in a follow-up session. Recommended ckpts: 40 k, 60 k, 80 k (the 3 uploaded here). Two precondition options for the held-out probe:
- Reuse v2's eval set (
psi0-g1-sneaker-6ep-v2-eval) — gives an apples-to-apples sneaker OOD axis vs v2 / v3 / v4.
- Add an HE pick-place held-out — carve a few episodes from each HE subset that were not used in training. Specific to v5 (no prior comparison axis), but probes the multi-task generalization claim directly.
The v5-specific eval question that matters most: does the policy execute sneaker pick-place as reliably as v4 on the same scenes, or does the HE mix degrade sneaker behavior in observable ways (e.g., grasp angle, approach trajectory, release timing)?
How to download
1import os
2from huggingface_hub import hf_hub_download
3
4repo = "cloudwalk-research/psi0-g1-sneaker-he-mix-465ep-v5-finetune"
5local_dir = ".runs/finetune/sneaker-he-mix-v5"
6for f in [
7 "run_config.json",
8 "argv.txt",
9 "checkpoints/ckpt_80000/model.safetensors", # one of {ckpt_40000, ckpt_60000, ckpt_80000}
10]:
11 hf_hub_download(repo_id=repo, filename=f, repo_type="model",
12 local_dir=local_dir)
How to evaluate (closed-loop sim)
1bash sim/scripts/run_closed_loop.sh \
2 "Pick up the sneaker and place it inside the box." \
3 .runs/finetune/sneaker-he-mix-v5 80000 \
4 --fixed_base --live_viewer
For v5, watch (on top of v4's negative-scene + bimanual + recovery signals): does the policy execute sneaker pick-place at v4-quality, or does multi-task training degrade in-domain manipulation? Optionally also evaluate with the HE pick-place prompts (e.g., "Stack the two cubes.") to confirm the multi-task half of the run learned anything.
Known caveats
- Generalization is not yet validated. Trained but not evaluated open-loop or on the robot; numbers and decisions land post-eval.
- The per-axis L1 regression vs v4 is the load-bearing open question. The Evaluation § captures the metric movement; whether it corresponds to observable closed-loop degradation is empirically unknown.
- HE pick-place tasks were NOT held out at training time — this fine-tune is in-distribution on both task families, so out-of-distribution generalization between sneaker and HE tasks is untested.
- Hand commands are still lockstep across the 6 active Inspire DoF (per side). Unchanged from v1–v4; a teleop-modality limitation.
- Task is bimanual but stationary on positives, fully stationary on negatives. Action dims for locomotion (vx/vy/vyaw/target_yaw) and torso RPY are constant in the sneaker dataset; HE pick-place subsets are also stationary.
- In-process
eval/loss is in-distribution and computed against the v4 sneaker val slice only. It does NOT report multi-task performance — HE-side quality is invisible to this metric and to the per-axis L1 numbers in the Evaluation table above.
- Inspire deploy bridge is required (resolved in v1, inherited in v2/v3/v4/v5).
- Step budget halved vs v4 — direct step-count-controlled comparison with v4 is not available. The like-for-like comparison would require evaluating v4 at step 80 k, which was not done at the time and is no longer accessible (v4 NFS retired).
Lineage
References
- Psi0 — Generalist VLA model, base policy fine-tuned here. Paper, GitHub.
- Cluster runbooks (v2, v3, v4, v5) — Container reuse strategy, full teardown, and ckpt retention details; tracked internally at CW-RL.
- v4 fine-tune —
cloudwalk-research/psi0-g1-sneaker-385ep-v4-finetune.
- HE adapter dataset —
cloudwalk-research/psi0-g1-he-pickplace-9x40ep.
- Pre-deploy checklist — Psi0 issue #39.
Attribution
Developed by
cloudwalk-research in the
CloudWalk Robotics Lab (CW-RL). Part of the
Psi0 — CW-RL collection. Fine-tuned from the
Psi0 base model; integrates with the
LeRobot framework.
Citation
1@misc{cwrl_psi0_sneaker_he_mix_v5_2026,
2 title = {Psi0 Fine-Tune v5 --- G1 Sneaker-in-Box + HE Adapter Mix (385 sneaker + 80 HE pick-place, Inspire RH56DFTP)},
3 author = {{CloudWalk Robotics Lab}},
4 year = {2026},
5 howpublished = {Hugging Face model repository},
6 url = {https://huggingface.co/cloudwalk-research/psi0-g1-sneaker-he-mix-465ep-v5-finetune}
7}
8
9@article{psi0_2026,
10 title = {Psi0: A Generalist Vision-Language-Action Model for Humanoid Robots},
11 author = {Wei, Songlin and others},
12 year = {2026},
13 eprint = {2603.12263},
14 archivePrefix = {arXiv},
15 url = {https://arxiv.org/abs/2603.12263}
16}
17
18@article{black2024pi0,
19 title = {{$\pi_0$}: A Vision-Language-Action Flow Model for General Robot Control},
20 author = {Black, Kevin and Brown, Noah and Driess, Danny and others},
21 journal = {arXiv preprint arXiv:2410.24164},
22 year = {2024}
23}
24
25@software{lerobot2024,
26 title = {{LeRobot}: State-of-the-art Machine Learning for Real-World Robotics in {PyTorch}},
27 author = {{Hugging Face}},
28 url = {https://github.com/huggingface/lerobot},
29 year = {2024}
30}
License
Released under Apache-2.0. This is a research preview: not intended for safety-critical use; closed-loop deployment on a physical humanoid requires human oversight.
