desk_view + wrist_left camera images, 6-dim joint state, English instruction
Outputs
6-dim continuous action chunks
Quality
Works really well across all four tasks. Best model produced by the project.
Checkpoint 008000 is the recommended one. Consistent with the V7 run, later checkpoints degrade — checkpoints beyond roughly step 20000 were unusable on the real robot. More steps is not better here.
Training
Run name pi05_6gpu_fsdp_V2 (SLURM job 2166960). 6x L40S, FSDP FULL_SHARD —
plain DDP OOMs, because the optimizer state is replicated per rank.
--policy.dtype=float32 is required under FSDP; bf16 comes from the accelerate config's
mixed_precision instead. Setting the policy dtype to bfloat16 directly breaks FSDP here.
Pi0.5 requires lerobot[pi] installed from GitHub main, not the PyPI release.
Usage
[!IMPORTANT]
Unusual repository layout — from_pretrained("Project-IRA/...") on the repo ID
will not work. Model files are nested under outputs_V8/; that folder name comes
from the training run and deliberately does not match the repo's V7_Full_V2 name.
outputs_V8/train/pi05_6gpu_fsdp_V2/checkpoints/<step>/pretrained_model/ <- weights
outputs_V8/train/pi05_6gpu_fsdp_V2/checkpoints/<step>/training_state/ <- resume only
Checkpoints present:002000, 004000, 006000, 008000, 010000, 012000,
014000, 016000. The run itself went to 30000 steps, but only these eight were
uploaded — the later ones were not useful (see Quality).
Fetch just the recommended checkpoint's weights (~13 GB instead of 298 GB):
The repo totals ~298 GB because training_state/ (optimizer moments, scheduler,
RNG) is published beside every checkpoint. You do not need it for inference — the
--include filter above skips it.
python
1from lerobot.policies.pi0.modeling_pi0 import PI0Policy # pi05 shares the PI0 module23policy = PI0Policy.from_pretrained("<local path to the checkpoint's pretrained_model/>")4policy = policy.to("cuda").eval()
Inference dtype: checkpoints are saved from a bfloat16 training run. If you hit
GPU OOM at inference, confirm the loaded policy is in bfloat16 and not silently
upcast to float32.
Pi0.5 is ~4B parameters. On-robot inference from the robot PC is impractical; we served
it over the asynchronous gRPC inference server shipped in the code repo
(https://github.com/Project-IRA/interactive-robotic-arm) and ran the robot as a thin client.
Robot setup
Robot
SO-101 follower arm (6-DOF), robot_type: so_follower
Inference note: both cameras are run at 640x480 during inference, not at their
recording resolutions, to reduce the payload sent to the inference server.
Environment notes
All training ran on a SLURM cluster with L40S GPUs. Two environment details were required
and are easy to miss when reproducing:
ffmpeg libraries for torchcodec. A minimal conda env supplies the shared libraries
that torchcodec discovers at runtime:
export LD_LIBRARY_PATH=$CONDA_PREFIX/envs/ffmpeg_libs_v8/lib:<venv>/lib/python3.12/site-packages/nvidia/npp/lib:$LD_LIBRARY_PATH
--tolerance_s=0.01 on every run, to accommodate timestamp jitter in the recorded
episodes.
Multi-GPU runs additionally set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
Datasets and the virtualenv were copied to node-local /scratch before training rather
than read from shared storage.
No Weights & Biases logging was enabled for any run (--wandb.enable=false), so there are
no public training curves — the job.*.err SLURM logs are the record.
Tasks and prompts
The model is conditioned on English natural-language instructions. Prompt phrasing was
varied roughly every 10 episodes during recording, giving 93 distinct prompts in the
merged dataset. Use one of the training prompts verbatim for best results — the full
lists are on the dataset card.
Limitations
Behaviour cloning. The policy imitates teleoperated demonstrations and has no notion
of recovery beyond what was demonstrated. It is susceptible to covariate shift and can
fail to recover from states outside the demonstration distribution.
Recovery data is incidental, not systematic. Recovery behaviour appears in the data
only where the operator happened to make and correct a mistake during recording; no
recovery episodes were scripted deliberately.
Single environment. All data comes from one lab desk with one lighting setup, one
camera geometry, and one set of physical objects. Expect degradation elsewhere.
Prompt sensitivity. Language conditioning was trained on a fixed set of phrasings
(listed in the dataset card). Prompts far from those phrasings may behave unpredictably.
No formal evaluation. Quality assessments below are qualitative, from operators
observing rollouts on the physical arm. There are no success-rate numbers.
Not safety-rated. Supervise all physical execution and keep the workspace clear.
Upstream licensing & attribution
This model is a derivative work of Apache-2.0 licensed components:
Apache-2.0 permits relicensing derivative works. We retain the upstream copyright
notices, license text, and NOTICE files for the incorporated material, as Apache-2.0
Section 4 requires. The upstream components remain under Apache-2.0 — only this
project's own contributions (the fine-tuned weights and training configuration) are
offered under CC BY-SA 4.0.
CC BY-SA 4.0 was chosen because it is share-alike: derivatives must be released under
the same licence, so this work cannot be taken closed-source. The project's source code
lives in a separate repository under its own licence — see
https://github.com/Project-IRA/interactive-robotic-arm.
Citation
bibtex
1@misc{project_ira_2026,
2 title = {Project-IRA: Interactive Robotic Arm},
3 author = {Baten, Cleo and Keppler, Bela and Sapper, Jonas},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/Project-IRA}},
6 note = {Code: \url{https://github.com/Project-IRA/interactive-robotic-arm}}
7}
