Views
No views yet
d actions which are emitted immediately, the buffer slides, and d fresh-noise
slots are appended. The robot is fed continuously instead of waiting for a whole chunk.staircase_warmup_prob=0.2), so
the weights retain the ability to denoise a chunk from pure noise — needed once to initialize the
buffer at the start of an episode.--inference.type=pir2, which refuses prefix-trained checkpoints.| base checkpoint | lerobot/pi05_base |
| dataset | nepyope/t-shirt_pick_and_place_clean — 47 episodes, 101,254 frames, 50 fps |
| robot | unitree_g1, 3 cameras (ego_view, left_wrist, right_wrist) at 480×640 |
| task | put the t-shirt on the table |
| action / state dim | 66 (64 joints + 2 grippers) / 31 (29 DOF + 2 grippers), padded to max_state_dim=32 |
| code | huggingface/lerobot @ 0a53c2f2e (pir2-staircase, PR #4427), stacked on training-time RTC (PR #4056) |
rtc_training_schedule | staircase |
rtc_training_max_delay | 1 (delays drawn uniformly from {0, 1} per example) |
staircase_time_jitter | 0.1 |
staircase_warmup_prob | 0.2 (default) |
chunk_size | 50 — at 50 fps this is 1.0 s of motion |
| trainable params | 693M of 4.14B (train_expert_only=true, VLM frozen) |
| hardware | 4×H100 80GB, one node, 39.9 GB per GPU |
| batch | 32 per GPU × 4 = 128, no gradient accumulation |
| optimizer | AdamW, LR 1e-4, weight decay 1e-4, betas (0.9, 0.95), cosine decay to 1e-5 with 500 warm-up steps over 8000 |
| throughput | 2.70 s/step, 47 samples/s — 8000 steps in 6.0 h (10.1 epochs) |
1accelerate launch --num_processes=4 --mixed_precision=bf16 \
2 -m lerobot.scripts.lerobot_train \
3 --policy.type=pi05 \
4 --policy.pretrained_path=lerobot/pi05_base \
5 --policy.max_state_dim=32 --policy.max_action_dim=66 \
6 --policy.rtc_training_schedule=staircase \
7 --policy.rtc_training_max_delay=1 \
8 --policy.staircase_time_jitter=0.1 \
9 --policy.train_expert_only=true \
10 --policy.freeze_vision_encoder=false \
11 --policy.gradient_checkpointing=true \
12 --policy.push_to_hub=false \
13 --policy.chunk_size=50 --policy.n_action_steps=50 \
14 --dataset.repo_id=nepyope/t-shirt_pick_and_place_clean \
15 --dataset.root=/path/to/t-shirt_pick_and_place_clean \
16 --batch_size=32 --num_workers=10 --steps=8000 \
17 --use_policy_training_preset=false \
18 --optimizer.type=adamw --optimizer.lr=1e-4 --optimizer.weight_decay=1e-4 \
19 --optimizer.betas="[0.9,0.95]" \
20 --scheduler.type=cosine_decay_with_warmup \
21 --scheduler.num_warmup_steps=500 --scheduler.num_decay_steps=8000 \
22 --scheduler.peak_lr=1e-4 --scheduler.decay_lr=1e-5 \
23 --wandb.enable=true --wandb.project=tshirt-staircase --wandb.disable_artifact=true \
24 --save_freq=2000 --log_freq=50 \
25 --output_dir=/path/to/output --job_name=tshirt_staircase_8kNormalizationMode.QUANTILES, but the
dataset shipped with only count/max/mean/min/std, so training aborts on the first batch with
QUANTILES normalization mode requires q01 and q99 stats. Fixed by copying the dataset to a
writable location (dereferencing symlinks, so the shared HF cache blobs are never written through)
and running:1python src/lerobot/scripts/augment_dataset_quantile_stats.py \
2 --repo-id=nepyope/t-shirt_pick_and_place_clean \
3 --root=/path/to/writable/copy --skip-images --overwrite--skip-images is safe and fast here because pi0.5 maps VISUAL to IDENTITY, so no video needs
decoding. Note the script calls push_to_hub() unconditionally after writing meta/stats.json
locally, so it can appear to fail after having already done the useful work. The resulting stats are
baked into this checkpoint's preprocessor.max_action_dim=66 disagrees with pi05_base's 32, so
action_in_proj / action_out_proj cannot be loaded. In PI05Policy.from_pretrained the resulting
load_state_dict error is swallowed by a broad except, which returns a randomly initialized
model that then trains and logs normally. A local patch drops only the shape-mismatched tensors so
the backbone loads and just those two projections start fresh. Without it this run would have
silently trained from scratch.| epoch | 0.06 | 0.19 | 1.64 | 3.22 | 4.80 | 6.38 | 7.96 | 10.11 |
|---|---|---|---|---|---|---|---|---|
| loss | 1.245 | 0.633 | 0.111 | 0.094 | 0.085 | 0.078 | 0.075 | 0.073 |
| grad norm | 0.288 | 1.346 | 0.190 | 0.092 | 0.072 | 0.061 | 0.056 | 0.054 |
1lerobot-rollout \
2 --strategy.type=base \
3 --policy.path=<this repo or a local download> \
4 --inference.type=pir2 \
5 --inference.max_delay=1 \
6 --inference.latency_window=20 \
7 --robot.type=unitree_g1 \
8 --task="put the t-shirt on the table" \
9 --fps=50--fps=50 is not optional: the dataset is 50 fps, so consecutive actions are 20 ms apart and the
policy has no notion of frame rate beyond that spacing. Running at 30 fps executes the motion 1.67×
slower than demonstrated.--inference.max_delay=1 matters. The engine derives the per-call delay as
max(1, min(max_delay, round(mean_latency / (1/fps)))) from a rolling window of measured action-head
latencies, and max_delay defaults to chunk_size // 2 = 25 — it is not clamped to the value used
in training. Since this checkpoint only ever saw d ∈ {0, 1}, letting the engine pick d=2 or more
extrapolates past the trained schedule. A 20 ms tick is tight (one expert substep against a cached
prefix measures ~12 ms on an RTX 5090 laptop, before capture and transport overhead), so that is a
realistic risk rather than a theoretical one. Pinning max_delay=1 keeps inference in distribution;
if calls do overrun, the buffer simply runs a cycle behind rather than going off-schedule.--policy.rtc_training_max_delay=3 would cover d up to 3 (60 ms per call at
50 fps) while keeping d=1 in distribution, and would be the more deployable choice.