Vision encoder follows Chi et al. §3.2: ResNet-18 without pretraining, spatial softmax, GroupNorm. The paired run is dp_so101_cubcyl_recovery_tp48_lerobot_3cam, identical except it uses lerobot's default BatchNorm + ImageNet encoder.
🚨 THIS IS THE 20,000-STEP CHECKPOINT, NOT THE FINAL ONE
The run went to 100k steps. Held-out loss got monotonically worse the entire time.
step
10k
20k
40k
60k
80k
100k
eval_loss
0.0155
0.0173 ← this
0.0248
0.0369
0.0532
0.0636
Train loss ended at 0.002 while held-out loss rose 4× — a ~32× gap, over 98 epochs of 113 training episodes.
The true optimum is 10k and it was never saved (save_freq=20000). This is the best checkpoint that exists, not the best checkpoint possible. A re-run with dense checkpointing is the obvious fix and is not done yet.
Leading explanation: the paper's recipe assumes an EMA of the weights, and lerobot implements no EMA — no EMAModel, no use_ema anywhere in the package. EMA is a strong implicit regulariser, so this is plausibly "we removed the paper's regulariser and then trained 98 epochs", not "the dataset is too small". Untested.
⚠️ eval_loss here is noise-prediction MSE, not a task metric. It averages over uniformly sampled diffusion timesteps, where the easy high-noise regime dominates and the hard low-noise regime is what actually sets action precision. A 4× degradation is a strong negative signal, not proof the policy is unusable. Nobody has rolled this out yet.
🚨 Three things will silently break this policy
1. Inference latency — Diffusion Policy has a hard real-time deadline
Unlike ACT, which runs its network once per chunk, DP runs the UNet num_inference_steps times. Measured on a MacBook Air (MPS, fp32, batch 1):
component
cost
3 × ResNet-18 encoders, 2 frames each, 216×288
74.6 ms
UNet × DDIM-16 at Tp=48
219.8 ms
total per chunk
294.4 ms
The deadline is n_action_steps / fps. At 30 fps:
n_action_steps
budget
verdict
8
267 ms
❌ misses
24 (shipped)
800 ms
✅ 506 ms headroom
⇒ Do not lower n_action_steps below ~10 on this hardware. And do not raise num_inference_steps toward lerobot's default of 100: that alone costs ~1370 ms per chunk and the policy simply will not keep up. Missing the deadline raises no error — lerobot-rollout logs Record loop is running slower than the target FPS and the arm stalls or stutters.
n_action_steps is safely tunable at rollout across 1…48 (it does not enter the loss), so sweep it — just stay above ~10.
2. Calibration frame
A policy emits joint angles in the calibration frame of the machine that recorded its training data. Run it against a different one and the arm moves smoothly to the wrong place, with no error.
Confirmed on the arm 2026-08-10. And separately: where the arm's base is clamped is part of that frame too — a re-clamp shifted ours right and every policy reached right of the object, while every config file stayed byte-identical. See so-arm101 setup §3b-bis and §3c.
3. Camera keys and their physical mapping
Needs exactly three: observation.images.wrist, observation.images.front, observation.images.top, all 3 × 480 × 640 (resized to 240×320 and centre-cropped to 216×288 internally). Miss one and it will not load; swap two and it loads fine and behaves badly.
⚠️ Verify index → physical camera every session. macOS has no udev and indices reorder silently.
What it is
Policy
Diffusion Policy (CNN/UNet, FiLM conditioning)
Parameters
102,256,864 tensors total · 68,665,222 UNet
Vision encoder
ResNet-18 from scratch, spatial softmax (32 keypoints), GroupNorm
n_obs_steps / horizon / n_action_steps
2 / 48 / 24
Scheduler
DDIM, 100 train timesteps, 16 inference
Input resolution
resize 240×320 → random crop 216×288 (centre crop at eval)
Steps · batch · seed
20,000 (of a 100k run) · 64 · 1000
lr / wd / schedule
1e-4 / 1e-6 / cosine, 500 warmup
Parameters follow Table 7's real-world rows of the paper, with one deliberate departure: Tp/Ta = 48/24 rather than the paper's 16/8. The paper ran at 10 Hz, where Ta=8 is 800 ms of open loop. At 30 fps the same duration needs Ta=24. Copying the step counts would have produced a policy that misses its deadline every chunk (see §1).
Training data
phi_so101_cubes_cylinder_recovery_v1 — 143 episodes / 81,943 frames = 120 clean + 23 failure-then-recovery. All 23 recovery episodes in training; the 30-episode holdout is clean-only:
Identical to the split used by the ACT models, so the two are comparable on the same held-out episodes.
Eval
Not scored on the arm. No success rate exists.
The baseline to beat, from the ACT work — act_so101_cubcyl_poshold_chunk50_cvae_3cam, 11 held-out rollouts, 2026-08-10: 27% success / 0.418 mean progress. Per object: red cube 67%, yellow cylinder 25%, white cube (45 mm) 0% (0/4).
⚠️ DP's eval_loss cannot be compared to ACT's. DP's is noise-MSE; ACT's is L1 + KL on actions. Different quantities, different scales. DP vs ACT is a rollout question only.
Three confounds to report alongside any DP-vs-ACT number:
this model
ACT
input resolution
216×288 → 7×9 ResNet grid
480×640 → 15×20
open-loop commit
800 ms
1670 ms
encoder init
scratch
ImageNet
Method vs method as published, not a controlled comparison of action decoders.
Provenance
Script configs/hpc/train_diffusion_recovery.sbatch, write-up experiments/2026-08-12_dp-recovery-encoder-ab.md, W&B project phi_dp, SLURM array 9088486 task 0, 7 h 26 m on one H200.
