Diffusion Policy with a transformer denoiser and a frozen DINOv2 ViT-S/14 vision
encoder, whose patch tokens are mean-pooled to one token per camera.
Trained for the Φ robot-learning project on a 3-camera SO-101 arm.
Trainable params
9,072,774
Frozen params (DINOv2 ViT-S/14)
22,056,576
Held-out loss
0.0117 @28k, 0.0121 @30k
Inference
~403 ms / 48-step chunk (batch 1, Apple-silicon MPS, 16 DDIM steps)
Why this model exists — and an honest negative result
This was the control arm in a study testing Patch Policy
(Zhou, Cui, Langford, Tan, LeCun, Pinto), which argues that dense ViT patch tokens beat
pooled features for manipulation.
On this task, it came out the other way. Identical frozen backbone, identical crop,
identical decoder, identical seed and data — the only difference is whether the 300 patch
tokens per camera are kept or averaged:
encoder
attention
memory tokens
min eval_loss
DINOv2, avg-pooled (this model)
bidirectional
9
0.0117
DINOv2, CLS token
bidirectional
9
0.0119
ResNet-18 (trained from scratch) + SpatialSoftmax
bidirectional
3
0.0132
DINOv2, dense patches
bidirectional
1803
0.0163
DINOv2, dense patches
causal
1803
0.0380
ResNet-18 + SpatialSoftmax
causal
3
0.0287 → 0.0361 (overfits)
Pooling beat dense by 28%, and was ahead at every evaluation from step 4000 onward.
avg and cls agree within 1.7%, so the effect is about token count, not the choice of
summary. Adding a fusion stage to the dense arm (n_cond_layers=2, +1.05M params) recovered
only 1.8%, so "nothing fuses the patches" does not explain it either.
Scope: one task, one dataset, held-out denoising loss. The paper's real-world gains are on
low-tolerance tasks (~2 mm cable insertion); this is a coarser pick-and-place. This is not a
refutation of Patch Policy — it is one setting where the effect did not reproduce.
The larger effect: bidirectional attention
The biggest factor in the whole study was not the encoder but the attention mask.
Diffusion denoises all 48 action positions simultaneously, so causal masking among action
tokens forbids reading information that is already present. Removing it gave 2.67× lower
loss on the ResNet arm and 2.28× on the dense-patch arm, at zero extra parameters —
and eliminated overfitting that weight decay had not.
Held-out denoising loss is not task success. It has not been rolled out on the arm.
Patch Policy's own Table 12 finds block-causal ≥ full attention on task success for a
diffusion head, which is the opposite of what we measured on loss.
Requires a custom policy type (diffusion_patch); it will not load with stock lerobot.
train_config.json retains the original scratch paths from the training cluster.
