Best checkpoint in this repo: 001500 (eval_loss 0.1494).
🚨 Read before running a rollout
1. You must pass one of the six instructions, verbatim. This model is language-conditioned:
the instruction selects the behaviour. LeLab's task field defaults to an EMPTY string and does
not auto-fill, so a blank task is off-distribution for everything the model learned.
pick up the red cube and place it in the cardboard box
pick up the red cube and place it in the white bin
pick up the white cube and place it in the cardboard box
pick up the white cube and place it in the white bin
pick up the yellow cylinder and place it in the cardboard box
pick up the yellow cylinder and place it in the white bin
2. Camera order is baked in.smolvla_base was pretrained on camera1/2/3, and make_policy
does not re-derive feature names when a pretrained path is set. Training used an explicit rename,
stored in policy_preprocessor.json and applied automatically:
dataset key
policy key
observation.images.wrist
observation.images.camera1
observation.images.front
observation.images.camera2
observation.images.top
observation.images.camera3
Physically reassigning cameras silently feeds the model the wrong views.
Held-out loss
eval_split=0.1 holds out 12 episodes, 2 per instruction, leaving 108 train / 60,281 frames.
The 20k run follows the documented SmolVLA recipe and overfits monotonically from its first
measurement: held-out loss rises 0.1659 → 0.3444 while train loss falls 14x (0.324 → 0.023).
That recipe assumes ~50 episodes of one task; this dataset saturates in about 2 epochs. The 4k
run anneals properly over its own length and wins.
⚠️ Note the two runs are not the same LR schedule. Both configs record warmup 1000 / decay
20000, but LeRobot auto-rescales whenever steps < num_decay_steps, so the 4k run actually ran
~200 warmup with full decay by step 4000. The 4k run is not a prefix of the 20k run.
Training
Base
lerobot/smolvla_base, 450M
Trainable
99.9M of 450M (train_expert_only, freeze_vision_encoder)
No scored rollouts yet. These are held-out losses, not success rates.
Not comparable to Φ's ACT numbers: ACT's eval_loss is pure L1 with the CVAE latent zeroed;
this is a flow-matching regression loss on a different split.
The eval set is 12 episodes, so differences under ~0.005 are inside the noise. The curve here is
non-monotonic, which is why 1500-2000 should be treated as a basin, not a sharp optimum.
