SmolVLA SO-101 Pick-and-Place (10cm, dual-camera)
lerobot/smolvla_base fine-tuned on 75 teleoperated demonstrations of a
cube pick-and-place task with an
SO-101 follower arm.
Task instruction: "Pick up the cube and place it in the bowl"
Results
60–80% success rate on the real robot, typical across multiple 5-episode rollouts at the 20k-step checkpoint.
Success = cube grasped and released inside the bowl within a 10s episode.
Head-to-head against the alternatives, all trained on the same dataset and evaluated at 20k with 5 rollouts each:
| Model | Cameras | Success |
|---|
| SmolVLA dual (this model) | wrist + overhead | 100% (5/5) |
| SmolVLA wrist-only | wrist | 80% (4/5) |
| ACT (52M, from scratch) | wrist + overhead | 80% (4/5) |
That 100% is a single favourable 5-episode run, not a contradiction of the 60–80% figure — treat 60–80% as
the honest expectation and note that 5-episode samples are noisy. The overhead camera appears to supply global
spatial context that helps the placing phase.
Training
| |
|---|
| Base model | lerobot/smolvla_base |
| Dataset | 75 episodes, 22,436 frames, 30 fps |
| Steps | 20,000 |
| Batch size | 64 |
| Optimizer | AdamW, lr 1e-4, cosine decay w/ 1k warmup |
| Image transforms | enabled (brightness / contrast / saturation / hue / sharpness) |
| Final loss | 0.005 (grad norm 0.11) |
| Epochs | ~56.8 over the dataset |
| Hardware | 1x RTX 3090, ~10h 24m wall time |
Vision encoder frozen, action expert trained (freeze_vision_encoder=true, train_expert_only=true).
Observation / action spec
Both cameras are required — this is the dual-camera variant.
| Key | Shape | Notes |
|---|
observation.images.wrist | (3, 480, 640) | Intel RealSense D405, wrist-mounted |
observation.images.overhead | (3, 480, 640) | Logitech C920, overhead |
observation.state | (6,) | joint positions |
action | (6,) | shoulder_pan, shoulder_lift, elbow, wrist_flex, wrist_roll, gripper |
Chunk size 50, n_action_steps 50, n_obs_steps 1. Images are resized with padding to 512x512 internally.
State and action use MEAN_STD normalization; visual input is IDENTITY (dataset stats are baked into the
checkpoint, so no external normalization file is needed).
Usage
1from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
2
3policy = SmolVLAPolicy.from_pretrained("gtgando/smolvla-so101-pick-place-10cm-v3-dual")
4policy.to("cuda")
5
6# batch requires both camera keys, observation.state, and task
7action = policy.select_action(batch)
Inference loop used for the reported numbers:
scripts/infer.py.
Training used a
LeRobot fork (adds a
--resume-from convenience flag and
TensorBoard logging), but the checkpoint is a standard LeRobot SmolVLA checkpoint and loads with upstream LeRobot.
Limitations
- Real-robot only. Trained and evaluated entirely on physical SO-101 hardware. Zero-shot transfer to a
simulator is untested — camera intrinsics, extrinsics, lighting, and cube/bowl appearance all differ, and this
policy has no domain randomization.
- Camera placement is baked in. The overhead and wrist views are fixed to the rig described above. Moving
either camera meaningfully degrades performance.
- Fixed workspace. Demonstrations covered a ~10cm cube placement range on a single table setup.
- Color-specific visual features. Trained on a red cube only. Tested unchanged on other colors
(prompt not modified): orange, blue, and green each grasped successfully but failed or nearly failed during
the place phase — 1/3 success vs 5/5 on red. Grasping generalizes; the place trajectory does not. This is
the clearest evidence that the policy keys on appearance rather than an abstract "cube" concept, and it is the
main reason to expect trouble on any setup whose object appearance differs from the training rig.
- Action jitter. Visible shaking during execution, likely action-chunk boundary discontinuity. Temporal
ensembling was not applied and would probably help.
- Undertrained. Loss was still dropping at 20k steps, and 75 episodes is a small dataset. More data and more
steps are the obvious next levers.
Other checkpoints
A wrist-camera-only variant trained on the same dataset also exists, evaluated at
80% (4/5) under the same
protocol. It needs only one camera, which may make it easier to wire into a new setup. Not uploaded here, but
open an issue on
ggand0/vla-so101 if it would be useful.
License
Apache 2.0, inherited from lerobot/smolvla_base.
