SmolVLA — language-conditioned two-disc pick and place (SO-101, MuJoCo)
A SmolVLA policy fine-tuned to pick up a cube and place it on the disc named in a
natural-language instruction. Two coloured target discs are visible at all times, so the
picture alone cannot say which one is correct — the instruction is the only signal that
identifies the goal.
This is the reported final model from the project: the checkpoint at 60,000 steps,
about two passes over the dataset. zwan1003/pickplace_skills_vla_v3_2 holds the same run
at 90,000 steps, which overfits and scores lower; it is kept for comparison.
Code: the environment, scripted expert controller, data collection pipeline and evaluation
harness are at github.com/zwaneiz/so101-vla-pickplace.
You need it to reproduce the evaluation, since the two-disc scene is a custom subclass of the
so101-nexus environment.
What it does
Instruction
Behaviour
pick up the cube
Grasp and lift the cube
place it on the left / place it on the right
From a held cube, carry and set down on the named disc
pick up the cube then place it on the left
The whole sequence
Use these strings exactly. The policy was trained on them and is sensitive to wording (see
Limitations).
Results
Measured on starting positions the model never trained on, averaged over three seeds.
Measure
Result
Episodes
Grasp the cube
33%
150
Place it, once held
97%
30
Whole task
37%
120
Went to the instructed disc
96–100%
—
Went to the wrong disc
2 of 120
—
The whole-task score is close to grasp × place. Placement is close to solved; grasping is the
bottleneck, caused by a gripper tolerance of under 4 mm on a 25 mm cube.
It follows the words, not the colours. Recolour both discs grey and it still scores 87%.
Swap the two colours so they contradict the instruction and it scores 100%, going to the
instructed position every time. Remove the direction word and it drops to 3%.
The environment is a two-disc variant of the so101-nexus pick-and-place scene: a second
target disc is injected into the model so both a left and a right target are always present.
That subclass, and everything else needed to reproduce the evaluation, is in the
code repository.
Get this wrong and the policy receives the wrist image where it expects the overhead view. It
does not raise an error, it just behaves badly.
2. Match the physics. The training data was collected with a stiffened contact model and
tangential friction scaled by 3.0, representing rubber fingertip pads. Evaluate with the same
settings or the grasp will not transfer.
3. Seed torch per episode. SmolVLA samples noise at every inference through its
flow-matching action head. Seeding the simulator alone is not enough; without seeding torch,
the same command produces different results.
4. Normalisation comes from the dataset. State is normalised on input and actions are
de-normalised on output using the statistics of
zwan1003/pickplace_skills_v3_2.
Omitting this produces smooth but completely misdirected motion.
Limitations
Narrow language. Near-synonyms are free: grab the cube and pick up the block score
the same as the trained wording. Distant phrasing is not: carry the object drops to 7%,
colour naming (place it on the blue disc) and negation both fail outright.
Grasping is the ceiling. The gripper needs better than 4 mm of accuracy. Raising contact
friction further lifts whole-task success to about 56% without retraining, which shows the
limit is mechanical rather than learned.
Simulation only. No transfer to a physical arm has been tested or claimed.
Restricted workspace. Cube positions are drawn from a measured reachable region, not the
arm's full range.
Context
Produced for a final-year project investigating whether a compact vision-language-action model,
fine-tuned on a small dataset using consumer hardware, genuinely conditions its behaviour on
language rather than on visual shortcuts. The whole pipeline — data collection, training and
evaluation — ran on a single laptop with a 4 GB GPU.
