orange-bot — SmolVLA fine-tune for marker-directed pick-and-place
An SO-101 arm policy that picks up whichever die carries a magenta X marker in
the overhead camera frame and places it in a tray. Fine-tuned from
lerobot/smolvla_base.
The point of this policy is what it does not do: it has no idea what a die is,
which one you want, or how to count pips. A constant task string is the only
language it ever receives. Target selection happens upstream, in pixel space, by
drawing a marker onto the overhead frame. That is the entire interface — which
means the thing choosing the target can be a scripted click today and a
vision-language model tomorrow, with no retraining.
Honest results first
| run | trials | successes | rate | 95% CI (Wilson) |
|---|
eval_20260810_185853 | 10 | 4 | 40% | 17%–69% |
eval_20260810_193424 | 19 | 4 | 21% | 9%–43% |
eval_20260810_203613 | 4 | 1 | 25% | 5%–70% |
| pooled | 33 | 9 | 27% | 15%–44% |
Scored on a binary criterion: is the marked die in the tray? A die that lands
in the tray and slides back out is a failure. Trials run to a 45 s deadline.
Pooled: 9/33 = 27% (Wilson 95% CI 15%–44%).
Three caveats travel with that number and none are optional.
- It is uncontrolled. No unmarked control run has been done. This is
evidence that the policy performs the task, not that it follows the marker.
A policy that had learned "grasp whichever die is easiest to reach" would
score similarly on this protocol.
- Trials are spread uniformly over the six reachable workspace cells, not
matched to the training distribution (which is skewed 19/41/0 across
columns). So this rate is not comparable to a distribution-matched one, in
either direction.
- n is small. Distinguishing, say, 21% from 40% at 80% power needs roughly
90 trials per arm. Do not read a difference between the individual runs above
as an effect.
Failures are attributed to one of two layers and never summed: planner (pip
misread, marker placed on the wrong die) or executor (missed grasp, dropped
in transit, grasped an unmarked die). Across every run above the planner
contributed zero failures; the dominant executor failure is missed_grasp —
the gripper closes, nudges the die, and then keeps grasping at empty table.
The likely reason is structural rather than a bug, and it is worth knowing before
you use this: the marker is composited after frame capture, so it can never be
occluded. It is locatable in 60/60 training grasp frames, in 27 of which it is
drawn over the arm's own pixels. The policy therefore always knows where to go
and can never see whether the die is still there. Closing that gap is a job for
a continuously-verifying planner, not for the executor.
Inputs and outputs
| |
|---|
| Robot | SO-101, 6-DoF follower |
| Cameras | overhead (marked) → observation.images.camera1, wrist → camera2, both 640×480 → 256×256 |
| State | observation.state, 6 joint positions |
| Action | 6 joint targets, chunk size 50 |
| Task string | "pick the marked dice and put it in the tray" — constant, always |
smolvla_base declares a third camera slot; it is never filled, and
prepare_images() skips it cleanly. Do not map a stream into it expecting this
checkpoint to use it.
The marker render spec is frozen at v1:158c71b8e2b9 — size_px=17, thickness_px=5, rgb=(255,0,255), alpha=0.75, composited onto the overhead frame before the
policy sees it. This is not a cosmetic setting. The marked frame is the
training distribution, so a marker drawn at a different size or opacity puts this
checkpoint out of distribution with no error anywhere — just a mysteriously
mediocre success rate. Match the spec or re-record.
Cameras, lighting and table layout were held fixed during recording, and no image
augmentation was used (image_transforms.enable=False). This policy has no reason to generalise
across viewpoint or lighting changes, and should not be expected to.
Training
Fine-tuned on
arpitjain2811/pick-and-place-marked-dice_20260809_170651 — 60 teleoperated episodes,
26,609 frames, recorded in two blocks of 30 with a dataset health check between
them.
| |
|---|
| Base | lerobot/smolvla_base (450M) · VLM backbone HuggingFaceTB/SmolVLM2-500M-Video-Instruct |
| Trainable | action expert only — freeze_vision_encoder=True, train_expert_only=True |
| Steps | 20,000 · batch 32 · 640,000 samples ≈ 24.1 epochs |
| Optimizer | adamw lr 0.0001 → 2.5e-06 (cosine, 1000 warmup steps), grad clip 10.0 |
| Final | loss 0.0292, grad norm 0.483 |
| Hardware | RTX 5080 16 GB, 1 h 51 m |
Batch size was measured rather than guessed: 2 → 1.71 GB, 16 → 4.66 GB,
32 → 7.96 GB, 64 → OOM at 15.44 GB. Throughput saturates near 91 samples/s from
batch 16 upward, so 32 was chosen for the gradient, not for speed.
The low training loss is not a success signal. There is no validation split,
and ~24 epochs over 60 episodes can memorise. The only real number is the eval
table above. Training longer is also not available as an improvement: the loss
had plateaued (mean by fifth 0.156 / 0.067 / 0.041 / 0.029 / 0.025) and the
cosine schedule had decayed the learning rate to 2.5e-06.
Using it
1from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
2
3policy = SmolVLAPolicy.from_pretrained("arpitjain2811/orange-bot-smolvla-marked-dice")
Inference measured on Apple MPS: 219 ms per 50-step chunk, 3 ms for
cached ticks, and 5.6 s for the very first forward pass — warm the policy up
before you time anything. CPU inference is 4.5 s per chunk and unusable on a
30 fps control loop.
Note that serving a 50-step chunk at 30 fps means 1.67 s of
open-loop motion between observations, during which the worst joint moves a
median 31° in the training demos. Lowering n_action_steps at inference time is
a supported knob and costs nothing but compute.
Limitations
- One table, one lighting setup, one camera rig. Nothing here was randomised.
- Five dice in the scene, six reachable workspace cells.
- Failure episodes were deliberately not recorded, so this checkpoint cannot
be used as a starting point for reward-model or HIL-SERL work.
- Not evaluated against an unmarked control. See caveat 1.
Provenance
Training run
orange-bot_smolvla_marked_20260810_153554. Code, eval logs and the full decision history:
https://github.com/arpitjain2811/orange-bot. Every eval run in this repo is
dated and committed, including the bad ones — the measurement trail is the point.
