SmolVLA (~450M) fine-tuned on all four tasks, with the vision encoder unfrozen and image augmentation enabled. The best SmolVLA model in the project — though still meaningfully behind Pi0.5.
camera1 (wrist) + camera2 (desk) images — renamed keys, see Usage — 6-dim joint state, English instruction
Outputs
6-dim continuous action chunks
Quality
OK-ish. Roughly the same quality as the single-task V2 Lego model, but across most tasks rather than just Lego — so the multi-task generalisation worked, at the same per-task quality level.
Quality was assessed around checkpoint 150000, but checkpoints were not compared systematically for this run — 150000 is a rough indication, not a validated optimum. Checkpoints exist every 10000 steps to 200000 plus last, so it is worth trying several.
For comparison, Pi0.5 on the same dataset works really well at checkpoint 8000. SmolVLA at ~450M parameters appears capacity-limited for this four-task set.
Training
SLURM job 2164957. Single L40S GPU.
Setting
Value
Base
lerobot/smolvla_base
Dataset
930-episode merged set
Steps
200000, completed (--save_freq=10000); assessed around 150000
V5 used SmolVLA's defaults, where the vision encoder is frozen — the model's "eyes"
stayed locked to their pretrained state and could not adapt to our bricks, lighting and
camera angles. V6 unfroze it, added image augmentation and AMP, and doubled the step
count from 100k to 200k.
A caveat on freeze_vision_encoder=false
Unfreezing raised trainable parameters only modestly: the run logs report
num_learnable_params=99880992 of num_total_params=450046176 — the same ~100M as the
frozen V5 run. SmolVLA is a SmolVLM2-500M backbone plus a smaller action expert, and
most of the backbone stays frozen regardless of this flag. Do not expect this setting
alone to make all 450M parameters trainable.
Usage
[!CAUTION]
This model expects renamed camera keys. Training used
--rename_map to remap the dataset's camera features:
Dataset feature
What the policy expects
Physical camera
observation.images.wrist_left
observation.images.camera1
wrist
observation.images.desk_view
observation.images.camera2
desk
If you feed this policy wrist_left / desk_view it will fail or silently misbehave.
Name your cameras camera1 (wrist) and camera2 (desk) at inference time, or apply the
same --rename_map when re-training. The Pi0.5 models do not do this — they use the
native wrist_left / desk_view names.
[!IMPORTANT]
Model files are nested under outputs_V6/, so from_pretrained on the repo ID
will not work:
Use one of the exact training prompts (see the dataset card) as the task string.
Both cameras run at 640x480 at inference time even though desk_view was
recorded at 800x600.
Robot setup
Robot
SO-101 follower arm (6-DOF), robot_type: so_follower
Inference note: both cameras are run at 640x480 during inference, not at their
recording resolutions, to reduce the payload sent to the inference server.
Environment notes
All training ran on a SLURM cluster with L40S GPUs. Two environment details were required
and are easy to miss when reproducing:
ffmpeg libraries for torchcodec. A minimal conda env supplies the shared libraries
that torchcodec discovers at runtime:
export LD_LIBRARY_PATH=$CONDA_PREFIX/envs/ffmpeg_libs_v8/lib:<venv>/lib/python3.12/site-packages/nvidia/npp/lib:$LD_LIBRARY_PATH
--tolerance_s=0.01 on every run, to accommodate timestamp jitter in the recorded
episodes.
Multi-GPU runs additionally set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
Datasets and the virtualenv were copied to node-local /scratch before training rather
than read from shared storage.
No Weights & Biases logging was enabled for any run (--wandb.enable=false), so there are
no public training curves — the job.*.err SLURM logs are the record.
Tasks and prompts
The model is conditioned on English natural-language instructions. Prompt phrasing was
varied roughly every 10 episodes during recording, giving 93 distinct prompts in the
merged dataset. Use one of the training prompts verbatim for best results — the full
lists are on the dataset card.
Limitations
Behaviour cloning. The policy imitates teleoperated demonstrations and has no notion
of recovery beyond what was demonstrated. It is susceptible to covariate shift and can
fail to recover from states outside the demonstration distribution.
Recovery data is incidental, not systematic. Recovery behaviour appears in the data
only where the operator happened to make and correct a mistake during recording; no
recovery episodes were scripted deliberately.
Single environment. All data comes from one lab desk with one lighting setup, one
camera geometry, and one set of physical objects. Expect degradation elsewhere.
Prompt sensitivity. Language conditioning was trained on a fixed set of phrasings
(listed in the dataset card). Prompts far from those phrasings may behave unpredictably.
No formal evaluation. Quality assessments below are qualitative, from operators
observing rollouts on the physical arm. There are no success-rate numbers.
Not safety-rated. Supervise all physical execution and keep the workspace clear.
Upstream licensing & attribution
This model is a derivative work of Apache-2.0 licensed components:
Apache-2.0 permits relicensing derivative works. We retain the upstream copyright
notices, license text, and NOTICE files for the incorporated material, as Apache-2.0
Section 4 requires. The upstream components remain under Apache-2.0 — only this
project's own contributions (the fine-tuned weights and training configuration) are
offered under CC BY-SA 4.0.
CC BY-SA 4.0 was chosen because it is share-alike: derivatives must be released under
the same licence, so this work cannot be taken closed-source. The project's source code
lives in a separate repository under its own licence — see
https://github.com/Project-IRA/interactive-robotic-arm.
Citation
bibtex
1@misc{project_ira_2026,
2 title = {Project-IRA: Interactive Robotic Arm},
3 author = {Baten, Cleo and Keppler, Bela and Sapper, Jonas},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/Project-IRA}},
6 note = {Code: \url{https://github.com/Project-IRA/interactive-robotic-arm}}
7}
