You can deploy this model and start earning money today!

danilodjor/
smolvla-eval2-merged-auxHead-bs64-step50k

Name: smolvla-eval2-merged-auxHead-bs64-step50k
Availability: InStock

danilodjor/SO101-eval2-merged-25prompts

lerobot/smolvla_base

finetune

apache-2.0

Views

No views yet

SmolVLA + Slot-Aux Head — SO101 Eval2 (25 prompts)

Fine-tune of lerobot/smolvla_base on the danilodjor/SO101-eval2-merged-25prompts dataset, trained jointly with an auxiliary 3-way "slot" classifier that reads off SmolVLA's cross-modal prefix representation.

The auxiliary head exists only to shape the shared VLM trunk during training. It is not used at inference — the released checkpoint can be loaded as a plain SmolVLA.

TL;DR

Base policy: lerobot/smolvla_base (vision-language-action, flow-matching action expert).
Dataset: SO101 teleop demonstrations spanning 27 task variants in 4 families (colour, ordinal, relative, negation).
Training objective: total = action_loss + slot_loss_weight * slot_loss with slot_loss_weight = [0.5].
At inference: drop-in compatible with vanilla SmolVLA. Bit-equivalent to the unwrapped policy (verified allclose, atol=1e-5).

Intended use

Closed-loop control of an SO-101 arm on the eval2 tabletop task suite, driven by a natural-language instruction and a front camera.
Research baseline for studying whether auxiliary supervisory signals on the VLM prefix improve compositional generalisation across the four prompt families.

Not intended for: deployment outside the SO-101 hardware setup, safety-critical control, or any task outside the training distribution.

Architecture

              image + instruction + state
                        │   
                        ▼   
            ┌──────────────────────┐
            │   SmolVLA VLM trunk  │  ← updated by BOTH losses
            └──────────────────────┘
                 │              │
        prefix_out            suffix_out
                 │              │
                 ▼              ▼
         ┌────────────┐  ┌────────────────┐
         │ slot head  │  │ action expert  │  ← updated by action_loss
         │ (aux only) │  │ (flow matching)│
         └────────────┘  └────────────────┘
            3-way CE        action chunk
        (train only,        (kept at
        dropped at infer)    inference)

The slot head is a small MLP (Linear(hidden → 256) → GELU → Dropout → Linear(256 → 3)) applied to a mean-pooled prefix_out over the valid (image + text + state) tokens.

Gradient flow during training:

Component	`action_loss`	`slot_loss`
VLM trunk (shared)	✅	✅
Action expert	✅	❌
Slot head MLP	❌	✅

The slot head is throwaway — its purpose is to push the shared trunk to encode slot identity legibly. At inference the head is bypassed entirely.

Training details

Hyperparameter	Value
Base policy	`lerobot/smolvla_base`
Optimizer	AdamW
Learning rate	1e-4
Weight decay	0.01
Grad clip	10.0
Batch size	64
Steps	60 000
Slot-loss weight	0.5
Aux hidden dim	256
Aux dropout	0.1
Seed	1000
Precision	torch.amp.autocast
Hardware	[single GPU — e.g. 1× H100]
Wall time	[fill in]
Weighted sampler	[yes/no]

Slot labels are sourced from meta/episode_target_slots.parquet. Task families (by task_index):
colour 0–2 · ordinal 3–14 · relative 15–18, 25, 26 · negation 19–24.

How to use

The checkpoint is saved as a standard SmolVLA folder, so you do not need the wrapper code to run inference:

python

from lerobot.policies.factory import make_policy
from lerobot.configs.policies import PreTrainedConfig

cfg = PreTrainedConfig.from_pretrained("danilodjor/<this-repo>")
policy = make_policy(cfg=cfg, ds_meta=...)  # standard lerobot loading
actions = policy.predict_action_chunk(batch)

For closed-loop rollout on hardware, use the standard lerobot-record CLI.

If you also want to inspect the auxiliary slot predictions, load the aux.pt
sidecar with the wrapper from 
euler_scripts/smolvla-aux-head/src/aux_head.py.

Files in this repository

- config.json, model.safetensors, processor configs — standard SmolVLA checkpoint.
- aux.pt (optional) — slot-head weights + optimizer state + training args, for
reproducibility and analysis only. Not loaded at inference.

Limitations and biases 

- Trained on a single tabletop scene with a fixed camera; expect degradation under
lighting / viewpoint / object shifts.
- The 3-way slot vocabulary is specific to this dataset's task structure and not
meaningful outside it.
- Auxiliary supervision reshapes the shared trunk; the resulting weights are not
identical to a vanilla SmolVLA fine-tune even though the loading path is.

danilodjor/smolvla-eval2-merged-auxHead-bs64-step50k