Views
No views yet
smolvla_base policy trained with the lerobot framework.smolvlagribok201/smolvla_koch4HuggingFaceTB/SmolVLM2-500M-Video-Instruct10000observation.image: type VISUAL, shape [3, 256, 256]observation.image2: type VISUAL, shape [3, 256, 256]observation.image3: type VISUAL, shape [3, 256, 256]observation.state: type STATE, shape [6]action: type ACTION, shape [6][512, 512] before being passed to the model.transformers.AutoModel with trust_remote_code=True.
You MUST have lerobot installed in your environment for this to work.
(pip install lerobot)1from transformers import AutoModel
2import torch
3from PIL import Image
4import torchvision.transforms as T
5
6# Replace with your model's repo_id
7repo_id = "Infatoshi/smolvla"
8
9# Load the model - CRITICAL: trust_remote_code=True
10# This executes the custom code in modeling_lerobot_policy.py
11model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
12model.eval()
13
14print("Model loaded successfully!")
15
16# Example Inference:
17# Create dummy inputs matching the model's expected schema.
18resize_shape = tuple(model.config.resize_imgs_with_padding)
19state_shape = tuple(model.config.input_features["observation.state"]["shape"])
20
21# Dummy observations dictionary
22dummy_observations = {
23 "state": torch.randn(1, *state_shape),
24 "images": {
25 "usb": torch.randn(1, 3, *resize_shape),
26 "brio": torch.randn(1, 3, *resize_shape),
27 }
28}
29dummy_language_instruction = "pick up the cube"
30
31with torch.no_grad():
32 output = model(
33 observations=dummy_observations,
34 language_instruction=dummy_language_instruction
35 )
36
37print("Inference output (predicted actions):", output)
38print("Output shape:", output.shape)