SmolVLA — SO-101 left arm, 11-task multitask

SmolVLA finetuned on the 11 datasets of the hungho77/SO101_LeftArmOnly_Task collection, merged into one multitask set. One checkpoint covers all 11 tasks; the language instruction selects the behaviour.

Tasks

Pass one of these strings verbatim as the task; the model has not been trained on paraphrases.

`task_index`	instruction	episodes
0	`Pick up the red cube and place it into the steel pot.`	50
1	`Pick up the blue cube and place it into the steel pot`	50
2	`Take the banana and place it into the steel pot then close the lid.`	50
3	`Pick up the blue cube and place it next to the blue cube.`	50
4	`Take the blue cube out of the steel pot and place it on the blue cube.`	51
5	`Take the red cube out of the steel pot and place it on the red cube`	50
6	`Stack the yellow cube on top of the orange cube.`	50
7	`Pick up one orange block and stack it neatly on top of the other orange block`	50
8	`Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.`	50
9	`Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board`	50
10	`Take out the blue cubes from shortest to tallest and place it on the table`	50

Note the inconsistent trailing periods: they are part of the strings as recorded and must be reproduced exactly. These are the exact strings from meta/tasks.jsonl.

Robot

SO-101 follower arm, left arm only, 5 joints plus gripper, 30 fps.


State / action dim	6 — `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll` + `gripper`
Cameras	`top` (overhead, 360×640) → `camera1`, `wrist` (480×640) → `camera2`
Action chunk	50
Absolute action std	28.67 (mean over dims)

Camera renaming is required. smolvla_base declares camera1/camera2/camera3 while the dataset names its cameras top and wrist. Training used:

--rename_map='{"observation.images.top":"observation.images.camera1",
               "observation.images.wrist":"observation.images.camera2"}'

Without it, lerobot stops with Feature mismatch between dataset/environment and policy config. Inference must apply the same mapping.

Training


Base	`lerobot/smolvla_base`
Data	551 episodes / 253,745 frames / 11 tasks
Steps × batch	20,000 × 128 = 2.56M samples = 10.1 epochs
Learnable params	100M of 450M — `freeze_vision_encoder` and `train_expert_only` are on
LR	1e-4, lerobot default schedule
Hardware	one H100 MIG 3g.40gb slice (39.5 GiB), peak 27.3 GB
Runtime	4.6 h at 0.835 s/step, 153 samples/s
Final loss	0.050

Freezing the VLM is what makes this fit a 40 GB slice at batch 128 where 3B models cannot: no activations are stored for the frozen tower, so the cost is 0.19 GB per sample against 0.82 GB for X-VLA and ~0.36 GB for π₀.₅.

Open-loop evaluation

One trajectory per task, walked in chunk-length strides, scored over the first 16 of the 50 predicted steps. Errors are in the dataset's own action units.

traj	task	MAE
0	red cube → pot	1.592
50	blue cube → pot	1.369
100	banana → pot, close lid	1.905
150	blue cube next to blue cube	2.430
200	blue cube out of pot	1.854
251	red cube out of pot	1.784
301	stack yellow on orange	1.950
351	stack orange blocks	2.580
401	balance red/blue on board	1.881
451	red cube → hollow cube	1.834
501	blue cubes shortest→tallest	2.029
	average	1.928

Dividing by the absolute action std (28.67) gives 6.72% of action scale.

Error grows along a chunk, so a horizon-50 number must never be compared against a horizon-16 one from another model. Use the absolute action std, not the checkpoint's normalizer statistics, which are a different quantity.

Limitations

No held-out split. All 551 episodes were used for training, so the numbers above are measured on training data. They show the policy fits its data and is not degenerate; they do not measure generalization and are not a success rate. Real-robot performance is unmeasured.

Scoring only three trajectories gives 5.17% — the first three tasks happen to be the easiest. The 11-trajectory figure of 6.72% is the one to quote.

Deployment note

The data is 30 fps. A client interpolating between model steps must derive its sub-step count from that: smooth_step = control_hz / 30. Reusing a value from a 15 fps robot stretches every trajectory by 2× and the arm creeps without finishing the task.