| --- |
| license: apache-2.0 |
| base_model: lerobot/smolvla_base |
| tags: |
| - robotics |
| - lerobot |
| - smolvla |
| - so100 |
| - so101 |
| - vla |
| - imitation-learning |
| pipeline_tag: robotics |
| --- |
| |
| # SmolVLA fine-tuned on SO-100 tic-tac-toe (pick-and-place) |
|
|
| Two full fine-tunes of [`lerobot/smolvla_base`](https://huggingface.co/lerobot/smolvla_base) (450M) |
| on a LeRobot **SO-100 "tic-tac-toe"** dataset — 180 episodes, 9 pick-and-place tasks |
| ("place the blue cross cube in the `<cell>` box"), top + wrist cameras (640×480 @30 fps), 6-DOF |
| joint actions. Seed-fixed split: 144 train / 18 val / 18 test. |
|
|
| Camera mapping used in training: `top → observation.images.camera1`, `wrist → observation.images.camera2`. |
|
|
| ### A note on the third camera |
|
|
| The underlying dataset has **only two cameras** (`top`, `wrist`). The model config, however, lists |
| **three** image inputs (`camera1`, `camera2`, `camera3`) — this 3-view layout is inherited from the |
| `lerobot/smolvla_base` pretrained model, **not** from the data: |
|
|
| | config input | source | content | |
| |---|---|---| |
| | `observation.images.camera1` | dataset `top` view | real | |
| | `observation.images.camera2` | dataset `wrist` view | real | |
| | `observation.images.camera3` | — | **padded empty/black image** (no information) | |
|
|
| `camera3` is a placeholder kept only to match the base model's expected input shape; SmolVLA tolerates |
| the missing view by feeding it a blank image. (Note: the saved `config.json` shows `empty_cameras: 0` |
| because the slot comes straight from the base model's `input_features` rather than being added as an |
| explicit empty camera — behaviour is unchanged either way.) |
|
|
| **For inference you only need to provide the two real views** (`camera1`=top, `camera2`=wrist), |
| `observation.state`, and the task string — the third slot is padded automatically. |
|
|
| ## Checkpoints |
|
|
| | folder | training data | eval | major-4 joint corr | |
| |---|---|---|---| |
| | `in_distribution/` | all 9 cells, 20k steps (loss 0.009) | all 9 cells (in-distribution fit) | **0.983** (per-joint all >0.96, MAE ~1.8°) | |
| | `heldout_cells/` | 6 cells, 15k steps (best ckpt) | 3 **held-out** cells (top-left / center / middle-right) | **0.928** (MAE 3.2°) | |
|
|
| The held-out-cell run demonstrates that a pretrained VLA **generalizes the pick-place skill across |
| board positions** (0.928 on unseen cells vs 0.983 in-distribution) — far above a bespoke Cosmos3 |
| action policy's 0.715 on *seen* cells. Generalization climbs with training (no overfit collapse). |
|
|
| ## Usage |
|
|
| Each folder is a standard LeRobot SmolVLA `pretrained_model/` (model + pre/post-processors). Load with |
| LeRobot 0.4.4+: |
|
|
| ```python |
| from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy |
| policy = SmolVLAPolicy.from_pretrained("azhicles/smolvla-so100-tictactoe", subfolder="in_distribution") |
| ``` |
|
|
| Provide observations as `observation.images.camera1` (top), `observation.images.camera2` (wrist), |
| `observation.state` (6-DOF), and the task language string; the policy returns a chunked 6-DOF action. |
|
|
| ## Inference speed |
|
|
| ~4.70 s / episode (16 chunked forwards @ 0.284 s) on 1× NVIDIA GB300 — faster *and* more accurate |
| than the bespoke Cosmos3 action policy baseline. |
|
|
