zero-runtime/smolvla_so101_pick_place_bowl

SmolVLA: SO-101 pick & place (orange cube → white bowl)

A SmolVLA policy that makes an SO-101 arm pick up an orange cube and place it in a white bowl, from three cameras and the arm's joint state.

It was fine-tuned on zero-runtime/so101_pick_place_bowl: 105 demonstrations teleoperated over the network with ZeroRuntime and recorded in the cloud. ZeroRuntime also runs the policy: the model runs on any GPU (a cloud box, a notebook, a Jetson or a laptop) and drives the arm over the network. Run it ↓

Base model	lerobot/smolvla_base (SmolVLM2-500M, 16 VLM layers + action expert)
Parameters	~450M, mixed bf16 / fp32
Fine-tuned	action expert only (~100M); vision encoder and VLM frozen
Task text	pick up the orange cube and place it in the white bowl
Control rate	10 Hz
Action chunk	50 actions = 5 s of motion
Robot	SO-101 follower (so101_follower), single arm, 6-DoF
LeRobot version	0.6.1
License	Apache-2.0

Inputs and outputs

Task text: the sentence above, word for word. It is the only task the policy was trained on.

State: observation.state, float32[6], in SO-101 joint order:

shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos

The joint numbers are not in degrees. Each joint goes from −100 to 100, and the gripper goes from 0 (closed) to 100 (open). A value can go a little past 100, which is normal.

Cameras: three RGB cameras. SmolVLA's base model expects cameras named camera1 to camera3, so training renamed them:

Policy input	Camera	View
observation.images.camera1	observation.images.top	top-down
observation.images.camera2	observation.images.wrist	gripper / wrist
observation.images.camera3	observation.images.side	side

Send full 640×480 frames; the policy resizes them itself. Swapping two cameras raises no error, the arm simply behaves badly, so check the camera order first.

Output: action, float32[6], the joint positions to move to, in the same order and units as the state. Each inference returns 50 actions, played at 10 Hz.

Normalization is stored in the policy_preprocessor* and policy_postprocessor* files and matches the dataset.

Run it with ZeroRuntime

The arm runs the same follower that recorded the dataset. The model runs as a leader on any machine with a GPU. The follower treats it like a human operator, so every command still passes through its speed clamps, watchdog and deadman.

 SO-101 + 3 cameras ── follower.py ──── ZeroRuntime room ──── smolvla_leader.py ── SmolVLA
 (robot)               clamps · watchdog · deadman              (any GPU, anywhere)

The leader:

• downloads this model from the Hub and reads the camera names, joint order and rate from the model and its dataset;
• moves the arm smoothly to the pose the demonstrations started from, then hands over to the policy;
• runs inference while the arm moves, so a slow GPU never makes the arm stop and wait;
• checks your machine first: --check times inference and tells you if it is fast enough.

Both scripts are in the examples repo under inference/, with the setup guide in the docs.

1. On the robot, set the camera names at the top of follower.py to the ones used for recording, then start it:

CAMERA_LABELS = ["top", "wrist", "side"]

python follower.py

2. On the GPU machine. This model and its dataset are gated: open both pages, accept the conditions, then log in.

hf auth login
pip install "lerobot[smolvla]==0.6.1" "zeroruntime-teleops[lerobot]" python-dotenv opencv-python-headless

python smolvla_leader.py \
  --policy zero-runtime/smolvla_so101_pick_place_bowl \
  --dataset zero-runtime/so101_pick_place_bowl \
  --task "pick up the orange cube and place it in the white bowl" \
  --check

--check needs no robot. When it says the machine is fast enough, run the same command without --check to drive the arm. On a Jetson or a laptop, add --num-steps 5.

Press Ctrl-C once to stop. The leader lets go of control and the arm holds its pose.

It works best on a setup like the one it was recorded on: same camera views, similar placement and lighting, an orange cube and a white bowl. A different scene or task needs new demonstrations and a new fine-tune.

Training

Setting	Value
Start from	lerobot/smolvla_base
Dataset	zero-runtime/so101_pick_place_bowl: 105 episodes, 21,406 frames, 10 fps
Steps	20,000
Batch size	16 (8 per GPU × 2 GPUs)
Fine-tuned	action expert (train_expert_only=true, freeze_vision_encoder=true)
Optimizer	AdamW, β = (0.9, 0.95), weight decay 1e-10, gradient clip 10
Learning rate	1e-4, 1,000 warm-up steps, cosine decay to 2.5e-6
Precision	fp32
Augmentation	LeRobot image augmentation on
Seed	1000
Hardware	2 × NVIDIA T4, about 20 GPU-hours

lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --policy.train_expert_only=true \
  --policy.freeze_vision_encoder=true \
  --dataset.repo_id=zero-runtime/so101_pick_place_bowl \
  --dataset.image_transforms.enable=true \
  --rename_map='{"observation.images.top": "observation.images.camera1", "observation.images.wrist": "observation.images.camera2", "observation.images.side": "observation.images.camera3"}' \
  --batch_size=16 --steps=20000 --seed=1000 \
  --output_dir=outputs/smolvla_so101_pick_place_bowl

This is the single-GPU form. The original run used accelerate launch --multi_gpu --num_processes=2 with --batch_size=8, which is the same total batch.

Safety

This policy moves a real robot. Watch every run, keep an emergency stop within reach, and keep people and fragile objects out of the workspace. With ZeroRuntime the leader limits joint speed before sending, and the follower applies its own speed clamps, a watchdog that holds the arm when commands stop, and a deadman. None of these depend on the model.

Citation

@misc{zeroruntime2026smolvlaso101pickplacebowl,
  title  = {SmolVLA for SO-101 pick and place (orange cube to white bowl)},
  author = {ZeroRuntime},
  year   = {2026},
  url    = {https://huggingface.co/zero-runtime/smolvla_so101_pick_place_bowl}
}
}