mm_SO101_teleop — SmolVLA on SO-101: teleop-only vs 50/50 ego co-training

Two SmolVLA fine-tunes of lerobot/smolvla_base on the same SO-101 pick-and-place task ("pick up blue cube and place it in orange box"), differing only in the training mixture, plus the full offline eval suite used to compare them.

_{Alternate view: winner's-margin diverging chart}

	teleop100	mix5050
training data	193 teleop eps / 54,275 frames	130 teleop + 324 ego eps / 72,986 frames
mixture	100 % real teleop	50.1 % teleop / 49.9 % ego by frames
tasks	2	91 (2 teleop + 89 ego)
steps	20,000	20,000
effective batch	64 (16/GPU × 4× RTX 4090)	64 (16/GPU × 4× RTX 4090)
final train loss	0.047	0.044
held-out val loss	0.3861 (min 0.2141 @ step 4000)	not measured in-run — see note

Both start from lerobot/smolvla_base with identical hyperparameters, taken verbatim from makermods/smolvla_200ep_blue_cube_orange_box: chunk 50, lr 1e-4, cosine decay 20k with 1k warmup, frozen vision encoder, train_expert_only, bf16. 100M of 450M parameters are trainable.

Results (6 held-out episodes / 1,622 frames, never trained on)

Errors are in native degrees — the checkpoints' postprocessors un-normalize, so the two models are directly comparable despite different normalization statistics.

metric	teleop100	mix5050	better
action MSE (clean)	146.83	142.82	mix, −2.7 %
action MAE (clean)	5.840°	6.194°	teleop, −5.7 %
CI-MSE mean (critical intervals)	50.28	46.70	mix, −7.1 %
CI-MSE median	23.75	30.53	teleop, −22.2 %
CI-MSE grasp interval	60.43	52.19	mix, −13.6 %
CI-MSE release interval	38.09	43.12	teleop, −11.7 %
CI-MSE full episode, mean	59.96	48.82	mix, −18.6 %
CI-MSE full episode, median	21.43	24.26	teleop, −11.7 %
perturbation MAE penalty	+32.4 %	+35.4 %	teleop
perturbation MSE penalty	+30.2 %	+42.6 %	teleop

The headline is a mean/median split. Ego co-training lowers every mean error and raises every median error. Means are tail-driven, so the mixture trades typical precision for fewer catastrophic frames — it makes the model's worst moments less bad and its ordinary moments slightly worse. The clearest single effect is on grasp (−13.6 %), the phase where ego demonstrations carry the most transferable information; release gets worse (+11.7 %).

Neither model is "better" outright on 6 episodes. See REPORT.md for the caveats, which are substantial.

Repo layout

teleop100/   step-20000 checkpoint (teleop-only)
mix5050/     step-20000 checkpoint (50/50 ego+teleop)
configs/     the train configs actually used, and the episode selections
data/val/    the 6-episode LeRobot v3.0 holdout (the ONLY data shipped here)
eval/        the full eval suite, ported to SmolVLA + lerobot v3.0
logs/        training and eval logs
metrics/     results.json, loss curves, CI-MSE summaries, paired perturbation tables

Reproducing the evals

pip install 'lerobot[smolvla,training]==0.6.0' h5py scipy dtw-python fastdtw zarr

EVAL_DIR=eval VAL_ROOT=data/val \
ANNOT=eval/annotations/test_zero_shot_gemini-flash-lite-latest.json \
MANIFEST=eval/annotations/perturbation_manifest_all.json \
  eval/run_evals.sh teleop100 out/teleop100 teleop100

Inference is seeded per dataset row (--seed, default 1000). This matters: SmolVLA integrates flow matching from random noise, and three unseeded repeats of the same checkpoint gave 146.83 / 147.28 / 148.98 MSE — a ~1.5 % spread, the same order as the teleop-vs-mix difference. Seeded, results are exact and independent of how episodes are sharded across GPUs.

Note on the mixture's missing val curve

lerobot's dataset.eval_split holds out the last ceil(n × split) episodes of each task. The mixture has 91 tasks, so any split > 0 would hold out ≥ 91 episodes rather than 6. The mixture therefore trained with in-run eval disabled, and both models were scored afterwards by the same offline suite against the same 6 clips — which makes the comparison more paired, not less.

Provenance

• Base: lerobot/smolvla_base; hyperparameters from makermods/smolvla_200ep_blue_cube_orange_box
• Teleop: makermods/200ep_blue_cube_orange_box (199 eps, real SO-101 teleop, 30 fps)
• Ego: angkul07/ego-data (EgoDex) retargeted to SO-101 via the DT-pipeline stage-6 run F
• Eval suite: fd-studio/eval (offline proxy, perturbation, CI-MSE), ported here