Twu31/rt1-lerobot-mps

RT-1 on LeRobot v3 datasets — a controlled head-to-head against SmolVLA

Five RT-1 checkpoints trained on LeRobot v3 datasets on an M5 MacBook (Apple MPS, no NVIDIA GPU), using the lucidrains PyTorch port of Google's RT-1 (Brohan et al., 2022).

These exist to isolate one variable. Each dataset here also has a SmolVLA checkpoint in Twu31/smolvla-cross-embodiment-mps trained on the same data, the same hardware, and evaluated by the same script — so the difference between the two is the architecture: discrete 256-bin action tokens versus continuous flow matching, and a from-scratch MaxViT backbone versus a pretrained SmolVLM2 one.

Code: https://github.com/twu3202/SmolVLA_ALOHA (RT1_repro/)

---

Results

Open-loop L2 against held-out episodes, native action units.

Dataset	SmolVLA L2	RT-1 L2	Δ	Winner
aloha_transfer	0.729	0.589	−19%	RT-1
aloha_insertion	0.716	0.744	+4%	SmolVLA (tie)
aloha_static_battery ★real	0.651	0.394	−40%	RT-1
xarm_push (3-DOF, no gripper)	0.216	0.123	−43%	RT-1
xarm_lift (4-DOF, with gripper)	1.324	1.890	+43%	SmolVLA

3–2 to RT-1, and the split follows a rule rather than noise:

• RT-1 wins on stereotyped continuous control. On aloha_static_battery its left-arm per-dim MAE is 0.006–0.078 against SmolVLA's 0.025–0.274 — up to 10× lower. On xarm_push it is a flat 0.055–0.065 against 0.064–0.159. 256 bins quantise a tremor-stable demonstrator's trajectory almost losslessly.
• RT-1 collapses on the hard task. On xarm_lift its per-dim MAE is 0.70/0.83/0.81/0.80 against SmolVLA's 0.53/0.58/0.59/0.56 — uniformly 30–40% worse on every dimension, not just the gripper. With 2000 steps and no pretrained vision-language backbone it cannot learn the tighter dynamics of 84×84 images plus a binary gripper plus a wide z-range. The 206 M parameter gap buys robustness precisely when the task is genuinely hard.
• Precision contact is a wash. Peg insertion needs sub-bin angular precision; SmolVLA wins by 4%, which is noise.
• The gripper-binary problem is architecture-independent. Both models' worst dimension on every ALOHA task is the right-arm gripper (dim 13): RT-1 MAE 0.13–0.24, SmolVLA 0.14–0.27. Transitions are ~1% of frames — a data-side bottleneck neither action head escapes.

Regime	Better architecture
Easy, stereotyped continuous task, small data	RT-1 / discrete actions (−19% to −43%)
Precision contact in sim	SmolVLA, barely (+4%)
Hard task, low-resolution images	SmolVLA / pretrained VLM (+43%)
Binary signals (gripper)	neither — data problem

---

Contents

<dataset>/step_002000.pt      # RT-1 weights + training log + config
<dataset>/action_stats.json   # per-dim action min/max and bin count — required to decode actions
eval_output/                  # per-dataset .npz + .png, and the head-to-head figure

Checkpoint	Action dim	Final-step loss (256-way CE)
aloha_transfer	14	3.837
aloha_insertion	14	4.562
aloha_static_battery ★real	14	3.597
xarm_push	3	3.964
xarm_lift	4	4.879

Cross-entropy over 256 bins; chance is ln(256) ≈ 5.545.

---

Model

Aspect	This RT-1	SmolVLA (companion repo)
Params	244.0 M	450.0 M
Backbone	MaxViT (dim=96, depth=(2,2,5,2), window 7)	SmolVLM2-500M, pretrained
Trunk	6-layer Transformer, 8 heads, Token Learner	flow-matching action expert
Action head	discrete, 256 bins/dim, cross-entropy	continuous, 10-step denoising
Observation history	6 frames	1 frame
Image size	224×224	512×512 (padded resize)
Training	2000 steps, batch 4	3000 steps, batch 4–16
Pretrained weights	none (CLIP text encoder frozen)	SmolVLM2 backbone

Parameter breakdown: conditioner 120.0 M · MaxViT 80.1 M · Transformer 33.7 M · Token Learner 9.5 M · output logits 0.8 M. cond_drop_prob=0.2. Weights are float32.

Everything except the frozen CLIP text encoder is trained from random initialisation — this is a reproduction of the architecture, not of Google's 130k-episode RT-1 model, and it should not be expected to behave like the published one.

---

Usage

import torch, json
from huggingface_hub import hf_hub_download

REPO = "Twu31/rt1-lerobot-mps"
ck = torch.load(hf_hub_download(REPO, "aloha_static_battery/step_002000.pt"),
                map_location="cpu", weights_only=False)

ck["step"]      # 2000
ck["loss"]      # final-step cross-entropy
ck["dataset"]   # "aloha_static_battery"
ck["cfg"]       # image/state/action keys, dims, fps
ck["loss_log"]  # per-step loss
ck["model"]     # RT1 state_dict, 694 tensors, float32
ck["act_min"], ck["act_max"]   # per-dim action range used for binning

Rebuilding:

from robotic_transformer_pytorch import RT1, MaxViT
import numpy as np

vit = MaxViT(num_classes=1000, dim_conv_stem=64, dim=96, dim_head=32,
             depth=(2, 2, 5, 2), window_size=7, mbconv_expansion_rate=4)
model = RT1(vit=vit, num_actions=ck["cfg"]["action_dim"], action_bins=256,
            depth=6, heads=8, dim_head=64, cond_drop_prob=0.2)
model.load_state_dict(ck["model"])

# logits -> (B, T, action_dim, 256); decode bin centres back to action units
bins  = logits.argmax(-1).float() / 255.0
action = np.array(ck["act_min"]) + bins * (np.array(ck["act_max"]) - np.array(ck["act_min"]))

Inputs are 6 stacked 224×224 RGB frames plus a task description string. pip install robotic-transformer-pytorch tiktoken sentencepiece.

---

Training data

Five public LeRobot v3 datasets (listed in the metadata above), not redistributed here — training auto-downloads them. The RT-1 trainer reuses dataset_configs.py from the SmolVLA side of the project, so both arms see byte-identical data and splits.

One exception, so it is on the record: each eval_output/eval_rt1_*.npz stores a gts array alongside preds — the ground-truth action vectors for the held-out frames the evaluation ran on (160–4,760 rows of 3–14 floats per dataset, ~2 MB in total). No images, no video, no other observations. They are included so the reported L2 and per-dim MAE can be recomputed and checked independently rather than taken on trust; the source datasets are Apache-2.0.

---

Limitations

• Open-loop evaluation only — next-action prediction error against recorded demos. No simulator rollouts and no success rates, so "RT-1 wins" means lower prediction error, not better task completion.
• Five datasets, one seed each, 2000 steps. A 3–2 split on five tasks is a thin margin; the pattern (stereotyped vs hard) is the claim, not the score.
• RT-1 got 2000 steps against SmolVLA's 3000. The step budgets were matched to wall-clock, not to convergence, so some of the gap on xarm_lift may be undertraining rather than architecture.
• This is the lucidrains port at small scale, not Google's RT-1 model or its data.

---

Related releases

Repo	What it holds
Twu31/smolvla-cross-embodiment-mps	The SmolVLA half of this comparison, plus 11 more datasets
Twu31/smolvla-libero-eeg	SmolVLA + LIBERO + EEG as a fourth modality
Twu31/so101_hand_blue_napkin	Real SO-ARM101 handover demonstrations

Combined, the two model repos are 693 M parameters of policies trained on one MacBook in about 20 hours — the entry barrier to VLA research is lower than it is usually assumed to be.

---

Citation

@software{rt1_lerobot_mps_2026,
  author = {Twu31},
  title  = {RT-1 on LeRobot v3 datasets: a controlled architecture comparison against
            SmolVLA on Apple MPS},
  year   = {2026},
  url    = {https://github.com/twu3202/SmolVLA_ALOHA}
}

RT-1: Brohan et al., RT-1: Robotics Transformer for Real-World Control at Scale, 2022. PyTorch port by lucidrains (MIT).