twanghcmut/X-VLA-SO101-LeftArm-Multitask

X-VLA fine-tuned on SO-101 left arm, 11 tasks

2toINF/X-VLA-Pt (Florence-2-large encoder + soft-prompted flow-matching action transformer, 0.9B) fine-tuned with the official X-VLA repository (commit 6bc2513) on hungho77/so101-left-arm-multitask: 551 episodes, 253,745 frames at 30 fps, 11 tabletop tasks, one SO-101 left arm, cameras top (360x640) and wrist (480x640), 6-dim state and action (5 joints in degrees + gripper aperture, continuous 0.7-52.7).

Tasks

Pass the task string verbatim, trailing period included where present.

id	task
0	Pick up the red cube and place it into the steel pot.
1	Pick up the blue cube and place it into the steel pot
2	Take the banana and place it into the steel pot then close the lid.
3	Pick up the blue cube and place it next to the blue cube.
4	Take the blue cube out of the steel pot and place it on the blue cube.
5	Take the red cube out of the steel pot and place it on the red cube
6	Stack the yellow cube on top of the orange cube.
7	Pick up one orange block and stack it neatly on top of the other orange block
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board
10	Take out the blue cubes from shortest to tallest and place it on the table

How the SO-101 data enters X-VLA

Everything below is in so101/ in this repo (xvla_so101_upstream.patch for the three upstream files, so101.py for the handler, config_auto_so101.json, the launcher and the evaluation script).

• Action space auto with real_action_dim 6, max_action_dim 20 (config.json). The model keeps its pretrained 20-dim action layers, the loss (MSE x 100) covers the 6 real dims, nothing is sigmoided: the gripper is a continuous aperture. The pretrained checkpoint ships with action_mode ee6d (end-effector, metres); its action layers are reused as they are.
• Handler (so_follower, domain id 19): cameras top, wrist, a zero third view; one action column; joints 0-4 are predicted as deltas from the proprio, the gripper as an absolute target. Actions are min-max normalized to [-1, 1] with the dataset's meta/stats.json -- the upstream handlers emit metres and never normalize, and raw degrees gave a first-step loss of 163,347. In training 1 frame in 8 is kept per episode visit, because the upstream loader yields whole episodes in sequence and a batch would otherwise come from a handful of episodes.
• Proprio is the previous action (action[t-1]), as in the upstream LeRobot v2.1 handler, not observation.state; feed the last commanded action at inference. The chunk is 30 steps over 1 s (qdur 1.0, 30 fps), i.e. actions t .. t+29.
• DaViT activation checkpointing: vision_config.enable_checkpoint exists in the Florence-2 config but DaViT.from_config did not forward it; the patch adds that one argument. Without it a 40 GB GPU fits 8 samples; with it, 16.

Training

Recipe	README fine-tuning command: accelerate launch --mixed_precision bf16 train.py --learning_rate 1e-4 --learning_coef 0.1 --freeze_steps 1000 --warmup_steps 2000
Batch	16 per GPU x 2 GPUs = 32 (no gradient accumulation in train.py)
Iterations	24,000 = 768k samples, the same sample budget as the pi0.5, GR00T N1.7, MolmoAct2 and RLDX-1 fine-tunes in this series, and close to the README's 50,000 x 16
Schedule	VLM and transformer core at LR 0 for 1,000 steps, linear warmup over 2,000, then constant (no cosine flag); VLM and soft prompts at 0.1x; AdamW betas (0.9, 0.95), weight decay 0, grad clip 1.0
Hardware	2 x H100 MIG 3g.40gb slices, DDP, 4 h 41 min (0.7 s/step)
Loss	6.2 at step 20, 0.74 at 2,000, 0.28 at 8,000, mean 0.158 over the last 1,000 steps (MSE x 100 in normalized units)

Evaluation (open loop, in-sample)

Protocol shared by every model in this series: eval_so101_open_loop.py on 11 training trajectories (one per task: episodes 0, 50, 100, 150, 200, 251, 301, 351, 401, 451, 501), the policy called every 16 steps from the dataset's own frames and previous action, the first 16 of the predicted 30 steps compared with the ground-truth actions, up to 400 steps per trajectory, 10 denoising steps, MAE in the dataset's units (degrees; gripper aperture). "Normalized" divides by the absolute action std of the dataset, 28.67. These trajectories were in the training set, so this measures fit, not generalization.

trajectory	task	MAE	MSE
0	0	2.359	14.19
50	1	1.710	8.43
100	2	2.967	32.26
150	3	3.217	30.05
200	4	1.966	11.30
251	5	2.390	27.26
301	6	2.184	13.71
351	7	2.639	19.78
401	8	2.348	13.95
451	9	1.803	8.69
501	10	2.199	13.18
average		2.344	17.53

Normalized MAE: 8.18% of the action scale. Trivial baselines on the same trajectories and protocol: holding the action at the chunk start for 16 steps gives 3.282 (11.45%), copying the current state 5.513 (19.23%). The error grows smoothly with the position in the chunk (MAE 0.63 at step 0, 1.75 at step 4, 2.58 at step 8, 3.47 at step 14), which is what a policy that tracks the trajectory and drifts with the horizon looks like.

Same protocol, same dataset, other models in this series (all in-sample):

model	normalized MAE	checkpoint
MolmoAct2	4.37%	twanghcmut/molmoact2-so101-left-arm-multitask
TurboVLA	4.62%	twanghcmut/TurboVLA-SO101-LeftArm-Multitask
SmolVLA	6.72%	twanghcmut/SmolVLA-SO101-LeftArm-Multitask
X-VLA (this)	8.18%
pi0.5	9.82%	twanghcmut/pi05-so101-left-arm-multitask
GR00T N1.7	10.27%	twanghcmut/gr00t-n1d7-so101-left-arm-multitask
RLDX-1	15.85%	twanghcmut/RLDX-1-SO101-LeftArm-Multitask

Caveats that apply to the whole series: in-sample numbers overstate real performance (held-out evaluation of SmolVLA and TurboVLA on this dataset showed errors about 3.5x their in-sample values), the budgets differ (SmolVLA and TurboVLA used 20,000 x 128 samples, the others 768k), and none of these numbers is a real-robot success rate. This checkpoint has not been run on the real robot.

Usage

git clone https://github.com/2toinf/X-VLA && cd X-VLA && git checkout 6bc2513   # install per its README
hf download twanghcmut/X-VLA-SO101-LeftArm-Multitask --local-dir X-VLA-SO101-LeftArm-Multitask
git apply X-VLA-SO101-LeftArm-Multitask/so101/xvla_so101_upstream.patch
cp X-VLA-SO101-LeftArm-Multitask/so101/so101.py datasets/domain_handler/
cp X-VLA-SO101-LeftArm-Multitask/so101/eval_so101_open_loop.py .
python eval_so101_open_loop.py --model X-VLA-SO101-LeftArm-Multitask --data-root <so101-left-arm-multitask v2.1 root>

from models.modeling_xvla import XVLA
from models.processing_xvla import XVLAProcessor
model = XVLA.from_pretrained("X-VLA-SO101-LeftArm-Multitask").cuda().eval()
processor = XVLAProcessor.from_pretrained("X-VLA-SO101-LeftArm-Multitask")
# images: [top, wrist, zeros] resized to 224x224 bicubic, ImageNet-normalized, image_mask all True;
# proprio: previous action min-max normalized to [-1, 1], zero-padded to 20; domain_id 19.
chunk = model.generate_actions(input_ids, image_input, image_mask, domain_id, proprio, steps=10)  # [1, 30, 6]
# joints 0-4: add the normalized proprio back; then (x + 1) / 2 * (max - min) + min with meta/stats.json.

Files

model.safetensors (3.5 GB, fp32), config.json (action_mode auto, real_action_dim 6, DaViT checkpointing on), tokenizer and preprocessor files, so101/ (handler, upstream patch, config copy, launcher, evaluation script).

License

Apache-2.0, as the base model 2toINF/X-VLA-Pt and the X-VLA repository.