twanghcmut/RLDX-1-SO101-LeftArm-Multitask

RLDX-1 fine-tuned on SO-101 left arm, 11 tasks

RLWRLD/RLDX-1-PT fine-tuned with the official RLDX-1 repository on hungho77/so101-left-arm-multitask: 551 episodes, 253,745 frames at 30 fps, 11 tabletop tasks, one SO-101 left arm, cameras top (360x640) and wrist (480x640), 6-dim state and action (5 joints in degrees + gripper aperture, continuous 0.7-52.7).

Read the evaluation section before using it: on the open-loop protocol used for every model in this series it is less accurate than holding the current action for 16 steps, and it has not been run on the real robot.

Tasks

Pass the task string verbatim, trailing period included where present.

id	task
0	Pick up the red cube and place it into the steel pot.
1	Pick up the blue cube and place it into the steel pot
2	Take the banana and place it into the steel pot then close the lid.
3	Pick up the blue cube and place it next to the blue cube.
4	Take the blue cube out of the steel pot and place it on the blue cube.
5	Take the red cube out of the steel pot and place it on the red cube
6	Stack the yellow cube on top of the orange cube.
7	Pick up one orange block and stack it neatly on top of the other orange block
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board
10	Take out the blue cubes from shortest to tallest and place it on the table

Training

Same sample budget as the pi0.5, GR00T N1.7 and MolmoAct2 fine-tunes in this series: 6,000 optimizer steps x 128 samples = 768k samples, about 3 epochs (the smaller SmolVLA and TurboVLA runs used 20,000 x 128).

Base model	RLWRLD/RLDX-1-PT (Qwen3-VL-8B backbone, truncated; flow-matching action expert)
Trainable	top 4 LLM layers, projector, action (diffusion) model, cog tokens; vision tower and the rest of the LLM frozen (tune_llm=False, tune_visual=False, tune_top_llm_layers=4)
Modality config	so101_config.py (included): video top, wrist; state/action single_arm (0:5) + gripper (5:6); single_arm action RELATIVE to the current state, gripper ABSOLUTE; language from task_index
Frame history	video_length 4, video_stride 2: frames t-6, t-4, t-2, t per camera
Action horizon	16 steps (0.53 s at 30 fps)
Batch	4 per GPU x 2 GPUs x 16 gradient accumulation = 128 effective
Steps / LR	6,000; AdamW (fused) 1e-4, cosine, 5% warmup, weight decay 1e-5, grad clip 1.0, bf16
Other	n_cog_tokens 64, physics_loss_weight 0.1, episode_sampling_rate 0.1, shard_size 256
Hardware	2 x H100 MIG 3g.40gb slices (40 GB each), DeepSpeed ZeRO-2, 20.4 h (12 s/step)
Loss	0.19 at step 1, 0.03 at step 6,000 (mean over the run 0.051)

Data: a LeRobot v2.1 view of the dataset with meta/relative_stats.json generated by rldx/data/stats.py (required by the RELATIVE action representation). Launcher: finetune_so101_leftarm.sh and supervise_leftarm_2mig.sh (included). Two things that cost a day and are worth knowing: a 40 GB slice fits at most 4 samples per step (8 fails at the first optimizer step; gradient checkpointing does not help), and the dataloader keeps a processed shard plus the next prefetched one per worker, so with 4 workers and shard 1024 the host RAM use exceeded a 64 GB job limit and the run was OOM-killed at the first checkpoint save. 2 workers and shard 256 run at the same speed within 64 GB.

Evaluation (open loop, in-sample)

Protocol shared by every model in this series: rldx/eval/open_loop_eval.py on 11 training trajectories (one per task: episodes 0, 50, 100, 150, 200, 251, 301, 351, 401, 451, 501), the policy called every 16 steps from the dataset's own observations, the predicted 16-step chunk compared with the ground-truth actions, up to 400 steps per trajectory, MAE in the dataset's units (degrees; gripper aperture). "Normalized" divides by the absolute action std of the dataset, 28.67. These trajectories were in the training set, so this measures fit, not generalization.

trajectory	task	MAE	MSE
0	0	3.859	35.47
50	1	3.749	32.32
100	2	5.506	80.76
150	3	5.191	72.58
200	4	3.958	48.46
251	5	4.183	48.97
301	6	4.061	43.55
351	7	5.683	83.93
401	8	4.658	50.74
451	9	3.885	36.31
501	10	5.271	75.08
average		4.546	55.29

Normalized MAE: 15.85% of the action scale. Two trivial baselines on the same trajectories and protocol:

baseline	MAE	normalized
hold the action at the chunk start for 16 steps	3.282	11.45%
copy the current state for 16 steps	5.513	19.23%

So this checkpoint is worse than holding the current action. The plots (open_loop_eval/traj_*.jpeg) show why: the predictions follow the ground truth in every joint with no offset, but each 16-step chunk carries high-frequency jitter of a few units and lags at transitions; the ground truth is nearly flat most of the time, so the jitter alone costs more than the hold baseline. It is not the sampler: the checkpoint default is 4 Euler steps, and re-running the same evaluation with 8 and 16 uniform steps gave 16.67% and 17.75% (slightly worse). Whether it comes from the RELATIVE representation or the frame-history padding at chunk starts was not isolated.

Same protocol, same dataset, same budget, other models in this series (all in-sample):

model	normalized MAE	checkpoint
MolmoAct2	4.37%	twanghcmut/molmoact2-so101-left-arm-multitask
TurboVLA	4.62%	twanghcmut/TurboVLA-SO101-LeftArm-Multitask
SmolVLA	6.72%	twanghcmut/SmolVLA-SO101-LeftArm-Multitask
pi0.5	9.82%	twanghcmut/pi05-so101-left-arm-multitask
GR00T N1.7	10.27%	twanghcmut/gr00t-n1d7-so101-left-arm-multitask
RLDX-1 (this)	15.85%

Caveats that apply to the whole series: in-sample numbers overstate real performance (held-out evaluation of SmolVLA and TurboVLA on this dataset showed errors about 3.5x their in-sample values), and none of these numbers is a real-robot success rate.

Usage

git clone https://github.com/RLWRLD/RLDX-1 && cd RLDX-1   # follow its install instructions
hf download twanghcmut/RLDX-1-SO101-LeftArm-Multitask --local-dir RLDX-1-SO101-LeftArm-Multitask

# open-loop check against the dataset (LeRobot v2.1 layout with meta/modality.json)
python rldx/eval/open_loop_eval.py \
  --dataset-path <so101-left-arm-multitask> --embodiment-tag GENERAL_EMBODIMENT \
  --model-path RLDX-1-SO101-LeftArm-Multitask \
  --traj-ids 0 50 100 150 200 251 301 351 401 451 501 --action-horizon 16 --steps 400

from rldx.policy.rldx_policy import RLDXPolicy
from rldx.data.embodiment_tags import EmbodimentTag

policy = RLDXPolicy(embodiment_tag=EmbodimentTag.GENERAL_EMBODIMENT,
                    model_path="RLDX-1-SO101-LeftArm-Multitask", device="cuda")
# observation: video.top / video.wrist as uint8 frames for t-6, t-4, t-2, t; state.single_arm (5,), state.gripper (1,)
# in float32; annotation.human.task_description = one of the task strings above. get_action returns a 16-step chunk.

The modality configuration is read from processor/ and experiment_cfg/ inside this repo; so101_config.py is the same configuration as a file, for re-training.

Files

model-*.safetensors + index (bf16, 13.8 GB), config.json, processor/, experiment_cfg/, trainer_state.json, so101_config.py, finetune_so101_leftarm.sh, supervise_leftarm_2mig.sh, open_loop_eval/ (the 11 evaluation plots), LICENSE.md.

License

Licensed under the RLWRLD Model License v1.0 (see LICENSE.md, inherited from RLWRLD/RLDX-1-PT): this is a Derivative Model of RLDX-1, distributed under the same license, which permits non-commercial use only (academic research, educational, personal or evaluation purposes) and requires share-alike for further derivatives. Changes made: fine-tuned on the dataset above as described in Training.