twanghcmut/OpenVLA-7B-SO101-LeftArm-Multitask

OpenVLA-7B fine-tuned on SO-101 left arm, 11 tasks (LoRA, merged)

openvla/openvla-7b LoRA-fine-tuned with the official OpenVLA repository (commit c8f03f4, vla-scripts/finetune.py with only its dataset block replaced, as the README's option (b) allows) on hungho77/so101-left-arm-multitask: 551 episodes, 253,745 frames at 30 fps, 11 tabletop tasks, one SO-101 left arm. The adapter is merged into the weights.

What this model is not: OpenVLA predicts one action per call, from one image. This checkpoint sees only the overhead top camera (the wrist camera is not used) and outputs a single 6-dim absolute joint target (5 joints in degrees + the continuous gripper aperture) as 6 tokens. There is no action chunk, so it cannot be compared directly with the 16-step chunk numbers of the other models in this series; see the evaluation section.

Tasks

Pass the task string verbatim; it is lower-cased in the prompt, as in OpenVLA's pipeline.

id	task
0	Pick up the red cube and place it into the steel pot.
1	Pick up the blue cube and place it into the steel pot
2	Take the banana and place it into the steel pot then close the lid.
3	Pick up the blue cube and place it next to the blue cube.
4	Take the blue cube out of the steel pot and place it on the blue cube.
5	Take the red cube out of the steel pot and place it on the red cube
6	Stack the yellow cube on top of the orange cube.
7	Pick up one orange block and stack it neatly on top of the other orange block
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board
10	Take out the blue cubes from shortest to tallest and place it on the table

How the SO-101 data enters OpenVLA

Files in so101/: so101.py (the dataset class), finetune_so101.py, run_so101_leftarm.sh, eval_so101_open_loop.py.

• One sample is one frame: the top image (224x224 through OpenVLA's processor), the lower-cased task, and the action at that frame, absolute joint targets. No state or proprio input (OpenVLA has none).
• Each action dimension is mapped to [-1, 1] with the dataset's 1st/99th percentile and clipped (OpenVLA's BOUNDS_Q99 normalization), with the mask set on all 6 dims: the OXE pipeline leaves a binary gripper raw, but this gripper is a continuous aperture. 256 bins per dimension, so 6 action tokens.
• Prompt: In: What action should the robot take to {task}?\nOut:.
• config.json still carries the Open X-Embodiment norm_stats inherited from the base model, without a key for this dataset. Load this run's dataset_statistics.json and set vla.norm_stats to it before calling predict_action(..., unnorm_key="so101_left_arm_multitask"), as OpenVLA's README describes for fine-tuned models.
• Sampling: an infinite stream of uniformly random frames (episodes weighted by length), color-jitter augmentation.

Training

Method	LoRA rank 32 on all linear layers (OpenVLA's default), merged afterwards; lr 5e-4, AdamW
Batch	4 per GPU x 4 gradient accumulation x 2 GPUs = 32 per optimizer step
Steps	24,000 = 768k samples, the same sample budget as the pi0.5, GR00T N1.7, MolmoAct2, RLDX-1, X-VLA and Evo-1 fine-tunes in this series
Hardware	2 x H100 MIG 3g.40gb slices, DDP, bf16; 25.7 GiB peak per slice (batch 8 per GPU does not fit 40 GB: the bf16 7B weights alone are 15 GB); 9 h 38 min, 1.45 s/step
Training metrics	loss 4.2 at step 10, 1.15 at step 6,000, about 0.8 at the end; token accuracy 6% at the start, 62-65% at the end; normalized L1 0.011

Evaluation (open loop, in-sample)

eval_so101_open_loop.py on the same 11 training trajectories as the rest of the series (episodes 0, 50, 100, 150, 200, 251, 301, 351, 401, 451, 501, first 400 frames), greedy decoding, de-normalized with this run's statistics. Because the model emits one action per call, two numbers are reported (the "Normalized" figures divide by the absolute action std 28.67):

metric	MAE	normalized
1-step: called every 4th frame, compared with the ground-truth action at that frame	0.817	2.85%
baseline: the previous ground-truth action	0.446	1.55%
baseline: copy the current state	2.451	8.55%
hold-16: predicted action repeated for 16 steps, compared with the next 16 ground-truth actions (260 anchors)	3.537	12.34%
baseline for hold-16: the ground-truth action at the anchor, repeated 16 steps	3.234	11.28%

Reading it honestly: one step ahead the model is much better than copying the state but worse than just repeating the previous action; as a zero-order-hold chunk it is slightly worse than the same hold fed with the label. It is a fit check, not a success rate. Per-trajectory 1-step MAE ranges 0.57-1.04 except trajectory 351 (task 7, stacking orange blocks), which is 1.87. The 1-step errors per joint are 0.43, 1.01, 1.19, 0.68, 0.56 and 1.03 (the gripper).

Same dataset and sample budget, the 16-step chunk protocol, other models (in-sample): MolmoAct2 4.37%, TurboVLA 4.62%, SmolVLA 6.72%, X-VLA 8.18%, pi0.5 9.82%, GR00T N1.7 10.27%, Evo-1 11.46%, RLDX-1 15.85%. The OpenVLA numbers above are not on that scale (different protocol, one step, one camera) and the 12.34% hold-16 figure should be read only against its own 11.28% baseline. In-sample numbers overstate real performance (held-out evaluation of SmolVLA and TurboVLA on this dataset showed errors about 3.5x their in-sample values). No real-robot result exists for this model.

Usage

import json, torch
from PIL import Image
from transformers import AutoModelForVision2Seq, AutoProcessor

repo = "twanghcmut/OpenVLA-7B-SO101-LeftArm-Multitask"
proc = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
vla = AutoModelForVision2Seq.from_pretrained(repo, torch_dtype=torch.bfloat16, trust_remote_code=True).to("cuda").eval()
from huggingface_hub import hf_hub_download
vla.norm_stats = json.load(open(hf_hub_download(repo, "dataset_statistics.json")))

task = "Stack the yellow cube on top of the orange cube.".lower()
prompt = f"In: What action should the robot take to {task}?\nOut:"
inputs = proc(prompt, Image.open("top_frame.png")).to("cuda", dtype=torch.bfloat16)
action = vla.predict_action(**inputs, unnorm_key="so101_left_arm_multitask", do_sample=False)  # (6,) joint targets

Environment notes

The official requirements pin transformers==4.40.1, timm==0.9.10, torch>=2.2. With the TensorFlow stack the repo installs, tensorflow-metadata must be 1.14.0 and wandb 0.16.6 to coexist with protobuf 4.

License

The OpenVLA code and openvla-7b weights are MIT; the Llama-2 backbone inside OpenVLA is additionally covered by the Llama 2 Community License, as the OpenVLA README states.