act-tshirt-folding-lerobot

ACT (Action Chunking Transformer) policy for bimanual t-shirt folding on the Trossen AI stationary (WidowX AI) platform. Trained with lerobot. This Run Record documents the step-100000 (final) checkpoint.

Demo — on-robot eval

Prompt: Fold the tshirt

Offline on-robot evaluation of this checkpoint (step-100000) on the Trossen AI stationary bimanual platform — t-shirt folding.

TL;DR

• Policy: ACT (Action Chunking Transformer) — ResNet18 vision backbone + transformer encoder/decoder with CVAE.
• Framework: lerobot-latest (--policy.type=act).
• Method: Full fine-tune (no LoRA/adapters) from scratch on the target dataset.
• Robot: Trossen AI stationary bimanual, 14-DoF, 3 cameras, 30 fps.
• Checkpoint: step 100000.

Setup / rig configuration

Recorded on the Trossen AI Mobile kit with the mobile base disabled — arms mounted in a fixed, stationary configuration (functionally equivalent to the stationary kit for this task).

Dimension	Value
Arm base height off ground	39.5 in
Arm base-to-base spacing	95 in
Arm horizontal offset from table edge	8.25 in

Model configuration (config.json)

Field	Value
type	act
input_features	observation.state (14); images cam_high, cam_left_wrist, cam_right_wrist (3×480×640)
output_features	action (14)
chunk_size / n_action_steps	100 / 100
n_obs_steps	1
normalization	STATE=MEAN_STD, ACTION=MEAN_STD, VISUAL=MEAN_STD
vision_backbone	resnet18 (ResNet18_Weights.IMAGENET1K_V1)
dim_model	512
n_heads	8
dim_feedforward	3200
n_encoder_layers / n_decoder_layers	4 / 1
use_vae	true
latent_dim	32
n_vae_encoder_layers	4
dropout	0.1
kl_weight	10.0

Training hyperparameters

Field	Value
optimizer	AdamW
learning rate	1e-05
lr_backbone	1e-05
betas	(0.9, 0.999)
eps	1e-08
weight_decay	0.0001
grad_clip_norm	10.0
batch_size	8
total steps	100,000
save_freq	5,000

Inputs (reproducibility manifest)

• Dataset: TrossenRoboticsCommunity/mobile-baseoff-tshirt-folding-lerobot-v1.
• Framework: lerobot-latest (--policy.type=act).
• Batch size: 8.
• Total steps: 100,000 (this is the final checkpoint).
• Training env / hardware: local RTX 5090 (TOTL workstation, Trossen cloud pipeline).

Outputs

• Weights: model.safetensors (full fine-tune, no separate adapter).
• Config: config.json, train_config.json.
• Processors: policy_preprocessor.json (+ normalizer state), policy_postprocessor.json (+ unnormalizer state).

Evaluation (offline policy eval)

Verified to load and run directly via lerobot-record with --policy.path (no async policy server required for ACT). Run from the lerobot_trossen workspace:

uv run lerobot-record \
    --robot.type=bi_widowxai_follower_robot \
    --robot.left_arm_ip_address=192.168.1.4 \
    --robot.right_arm_ip_address=192.168.1.5 \
    --robot.id=bimanual_follower \
    --robot.cameras='{
        cam_high: {type: intelrealsense, serial_number_or_name: "230322276010", width: 640, height: 480, fps: 30},
        cam_left_wrist: {type: intelrealsense, serial_number_or_name: "230322271348", width: 640, height: 480, fps: 30},
        cam_right_wrist: {type: intelrealsense, serial_number_or_name: "128422270115", width: 640, height: 480, fps: 30}
        }' \
    --display_data=false \
    --dataset.repo_id=hamza-trossen/eval_act_tshirt_folding \
    --dataset.push_to_hub=false \
    --dataset.num_episodes=20 \
    --dataset.episode_time_s=105 \
    --dataset.reset_time_s=25 \
    --dataset.single_task="Fold the tshirt" \
    --policy.path=TrossenRoboticsCommunity/act-tshirt-folding-lerobot \
    --robot.min_time_to_move_multiplier=8.0

The task prompt used during recording and eval was "Fold the tshirt".

Eval Results

Eval Task	Reps	Success	Notes
tshirt-folding	10	6/10 (60%)	Policy confirmed working on hardware; min_time_to_move_multiplier=8.0 config performed best for ACT on this task

Links

• Dataset: https://huggingface.co/datasets/TrossenRoboticsCommunity/mobile-baseoff-tshirt-folding-lerobot-v1