PoP on the SO-101: pretraining and fine-tuning checkpoints

PoP (Pointer Policy) is told what to act on through referents: a 2D point on the object, a 2D point on the destination and a small skill code, never a sentence. This repository holds its pretraining checkpoint (SO-100 community data, no conditioning) and the checkpoints of a real-robot comparison on four SO-101 tasks: PoP against a sentence-conditioned policy with the same network, initialization, data and steps.

Fine-tuning checkpoints and validation loss

L1 error of the EMA weights on normalized 16-step action chunks of 20 held-out demonstrations (5 per task). This is an imitation error, not a task success rate: no checkpoint has been run on the robot yet.

Run Model Initialization Frames Seed Val L1, step 40k drawer towel push cups Lowest val L1 (step)
261121-so101-pop-s0 PoP pretraining labelled 0 0.298 0.354 0.281 0.414 0.205 0.278 (4k)
261121-so101-pop-s1 PoP pretraining labelled 1 0.292 0.349 0.272 0.390 0.206 0.276 (6k)
261121-so101-pop-scratch-s0 PoP, no pretraining DINOv2 trunk only labelled 0 0.284 0.310 0.289 0.423 0.198 0.271 (4k)
261121-so101-sent-lab-s0 Sentence baseline pretraining labelled 0 0.292 0.335 0.288 0.406 0.205 0.277 (4k)
261121-so101-sent-lab-s1 Sentence baseline pretraining labelled 1 0.294 0.328 0.301 0.414 0.208 0.277 (6k)
261121-so101-sent-s0 Sentence baseline pretraining all 0 0.294 0.341 0.286 0.409 0.205 0.276 (4k)
261121-so101-sent-s1 Sentence baseline pretraining all 1 0.298 0.337 0.283 0.419 0.217 0.277 (4k)
  • "labelled": the frames with referent labels (287 of 293 training episodes); every model in the comparison uses them. "all": all 293 training episodes, an earlier variant of the sentence baseline.
  • Each folder finetune/<run>/ has step_020000.pt and step_040000.pt (the final checkpoint, the one to use), config.json and train.log. No checkpoint was selected on validation loss.

Pretraining checkpoint

pretrain/261120-pretrain-community-s0/step_200000.pt: vision trunk and control head (27.74M parameters) trained by behaviour cloning on HuggingFaceVLA/community_dataset_v2 at revision 815dd11: 289 single-arm 6-joint datasets, 10,639 episodes, 1,774,216 frames at 10 Hz (49.28 h). 200k steps at batch 256, final training L1 0.125. Trained without language and without referents: the referent slots hold learned null vectors and the skill code is a reserved "unspecified" entry. data/report.json lists what was kept and dropped and why.

Fine-tuning data

Dataset Revision Instruction Episodes Train episodes / frames Val episodes / frames
tmhoanggg/so101-fold-towel cb8e3b4 fold the towel 94 89 / 10,226 5 / 647
tmhoanggg/so101-drawer-v3 537b4bf open the top drawer, put the red block inside, then close the drawer 50 45 / 11,821 5 / 1,315
tmhoanggg/so101-stack-cup-v4 f57265d stack the pink cup on the beige cup, then stack them on the white cup 120 114 / 27,309 5 / 1,396
tmhoanggg/so101-push bd41765 push the mug to the goal 50 45 / 5,352 5 / 606
  • Converted to 10 Hz and 256 Γ— 256 (aspect ratio not kept); cameras front (fixed, behind the robot base) and wrist. One normalization pooled over all four datasets (state, and actions relative to the current joints), so the normalization does not tell the policy which task it is in.
  • One joint policy over the four tasks per run.

Referent labels, made from the demonstrations without a detector or hand marks (finetune/data/):

  • Camera: one fit per dataset from its own frames: the projected surface of the SO-101 URDF (TheRobotStudio/SO-ARM100, so101_new_calib.urdf) driven by the recorded joints is matched to the orange pixels of the arm (camera pose, focal length, radial distortion, a constant offset per joint). Residual about 3 px at 256 px. cameras/*.json.
  • Points: the gripper's 3D tip from forward kinematics. Object = where the gripper closes, destination = where it opens; drawer handle = the reach before the pull; push: the mug found by colour on the table plane, destination = where it ends. Labelled training episodes: drawer-v3 39/45, fold-towel 89/89, push 45/45, stack-cup-v4 114/114.
  • Steps and skill codes: towel fold; cups pick-and-place twice; drawer open, pick-and-place, close; push.

Model

Part
Trunk DINOv2 ViT-S/14, fine-tuned, fixed and wrist view at 224 px (22.06M)
Head adaLN-zero transformer decoder, width 224, 4 layers, 7 heads (5.68M); 16 joint targets relative to the current joints, at 10 Hz
PoP conditioning Per role (object, destination): Fourier features of the point's position in the fixed view and of its offset from the projected gripper tip, an MLP and a role vector; missing destination = null vector. Skill code: 5 rows of the skill embedding. About 28M parameters in all
Sentence conditioning T5-Efficient-Tiny (11.4M, fine-tuned), mean-pooled and projected into the skill slot; referent slots at their null vectors. 39.2M parameters in all

Recipe for every fine-tuning run: 40k steps at batch 64, AdamW (0.9, 0.95), weight decay 0.05, learning rate 3e-4 (3e-5 for the trunk and the text encoder), 2k warmup then cosine to 10%, EMA 0.9995, bf16, random crop (scale 0.8–1) on the fixed view with the points moved accordingly, 1 cm of 3D noise on the referent points, L1 loss.

Loading

import torch
ckpt = torch.load("finetune/261121-so101-pop-s0/step_040000.pt", map_location="cpu", weights_only=False)  # pickle
ckpt["ema"]           # weights (state dict)
ckpt["norm"]          # state / action normalization (pooled)
ckpt["instructions"]  # tokenized instructions (sentence runs)
ckpt["config"]        # training arguments

The network definition (popt/model.py, PopFinetune; pretraining: PopPretrain) is not part of this repository.

Limitations

  • Validation loss only. Robot roll-outs (positions outside the training range, swapped objects) have not been run.
  • The validation loss is lowest at 4k–6k steps and 0.013–0.021 higher at 40k: with about 290 episodes the 40k-step budget overfits the held-out demonstrations. All runs use the same budget.
  • Labels are heuristics on the gripper signal: towel grasp points sit 1–2 cm outside the cloth (fingertip midpoint); the drawer's "close" point is where the drawer stood open, not the moving drawer front.
  • The skill codes fold, push, open and close each belong to one task, so on those tasks PoP's skill code identifies the task as the instruction does; pick-and-place is shared between the drawer and the cups.

License

Apache-2.0, as DINOv2, T5-Efficient-Tiny, HuggingFaceVLA/community_dataset_v2 and the four tmhoanggg datasets.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Model tree for khanhnd61/pop-so101

Finetuned
(43)
this model

Datasets used to train khanhnd61/pop-so101