Instructions to use khanhnd61/pop-so101 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use khanhnd61/pop-so101 with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
PoP on the SO-101: pretraining and fine-tuning checkpoints
PoP (Pointer Policy) is told what to act on through referents: a 2D point on the object, a 2D point on the destination and a small skill code, never a sentence. This repository holds its pretraining checkpoint (SO-100 community data, no conditioning) and the checkpoints of a real-robot comparison on four SO-101 tasks: PoP against a sentence-conditioned policy with the same network, initialization, data and steps.
Fine-tuning checkpoints and validation loss
L1 error of the EMA weights on normalized 16-step action chunks of 20 held-out demonstrations (5 per task). This is an imitation error, not a task success rate: no checkpoint has been run on the robot yet.
| Run | Model | Initialization | Frames | Seed | Val L1, step 40k | drawer | towel | push | cups | Lowest val L1 (step) |
|---|---|---|---|---|---|---|---|---|---|---|
261121-so101-pop-s0 |
PoP | pretraining | labelled | 0 | 0.298 | 0.354 | 0.281 | 0.414 | 0.205 | 0.278 (4k) |
261121-so101-pop-s1 |
PoP | pretraining | labelled | 1 | 0.292 | 0.349 | 0.272 | 0.390 | 0.206 | 0.276 (6k) |
261121-so101-pop-scratch-s0 |
PoP, no pretraining | DINOv2 trunk only | labelled | 0 | 0.284 | 0.310 | 0.289 | 0.423 | 0.198 | 0.271 (4k) |
261121-so101-sent-lab-s0 |
Sentence baseline | pretraining | labelled | 0 | 0.292 | 0.335 | 0.288 | 0.406 | 0.205 | 0.277 (4k) |
261121-so101-sent-lab-s1 |
Sentence baseline | pretraining | labelled | 1 | 0.294 | 0.328 | 0.301 | 0.414 | 0.208 | 0.277 (6k) |
261121-so101-sent-s0 |
Sentence baseline | pretraining | all | 0 | 0.294 | 0.341 | 0.286 | 0.409 | 0.205 | 0.276 (4k) |
261121-so101-sent-s1 |
Sentence baseline | pretraining | all | 1 | 0.298 | 0.337 | 0.283 | 0.419 | 0.217 | 0.277 (4k) |
- "labelled": the frames with referent labels (287 of 293 training episodes); every model in the comparison uses them. "all": all 293 training episodes, an earlier variant of the sentence baseline.
- Each folder
finetune/<run>/hasstep_020000.ptandstep_040000.pt(the final checkpoint, the one to use),config.jsonandtrain.log. No checkpoint was selected on validation loss.
Pretraining checkpoint
pretrain/261120-pretrain-community-s0/step_200000.pt: vision trunk and control head (27.74M parameters) trained by behaviour cloning
on HuggingFaceVLA/community_dataset_v2 at revision 815dd11: 289 single-arm 6-joint datasets,
10,639 episodes, 1,774,216 frames at 10 Hz (49.28 h). 200k steps at batch 256, final
training L1 0.125. Trained without language and without referents: the referent slots hold learned null vectors and the
skill code is a reserved "unspecified" entry. data/report.json lists what was kept and dropped and why.
Fine-tuning data
| Dataset | Revision | Instruction | Episodes | Train episodes / frames | Val episodes / frames |
|---|---|---|---|---|---|
tmhoanggg/so101-fold-towel |
cb8e3b4 |
fold the towel | 94 | 89 / 10,226 | 5 / 647 |
tmhoanggg/so101-drawer-v3 |
537b4bf |
open the top drawer, put the red block inside, then close the drawer | 50 | 45 / 11,821 | 5 / 1,315 |
tmhoanggg/so101-stack-cup-v4 |
f57265d |
stack the pink cup on the beige cup, then stack them on the white cup | 120 | 114 / 27,309 | 5 / 1,396 |
tmhoanggg/so101-push |
bd41765 |
push the mug to the goal | 50 | 45 / 5,352 | 5 / 606 |
- Converted to 10 Hz and 256 Γ 256 (aspect ratio not kept); cameras
front(fixed, behind the robot base) andwrist. One normalization pooled over all four datasets (state, and actions relative to the current joints), so the normalization does not tell the policy which task it is in. - One joint policy over the four tasks per run.
Referent labels, made from the demonstrations without a detector or hand marks (finetune/data/):
- Camera: one fit per dataset from its own frames: the projected surface of the SO-101 URDF (TheRobotStudio/SO-ARM100,
so101_new_calib.urdf) driven by the recorded joints is matched to the orange pixels of the arm (camera pose, focal length, radial distortion, a constant offset per joint). Residual about 3 px at 256 px.cameras/*.json. - Points: the gripper's 3D tip from forward kinematics. Object = where the gripper closes, destination = where it opens; drawer handle = the reach before the pull; push: the mug found by colour on the table plane, destination = where it ends. Labelled training episodes: drawer-v3 39/45, fold-towel 89/89, push 45/45, stack-cup-v4 114/114.
- Steps and skill codes: towel fold; cups pick-and-place twice; drawer open, pick-and-place, close; push.
Model
| Part | |
|---|---|
| Trunk | DINOv2 ViT-S/14, fine-tuned, fixed and wrist view at 224 px (22.06M) |
| Head | adaLN-zero transformer decoder, width 224, 4 layers, 7 heads (5.68M); 16 joint targets relative to the current joints, at 10 Hz |
| PoP conditioning | Per role (object, destination): Fourier features of the point's position in the fixed view and of its offset from the projected gripper tip, an MLP and a role vector; missing destination = null vector. Skill code: 5 rows of the skill embedding. About 28M parameters in all |
| Sentence conditioning | T5-Efficient-Tiny (11.4M, fine-tuned), mean-pooled and projected into the skill slot; referent slots at their null vectors. 39.2M parameters in all |
Recipe for every fine-tuning run: 40k steps at batch 64, AdamW (0.9, 0.95), weight decay 0.05, learning rate 3e-4 (3e-5 for the trunk and the text encoder), 2k warmup then cosine to 10%, EMA 0.9995, bf16, random crop (scale 0.8β1) on the fixed view with the points moved accordingly, 1 cm of 3D noise on the referent points, L1 loss.
Loading
import torch
ckpt = torch.load("finetune/261121-so101-pop-s0/step_040000.pt", map_location="cpu", weights_only=False) # pickle
ckpt["ema"] # weights (state dict)
ckpt["norm"] # state / action normalization (pooled)
ckpt["instructions"] # tokenized instructions (sentence runs)
ckpt["config"] # training arguments
The network definition (popt/model.py, PopFinetune; pretraining: PopPretrain) is not part of this repository.
Limitations
- Validation loss only. Robot roll-outs (positions outside the training range, swapped objects) have not been run.
- The validation loss is lowest at 4kβ6k steps and 0.013β0.021 higher at 40k: with about 290 episodes the 40k-step budget overfits the held-out demonstrations. All runs use the same budget.
- Labels are heuristics on the gripper signal: towel grasp points sit 1β2 cm outside the cloth (fingertip midpoint); the drawer's "close" point is where the drawer stood open, not the moving drawer front.
- The skill codes fold, push, open and close each belong to one task, so on those tasks PoP's skill code identifies the task as the instruction does; pick-and-place is shared between the drawer and the cups.
License
Apache-2.0, as DINOv2, T5-Efficient-Tiny, HuggingFaceVLA/community_dataset_v2 and the four tmhoanggg datasets.
Model tree for khanhnd61/pop-so101
Base model
facebook/dinov2-small