---
license: other
license_name: fair-noncommercial-research-license-v1
license_link: https://huggingface.co/Rice-RobotPI-Lab/robotok-public/blob/main/LICENSE-Action100M
library_name: pytorch
tags:
  - robotics
  - video-retrieval
  - hand-pose
  - trajectory
  - dynamic-time-warping
  - computer-vision
  - action100m
---

# RoboTok — An Internet-Scale Data Engine for Human Demonstration Video Retrieval and Dexterous Manipulation Learning

🌐 **Project website:** [rice-robotpi-lab.github.io/RoboTok](https://rice-robotpi-lab.github.io/RoboTok/)

![RoboTok hand-motion embedding space](hero.jpg)

Released checkpoints and evaluation keypoints for **RoboTok**, a model that
retrieves web video clips by 3D hand-motion similarity. Similarity is defined by
DTW over torso-relative 3D hand keypoints; the encoder is trained to reproduce
that DTW ranking in a fast embedding space.

Training and evaluation code is in the accompanying source release.

## Files

| File | Size | Description |
| --- | --- | --- |
| `models/best_abs_retrieval_model.pt` | 11 MB | Retrieval encoder. Cross-attention head over `[T, 126]` hand-trajectory features (21 joints x 3 coords x 2 hands, `T_max = 42`): 1 learned query token, 256-d input projection, 1 cross-attention layer (4 heads, sinusoidal PE), 2-layer MLP to a 256-d embedding. DTW design `abs_21j_coords`. |
| `models/best_abs_retrieval_model.yaml` | 2 KB | Minimal config to reload the encoder for inference. |
| `models/body_pose_est.pt` | 9.9 MB | Vector-neuron torso/body-frame estimator: 4-layer rotation-equivariant transformer mapping two-hand trajectories to a torso frame, with separate rotation and translation heads. |
| `eval_data/torso_relative_clip_keypoints.pt` | 6.5 GB | Torso-relative 3D hand keypoints per clip. Each entry has `video_number`, `node_number`, `node_uid`, `keypoints_per_frame` (`kpts_2d`, `kpts_3d`), and `infilled` / `depth_grounded` flags. |

## Loading

```python
import torch

ckpt = torch.load("models/best_abs_retrieval_model.pt", map_location="cpu", weights_only=True)
ckpt["head_state_dict"]   # encoder weights
ckpt["config"]            # full training configuration

vn = torch.load("models/body_pose_est.pt", map_location="cpu", weights_only=True)
vn["model"]               # torso estimator weights
```

## Citation

```bibtex
@article{qian2026robotok,
  title     = {RoboTok: An Internet-Scale Data Engine for Human
               Demonstration Video Retrieval and Dexterous Manipulation
               Learning},
  author    = {Qian, Howard and Chen, Yiting and Xie, Yunfei and
               Ren, Kejia and Chanrungmaneekul, Podshara and Wang, Gaotian and
               Wen, Bowen and Wei, Chen and Hang, Kaiyu},
  journal   = {arXiv preprint arXiv:2609.03199},
  year      = {2026}
}
```

## License

**FAIR Noncommercial Research License v1** (see
[LICENSE-Action100M](LICENSE-Action100M)). Noncommercial research only.

The released checkpoints (`models/*.pt`) and evaluation keypoints
(`eval_data/torso_relative_clip_keypoints.pt`) are derivative works of
Action100M (Meta FAIR) clips and are governed by that license. It covers
trained model weights as "Research Materials", and restricts both those
materials and any outputs or results obtained from them to noncommercial
research use. If you publish results obtained using these materials, the
license requires you to acknowledge that use.

MIT ([LICENSE](LICENSE)) covers only `models/best_abs_retrieval_model.yaml`
and the accompanying source release.

The MANO / SMPL-H body models required by parts of the pipeline are **not**
included and remain under their own MPI-IS license terms — register at
https://mano.is.tue.mpg.de to obtain them.