| --- |
| library_name: pytorch |
| tags: |
| - arxiv:2608.15984 |
| - motion-captioning |
| - motion-language |
| - human-motion |
| - 2d-pose |
| pipeline_tag: other |
| license: mit |
| --- |
| |
| # 2D Motion Interface — inference bundle (MotionGPT) |
|
|
| [Paper (arXiv:2608.15984)](https://arxiv.org/abs/2608.15984) · [Code](https://github.com/irajisamurai/2D-Motion-Interface) · [HCMIW @ ECCV 2026](https://hcmiw.github.io/hcmiw-eccv2026/) (Oral Presentation) |
|
|
| Inference-only weights for **A Plug-and-Play 2D Motion Interface for Real-World Motion |
| Language Models**, in the MotionGPT configuration. This repo backs the demo Space; it is |
| not a general-purpose checkpoint. |
|
|
| The interface captions human motion from **2D keypoints only** — no 3D pose estimation |
| anywhere in the pipeline. The pretrained motion-language model is frozen; only the 2D |
| encoder (9.6M params) and the real-video adapter `A_real` (0.3M params) are trained. |
|
|
| ## Contents |
|
|
| | File | Size | | |
| |---|---|---| |
| | `lm.safetensors` | 993.3 MB | flan-t5-base + 515 motion tokens. The three tied embedding copies present in the training checkpoint are dropped; `lm_head` is untied and kept. | |
| | `vqvae.safetensors` | 39.4 MB | 2D encoder + quantizer. The decoder is dropped — only `encode` is ever called. | |
| | `adapter.safetensors` | 0.3 MB | `A_real`, the real-video adapter (residual, hidden 512, 81-dim). | |
| | `flan-t5-base/` | 2.4 MB | Config + tokenizer only. No pretrained weights: `lm.safetensors` supplies all of them. | |
| | `stats.npz` | 3 KB | Feature mean/std for the 68-dim and 81-dim layouts. | |
| | `model_config.yaml` | 1 KB | VQ-VAE hyperparameters, adapter type, `unit_length`, `max_motion_length`. | |
|
|
| Everything is **fp32**. Dropping the evaluator weights, the duplicated embeddings and the |
| VQ-VAE decoder takes the 1.53 GB training checkpoint down to 1.03 GB with no numerical |
| change, so fp16 is unnecessary — and flan-T5 is known to be unstable in fp16 anyway. |
|
|
| ## Pipeline |
|
|
| ``` |
| COCO-17 keypoints + confidence |
| -> COCO-13 (drop eyes/ears) |
| -> 81-dim features (with confidence) --A_real--> zero-pad to 263 |
| -> VQ-VAE.encode -> MotionGPT (m2t) -> caption |
| ``` |
|
|
| Features are mid-hip centred and scale-normalised, so raw pixel coordinates work at any |
| resolution. Input is expected at 20 fps. |
|
|
| ## Usage |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from captioner import MotionCaptioner # from the demo Space |
| |
| bundle = snapshot_download("KanameYOkoYAMA/2d-motion-interface") |
| cap = MotionCaptioner(bundle, device="cpu") |
| print(cap.caption_json("clip.json")["caption"]) |
| ``` |
|
|
| Decoding is greedy and the full clip is used from frame 0, so captions are reproducible. |
| A caption takes ~1 s on CPU. |
|
|
| ## Reproducibility |
|
|
| Captions were verified byte-for-byte against the evaluation script on the four demo clips |
| (8/8 exact, adapter and adapter-less paths). Across all 132 real-world clips the match rate |
| is 256/264; the 8 differences are GPU-vs-CPU floating-point noise flipping a greedy argmax, |
| not a difference in weights. |
|
|
| Verified with `torch==2.9.0`, `transformers==4.57.1`. Bumping transformers can change greedy |
| decoding — re-run the parity check after any upgrade. |
|
|
| ## Licence and provenance |
|
|
| MIT, following the licences of the upstream work this builds on: |
|
|
| - [**MotionGPT**](https://github.com/OpenMotionLab/MotionGPT) (MIT) — `lm.safetensors` and the |
| VQ-VAE quantizer derive from its pretrained checkpoint. |
| - [**HumanML3D**](https://github.com/EricGuo5513/HumanML3D) (MIT) — `stats.npz` holds mean/std |
| vectors computed over its features. |
| - [**flan-t5-base**](https://huggingface.co/google/flan-t5-base) (Apache-2.0) — the language |
| model architecture and tokenizer. |
|
|
| HumanML3D is itself derived from **AMASS**, whose distribution policy does not permit |
| redistributing the motion data. **No AMASS or HumanML3D motion data is included here** — this |
| repo contains trained weights and a few hundred aggregate mean/std floats, nothing from which |
| motion sequences could be recovered. |
|
|
| MotionGPT's README notes that its dependencies (SMPL, SMPL-X, PyTorch3D) and the datasets it |
| uses each carry their own licences, which apply to downstream use of these weights as well. |
|
|
