File size: 4,149 Bytes
39f11de 9420741 39f11de 69c39a7 39f11de 9420741 39f11de 69c39a7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
library_name: pytorch
tags:
- arxiv:2608.15984
- motion-captioning
- motion-language
- human-motion
- 2d-pose
pipeline_tag: other
license: mit
---
# 2D Motion Interface — inference bundle (MotionGPT)
[Paper (arXiv:2608.15984)](https://arxiv.org/abs/2608.15984) · [Code](https://github.com/irajisamurai/2D-Motion-Interface) · [HCMIW @ ECCV 2026](https://hcmiw.github.io/hcmiw-eccv2026/) (Oral Presentation)
Inference-only weights for **A Plug-and-Play 2D Motion Interface for Real-World Motion
Language Models**, in the MotionGPT configuration. This repo backs the demo Space; it is
not a general-purpose checkpoint.
The interface captions human motion from **2D keypoints only** — no 3D pose estimation
anywhere in the pipeline. The pretrained motion-language model is frozen; only the 2D
encoder (9.6M params) and the real-video adapter `A_real` (0.3M params) are trained.
## Contents
| File | Size | |
|---|---|---|
| `lm.safetensors` | 993.3 MB | flan-t5-base + 515 motion tokens. The three tied embedding copies present in the training checkpoint are dropped; `lm_head` is untied and kept. |
| `vqvae.safetensors` | 39.4 MB | 2D encoder + quantizer. The decoder is dropped — only `encode` is ever called. |
| `adapter.safetensors` | 0.3 MB | `A_real`, the real-video adapter (residual, hidden 512, 81-dim). |
| `flan-t5-base/` | 2.4 MB | Config + tokenizer only. No pretrained weights: `lm.safetensors` supplies all of them. |
| `stats.npz` | 3 KB | Feature mean/std for the 68-dim and 81-dim layouts. |
| `model_config.yaml` | 1 KB | VQ-VAE hyperparameters, adapter type, `unit_length`, `max_motion_length`. |
Everything is **fp32**. Dropping the evaluator weights, the duplicated embeddings and the
VQ-VAE decoder takes the 1.53 GB training checkpoint down to 1.03 GB with no numerical
change, so fp16 is unnecessary — and flan-T5 is known to be unstable in fp16 anyway.
## Pipeline
```
COCO-17 keypoints + confidence
-> COCO-13 (drop eyes/ears)
-> 81-dim features (with confidence) --A_real--> zero-pad to 263
-> VQ-VAE.encode -> MotionGPT (m2t) -> caption
```
Features are mid-hip centred and scale-normalised, so raw pixel coordinates work at any
resolution. Input is expected at 20 fps.
## Usage
```python
from huggingface_hub import snapshot_download
from captioner import MotionCaptioner # from the demo Space
bundle = snapshot_download("KanameYOkoYAMA/2d-motion-interface")
cap = MotionCaptioner(bundle, device="cpu")
print(cap.caption_json("clip.json")["caption"])
```
Decoding is greedy and the full clip is used from frame 0, so captions are reproducible.
A caption takes ~1 s on CPU.
## Reproducibility
Captions were verified byte-for-byte against the evaluation script on the four demo clips
(8/8 exact, adapter and adapter-less paths). Across all 132 real-world clips the match rate
is 256/264; the 8 differences are GPU-vs-CPU floating-point noise flipping a greedy argmax,
not a difference in weights.
Verified with `torch==2.9.0`, `transformers==4.57.1`. Bumping transformers can change greedy
decoding — re-run the parity check after any upgrade.
## Licence and provenance
MIT, following the licences of the upstream work this builds on:
- [**MotionGPT**](https://github.com/OpenMotionLab/MotionGPT) (MIT) — `lm.safetensors` and the
VQ-VAE quantizer derive from its pretrained checkpoint.
- [**HumanML3D**](https://github.com/EricGuo5513/HumanML3D) (MIT) — `stats.npz` holds mean/std
vectors computed over its features.
- [**flan-t5-base**](https://huggingface.co/google/flan-t5-base) (Apache-2.0) — the language
model architecture and tokenizer.
HumanML3D is itself derived from **AMASS**, whose distribution policy does not permit
redistributing the motion data. **No AMASS or HumanML3D motion data is included here** — this
repo contains trained weights and a few hundred aggregate mean/std floats, nothing from which
motion sequences could be recovered.
MotionGPT's README notes that its dependencies (SMPL, SMPL-X, PyTorch3D) and the datasets it
uses each carry their own licences, which apply to downstream use of these weights as well.
|