KanameYOkoYAMA/2d-motion-interface

2D Motion Interface — inference bundle (MotionGPT)

Paper (arXiv:2608.15984) · Code · HCMIW @ ECCV 2026 (Oral Presentation)

Inference-only weights for A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models, in the MotionGPT configuration. This repo backs the demo Space; it is not a general-purpose checkpoint.

The interface captions human motion from 2D keypoints only — no 3D pose estimation anywhere in the pipeline. The pretrained motion-language model is frozen; only the 2D encoder (9.6M params) and the real-video adapter A_real (0.3M params) are trained.

Contents

File	Size
lm.safetensors	993.3 MB	flan-t5-base + 515 motion tokens. The three tied embedding copies present in the training checkpoint are dropped; lm_head is untied and kept.
vqvae.safetensors	39.4 MB	2D encoder + quantizer. The decoder is dropped — only encode is ever called.
adapter.safetensors	0.3 MB	A_real, the real-video adapter (residual, hidden 512, 81-dim).
flan-t5-base/	2.4 MB	Config + tokenizer only. No pretrained weights: lm.safetensors supplies all of them.
stats.npz	3 KB	Feature mean/std for the 68-dim and 81-dim layouts.
model_config.yaml	1 KB	VQ-VAE hyperparameters, adapter type, unit_length, max_motion_length.

Everything is fp32. Dropping the evaluator weights, the duplicated embeddings and the VQ-VAE decoder takes the 1.53 GB training checkpoint down to 1.03 GB with no numerical change, so fp16 is unnecessary — and flan-T5 is known to be unstable in fp16 anyway.

Pipeline

COCO-17 keypoints + confidence
  -> COCO-13 (drop eyes/ears)
  -> 81-dim features (with confidence)  --A_real-->  zero-pad to 263
  -> VQ-VAE.encode  ->  MotionGPT (m2t)  ->  caption

Features are mid-hip centred and scale-normalised, so raw pixel coordinates work at any resolution. Input is expected at 20 fps.

Usage

from huggingface_hub import snapshot_download
from captioner import MotionCaptioner          # from the demo Space

bundle = snapshot_download("KanameYOkoYAMA/2d-motion-interface")
cap = MotionCaptioner(bundle, device="cpu")
print(cap.caption_json("clip.json")["caption"])

Decoding is greedy and the full clip is used from frame 0, so captions are reproducible. A caption takes ~1 s on CPU.

Reproducibility

Captions were verified byte-for-byte against the evaluation script on the four demo clips (8/8 exact, adapter and adapter-less paths). Across all 132 real-world clips the match rate is 256/264; the 8 differences are GPU-vs-CPU floating-point noise flipping a greedy argmax, not a difference in weights.

Verified with torch==2.9.0, transformers==4.57.1. Bumping transformers can change greedy decoding — re-run the parity check after any upgrade.

Licence and provenance

MIT, following the licences of the upstream work this builds on:

• MotionGPT (MIT) — lm.safetensors and the VQ-VAE quantizer derive from its pretrained checkpoint.
• HumanML3D (MIT) — stats.npz holds mean/std vectors computed over its features.
• flan-t5-base (Apache-2.0) — the language model architecture and tokenizer.

HumanML3D is itself derived from AMASS, whose distribution policy does not permit redistributing the motion data. No AMASS or HumanML3D motion data is included here — this repo contains trained weights and a few hundred aggregate mean/std floats, nothing from which motion sequences could be recovered.

MotionGPT's README notes that its dependencies (SMPL, SMPL-X, PyTorch3D) and the datasets it uses each carry their own licences, which apply to downstream use of these weights as well.