File size: 5,744 Bytes
7711b8b abf6b05 7711b8b d0e0fbc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | ---
library_name: motius
pipeline_tag: other
tags:
- motion-to-text
- motion-captioning
- humanml3d
- motiongpt
---
<h1 align="center">MotionGPT Model Card</h1>
<p align="center">
<strong>Motion-language generation with discrete motion tokens, packaged as a Motius pipeline.</strong>
</p>
<p align="center">
<a href="https://arxiv.org/abs/2306.14795">Paper</a> |
<a href="https://motion-gpt.github.io/">Project Page</a> |
<a href="https://github.com/OpenMotionLab/MotionGPT">Original GitHub</a> |
<a href="https://huggingface.co/ZeyuLing/Motius-MotionGPT-HumanML3D">Motius Checkpoint</a>
</p>
MotionGPT is the motion-language baseline from *MotionGPT: Human Motion as a
Foreign Language* (Jiang et al., NeurIPS 2023). This Motius release packages
the HumanML3D motion tokenizer, FLAN-T5-base-style language model with motion
tokens, HumanML3D statistics, and task-facing text-to-motion / motion-to-text
pipeline methods without requiring the original checkout.
## Preview
| HumanML3D Sample | Input Text | SMPL Preview |
| ---------------- | ---------- | ------------ |
| `001840` | someone executes a roundhouse kick with their left foot. |  |
| `004545` | a person jumping while raising both hands and moving apart legs. |  |
| `006944` | a person moves their right hand left, right, up, and down. |  |
512px / 30fps GIF previews rendered from released HumanML3D test outputs.
## Release Snapshot
| Item | Value |
| ---- | ----- |
| Method | MotionGPT, language modeling over text and motion tokens |
| Tasks | T2M, M2T |
| Venue | NeurIPS 2023 |
| Motion representation | HumanML3D-263, 20 fps |
| Language backbone | FLAN-T5-base-style encoder-decoder with motion tokens |
| Motion tokenizer | VQ-VAE, 512-code codebook |
| Checkpoint | [`ZeyuLing/Motius-MotionGPT-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-MotionGPT-HumanML3D) |
| Pipeline | `motius.pipelines.motiongpt.MotionGPTPipeline` |
The checkpoint artifact contains `motiongpt_s3_h3d.tar`,
`assets/meta/mean.npy`, `assets/meta/std.npy`, `deps/flan-t5-base/`, and
`model_index.json`.
## Usage
```python
from motius.pipelines.motiongpt import MotionGPTPipeline
pipe = MotionGPTPipeline.from_pretrained(
"ZeyuLing/Motius-MotionGPT-HumanML3D",
bundle_kwargs={"local_files_only": False},
device="cuda",
)
motions = pipe.infer_t2m(
["a person walks forward then sits down"],
[120],
)
```
`motions` is a list of NumPy arrays. Each array has shape `(T, 263)` and is
denormalized to HumanML3D physical scale. The same pipeline also exposes
`infer_m2t` for captioning denormalized HumanML3D-263 motions.
```python
caption = pipe.infer_m2t(
[motions[0]],
lengths=[len(motions[0])],
)[0]
```
## Evaluation Results
Protocol: HumanML3D Official uses the selected-caption HumanML3D test protocol. MotionStreamer Evaluator and Motius Joint-Position Evaluator are computed after converting outputs through the shared SMPL-22 evaluation bridge. For FID and MM-Dist, lower is better.
| Evaluator | Variant | Samples | R@1 | R@2 | R@3 | FID | MM-Dist | Diversity | Status |
| --------- | ------- | ------: | --: | --: | --: | --: | ------: | --------: | ------ |
| HumanML3D Official | Default | 3,962 | 0.434 | 0.600 | 0.686 | 0.156 | 3.920 | 9.747 | Measured |
| MotionStreamer Evaluator | Default | 4,042 | 0.494 | 0.635 | 0.694 | 23.681 | 19.678 | 25.541 | Measured |
| Motius Joint-Position Evaluator | Default | 4,034 | 0.432 | 0.580 | 0.662 | 188.125 | 38.453 | 56.885 | Measured |
### Motion-to-Text
| Protocol | Samples | BLEU-4 | ROUGE-L | CIDEr | BERT F1 | R@1 | R@2 | R@3 | Matching |
| -------- | ------: | -----: | ------: | ----: | -------: | --: | --: | --: | -------: |
| [HumanML3D M2T](../tasks/m2t.md) | 4,400 | - | - | - | - | - | - | - | - |
## Motion Representation
MotionGPT generates HumanML3D-263 features at 20 fps. Per frame:
| Slice | Dim | Meaning |
| ----- | --- | ------- |
| `root_rot_vel` | 1 | root angular velocity |
| `root_lin_vel` | 2 | root linear velocity in the horizontal plane |
| `root_y` | 1 | root height |
| `ric_data` | 63 | local joint positions |
| `rot_data` | 126 | local joint rotations in continuous 6D format |
| `local_vel` | 66 | local joint velocities |
| `foot_contact` | 4 | binary foot-contact labels |
The VQ-VAE converts normalized HumanML3D features into discrete motion tokens.
MotionGPT then treats those tokens as a language vocabulary item alongside text
tokens.
## Motius Components
| Component | Path |
| --------- | ---- |
| Pipeline | `motius.pipelines.motiongpt.MotionGPTPipeline` |
| Bundle | `motius.models.motiongpt.MotionGPTBundle` |
| Runtime | `motius.models.motiongpt.network.mGPT.archs` |
Only the inference-time MotionGPT modules required by the bundle are included
in this public package.
## Citation
```bibtex
@inproceedings{jiang2023motiongpt,
title={MotionGPT: Human Motion as a Foreign Language},
author={Jiang, Biao and Chen, Xin and Liu, Wen and Yu, Jingyi and Yu, Gang and Chen, Tao},
booktitle={Advances in Neural Information Processing Systems},
year={2023}
}
```
## Direct Loading
```python
from motius import Pipeline
pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-MotionGPT-HumanML3D")
```
|