File size: 3,010 Bytes
4fc0322 a0329ac 4fc0322 a0329ac 4fc0322 16f719f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | ---
library_name: motius
pipeline_tag: other
tags:
- motion-to-text
- motion-captioning
- humanml3d
- tm2t
---
<h1 align="center">TM2T Model Card</h1>
<p align="center">
<strong>Tokenized reciprocal motion-language translation, reproduced as a standalone Motius M2T pipeline.</strong>
</p>
<p align="center">
<a href="https://arxiv.org/abs/2207.01696">Paper</a> |
<a href="https://ericguo5513.github.io/TM2T/">Project Page</a> |
<a href="https://github.com/EricGuo5513/TM2T">Original GitHub</a> |
<a href="https://huggingface.co/ZeyuLing/Motius-TM2T-HumanML3D">Motius Checkpoint</a>
</p>
TM2T is the ECCV 2022 reciprocal text-to-motion and motion-to-text method. The
Motius release contains the HumanML3D VQ tokenizer, motion-to-text Transformer,
vocabulary, statistics, and inference runtime. It does not import an original
repository checkout.
## Release Snapshot
| Item | Value |
| ---- | ----- |
| Released task | M2T |
| Motion representation | HumanML3D-263, 20 fps |
| Motion tokenizer | 1,024-code VQ tokenizer |
| Caption model | 4-layer encoder / 4-layer decoder Transformer |
| Decoding | Beam search, beam size 2 |
| Checkpoint | [`ZeyuLing/Motius-TM2T-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-TM2T-HumanML3D) |
| Pipeline | `motius.pipelines.tm2t.TM2TPipeline` |
## Usage
```python
import numpy as np
from motius.pipelines.tm2t import TM2TPipeline
pipe = TM2TPipeline.from_pretrained(
"ZeyuLing/Motius-TM2T-HumanML3D",
bundle_kwargs={"device": "cuda"},
)
motion = np.load("sample.npy") # denormalized HumanML3D-263
caption = pipe.infer_m2t([motion], lengths=[len(motion)])[0]
```
## M2T Evaluation
Full 4,400-sample evaluation follows the shared [HumanML3D M2T protocol](../tasks/m2t.md).
Results are published only after the complete prediction set and metric artifact
pass the population and sample-ID checks.
| Samples | BLEU-4 | ROUGE-L | CIDEr | BERT F1 | R@1 | R@2 | R@3 | Matching |
| ------: | -----: | ------: | ----: | -------: | --: | --: | --: | -------: |
| 4,400 | - | - | - | - | - | - | - | - |
## Motion Representation
TM2T normalizes HumanML3D-263 features with its released training statistics.
The VQ encoder removes four contact dimensions, maps each clip to discrete
motion tokens, and the reciprocal Transformer translates those tokens to text.
## Motius Components
| Component | Path |
| --------- | ---- |
| Pipeline | `motius/pipelines/tm2t/pipeline.py` |
| Bundle | `motius/models/tm2t/bundle.py` |
| Runtime | `motius/models/tm2t/network.py` |
| License | `motius/models/tm2t/LICENSE` |
## Citation
```bibtex
@inproceedings{guo2022tm2t,
title={TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts},
author={Guo, Chuan and Zuo, Xinxin and Wang, Sen and Cheng, Li},
booktitle={European Conference on Computer Vision},
year={2022}
}
```
## Direct Loading
```python
from motius import Pipeline
pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-TM2T-HumanML3D")
```
|