File size: 4,884 Bytes
58a1c51 aa7507d 09e6ee7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | ---
library_name: motius
pipeline_tag: video-to-video
license: other
tags:
- human-motion
- monocular-motion-capture
- smpl
---
# GEM-SMPL
GEM-SMPL is the SMPL video-motion-estimation release of **GEM: A Generalist
Model for Human Motion**, originally released as GENMO.
- Paper: [GEM: A Generalist Model for Human Motion](https://arxiv.org/abs/2505.01425)
- Official source: [NVlabs/GENMO](https://github.com/NVlabs/GENMO)
- Motius checkpoint:
[ZeyuLing/Motius-GEM-SMPL](https://huggingface.co/ZeyuLing/Motius-GEM-SMPL)
- Pinned source revision: `16bebf402d8893184249ee206d957b8248cd8310`
- Checkpoint SHA-256:
`1d15cbe2864d6de61a75e83fdbfe83bec3c7b183eee3d3dcdbd9107e4456454a`
**Tasks:** Monocular Motion Capture
## Supported Tasks
| Task | Public API | Input | Output |
| --- | --- | --- | --- |
| Monocular Motion Capture | `infer_monocular_motion_capture` | RGB video | `MonocularCaptureResult` |
## Checkpoint
The Motius Hugging Face artifact contains the exact official GEM-SMPL,
HMR2, ViTPose, and YOLO checkpoint bytes and a manifest with every SHA-256.
The runtime source is shipped inside the `motius` wheel at its pinned revision;
inference never imports another repository checkout.
SMPL and SMPL-X files are license-gated and are not redistributed. Download
them into `checkpoints/body_models/` using
[`checkpoints/body_models/README.md`](https://github.com/ZeyuLing/Motius/blob/main/checkpoints/body_models/README.md).
## Motion Representation
GEM-SMPL predicts camera-space and gravity-aligned global body parameters.
Motius preserves the native 21-joint body pose, root orientation, translation,
and ten shape coefficients. Geometry materialization exposes:
- SMPL-24 named joints;
- 6,890-vertex SMPL meshes;
- camera and world root trajectories;
- per-frame camera intrinsics;
- the source video clock without temporal resampling.
The internal SMPL-X body layer is converted with the same fixed sparse
SMPL-X-to-SMPL map used by the pinned implementation.
## Usage
Create an isolated environment without cloning GENMO:
```bash
python3.10 -m venv outputs/envs/gem-smpl
outputs/envs/gem-smpl/bin/pip install -e ".[gem-smpl]"
```
Run the standard task API:
```python
from pathlib import Path
from motius.motion.representation.monocular_capture import (
save_monocular_capture_result,
)
from motius.pipelines.gem_smpl import GemSmplPipeline
pipeline = GemSmplPipeline.from_pretrained(
"ZeyuLing/Motius-GEM-SMPL",
bundle_kwargs={
"python_executable": "outputs/envs/gem-smpl/bin/python",
"body_models_root": "checkpoints/body_models",
},
)
result = pipeline.infer_monocular_motion_capture(
"input.mp4",
output_root="outputs/gem_smpl/run_001",
materialize_geometry=True,
render=True,
)
save_monocular_capture_result(
result,
Path("outputs/gem_smpl/run_001/result.npz"),
)
```
`render=True` requests the upstream in-camera/world previews. It can be left
off for evaluation and batch inference.
## Demo
This 768px, 30 FPS preview renders the world-space SMPL vertices returned by
the public Motius pipeline.
[](https://github.com/ZeyuLing/Motius/blob/main/assets/model_zoo/gem_smpl/gem_smpl_tennis_world.mp4)
## Evaluation Results
Protocol: `3dpw_test_camera_v1`, one inference item per official 3DPW person
track using the released target-crop protocol.
| Coverage | MPJPE ↓ | PA-MPJPE ↓ | Acceleration ↓ |
| ---: | ---: | ---: | ---: |
| 100.00% | 64.46 mm | 46.45 mm | 5.713 m/s² |
## Stage Parity
The migration gate replays the same video, checkpoint bytes, licensed body
models, precomputed visual tensors, and random seed through the pinned official
source and the Motius package.
| Boundary | Fields | Requirement | Result |
| --- | ---: | --- | --- |
| Tracking | 1 | exact | pass |
| Keypoints | 1 | exact | pass |
| Visual features | 2 | exact | pass |
| Complete model input | 15 | exact | pass |
| Network and decoded output | 9 | exact | pass |
| SMPL geometry | 4 | exact | pass |
| Public result | 10 | exact | pass |
| **Total** | **42** | **`rtol=0`, `atol=0`** | **pass** |
```bash
python tools/verify_monocular_pipeline_parity.py \
--reference outputs/parity/gem_smpl/reference_trace.npz \
--candidate outputs/parity/gem_smpl/motius_trace.npz
```
## License
The vendored source retains the NVIDIA OneWay Noncommercial license. Public
weights retain the NVIDIA Open Model License. SMPL and SMPL-X have separate
terms. See the
[GEM-SMPL attributions](https://github.com/ZeyuLing/Motius/blob/main/motius/models/gem_smpl/ATTRIBUTIONS.md)
and the packaged license files before use.
## Direct Loading
```python
from motius import Pipeline
pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-GEM-SMPL")
```
|