--- library_name: motius pipeline_tag: other tags: - motion-generation - text-to-motion - humanml3d - mogents license: mit --- # MoGenTS - Motion Generation Based on Spatial-Temporal Joint Modeling Text-to-motion baseline integrated into the motius Model Zoo. The runtime is self-contained under `motius.models.motion.mogents.network` and does not import the original repository at inference time. | | | |---|---| | **Task** | Text-to-Motion (T2M) | | **Bundle / Pipeline** | `MoGenTSBundle` / `MoGenTSPipeline` | | **Processed HF artifact** | [`ZeyuLing/Motius-MoGenTS-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-MoGenTS-HumanML3D) | | **Motion representation** | **HumanML3D-263** (263-dim, 20 fps, 22 joints) | | **Tokenizer** | dual-stream RVQ-VAE: 1D auxiliary tokens + 2D spatial-temporal tokens | | **Generator** | 1D/2D MaskTransformers + 1D/2D ResidualTransformers | | **Text encoder** | CLIP ViT-B/32 (frozen) | | **Paper** | *MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling*, Yuan et al., NeurIPS 2024 - [arXiv:2409.17686](https://arxiv.org/abs/2409.17686) | | **Original code** | https://github.com/weihaosky/mogents | --- ## Weights Self-contained motius artifact: | Artifact | Location | Contents | Status | |---|---|---|---| | MoGenTS HumanML3D | [`ZeyuLing/Motius-MoGenTS-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-MoGenTS-HumanML3D) | `vq.safetensors` + `mask_aux.safetensors` + `mask_ts.safetensors` + `res_aux.safetensors` + `res_ts.safetensors` + `length_est.safetensors` + `clip.safetensors` + `mogents_config.json` + `Mean.npy` / `Std.npy` | public Hub artifact | | local mirror | `checkpoints/mogents/humanml3d` | same layout | optional local cache | Convert the official checkpoints into a self-contained motius artifact: ```bash python3 scripts/eval/convert_mogents_checkpoint.py \ --weights_root logs \ --length_root checkpoints \ --out_dir checkpoints/mogents/humanml3d \ --verify ``` Expected artifact layout: ```text checkpoints/mogents/humanml3d/ mogents_config.json model_index.json vq.safetensors mask_aux.safetensors mask_ts.safetensors res_aux.safetensors res_ts.safetensors length_est.safetensors clip.safetensors Mean.npy Std.npy ``` ## Use ```python from motius.pipelines.mogents import MoGenTSPipeline pipe = MoGenTSPipeline.from_pretrained( "ZeyuLing/Motius-MoGenTS-HumanML3D", device="cuda", ) motions = pipe.infer_t2m( ["a person walks forward then turns around"], [120], ) # list of (T, 263) ``` For a local mirror: ```python pipe = MoGenTSPipeline.from_pretrained("checkpoints/mogents/humanml3d", device="cuda") ``` ## Motion Representation MoGenTS natively generates **HumanML3D-263** at 20 fps. For cross-model comparison with SMPL or MotionStreamer-272 methods, first generate the native 263-dim outputs and then use the validated bridge: ```text HumanML3D-263 -> SMPL motion_135 via IK refine-80 -> MotionStreamer-272 ``` The bridge is a representation-conversion diagnostic. It should not be treated as the native MoGenTS paper metric space. --- ## Evaluation Generate under the official HumanML3D test protocol and score with the HumanML3D-263 evaluator: ```bash python3 scripts/eval/mogents_t2m_h3d263.py \ --model_path checkpoints/mogents/humanml3d \ --out_dir outputs/evaluation/t2m/humanml3d_official_test/hml263/mogents_ts10_cfg4_rescfg5_seed0 python3 scripts/eval/verify_evaluators.py --which hml263 \ --hml263-pred outputs/evaluation/t2m/humanml3d_official_test/hml263/mogents_ts10_cfg4_rescfg5_seed0 \ --out-dir outputs/evaluation/t2m/humanml3d_official_test/hml263/mogents_ts10_cfg4_rescfg5_seed0/metrics ``` ### HumanML3D-263 evaluator (native space, n=3970) Metric JSON: `outputs/evaluation/t2m/humanml3d_official_test/hml263/mogents_ts10_cfg4_rescfg5_seed0/metrics/verify_hml263.json`. | Metric | motius MoGenTS | |---|---:| | FID down | 0.0806 | | R-Precision Top-1 / 2 / 3 up | 0.5219 / 0.7128 / 0.8056 | | Diversity -> | 9.4063 | | MM-Dist down | 2.9290 | | GT(real) R-Precision Top-1 / 2 / 3 | 0.5135 / 0.7108 / 0.8069 | | GT(real) Diversity / MM-Dist | 9.4527 / 2.9323 | ### SMPL motion_135 + MotionStreamer-272 evaluator Convert the same HumanML3D test predictions to SMPL `motion_135` and then to MotionStreamer-272: ```bash NUM_GPUS=8 NUM_SHARDS=8 N_REPEATS=20 \ bash scripts/eval/run_mogents_hml263_to_ms272_chain.sh ``` The restartable script runs the following stages with `--skip-existing`: ```bash python3 scripts/eval/hml263_to_smpl_ik.py \ --in-dir outputs/evaluation/t2m/humanml3d_official_test/hml263/mogents_ts10_cfg4_rescfg5_seed0 \ --out-dir outputs/evaluation/t2m/humanml3d_official_test/motion135/mogents_ts10_cfg4_rescfg5_seed0_ik80 \ --model-dir ref_repo/MDM/body_models \ --source-fps 20 --target-fps 30 \ --floor-align --refine-iters 80 --refine-lr 0.02 \ --device cuda --skip-existing python3 scripts/data/convert_motion135_to_h3d272.py \ --in-dir outputs/evaluation/t2m/humanml3d_official_test/motion135/mogents_ts10_cfg4_rescfg5_seed0_ik80 \ --out-dir outputs/evaluation/t2m/humanml3d_official_test/ms272/mogents_ts10_cfg4_rescfg5_seed0_ik80 \ --workers 8 --skip-existing python3 scripts/eval/verify_evaluators.py --which ms272 \ --ms272-pred outputs/evaluation/t2m/humanml3d_official_test/ms272/mogents_ts10_cfg4_rescfg5_seed0_ik80 \ --n-repeats 20 \ --out-dir outputs/evaluation/t2m/humanml3d_official_test/ms272/mogents_ts10_cfg4_rescfg5_seed0_ik80/metrics ``` Metric JSON: `outputs/evaluation/t2m/humanml3d_official_test/ms272/mogents_ts10_cfg4_rescfg5_seed0_ik80/metrics/verify_ms272.json`. Run summary: `outputs/evaluation/t2m/humanml3d_official_test/ms272/mogents_ts10_cfg4_rescfg5_seed0_ik80/metrics/run_summary.json`. | Metric | MoGenTS HML263 -> SMPL135 -> MS272 | MS272 GT(real) | |---|---:|---:| | FID down | 113.0856 | 0.0 | | R-Precision Top-1 / 2 / 3 up | 0.4764 / 0.6321 / 0.7099 | 0.7059 / 0.8569 / 0.9106 | | Diversity -> | 25.3033 | 27.3692 | | MM-Dist down | 19.4679 | 15.0066 | | Samples used | 7392 | 7392 | | Missing predictions skipped | 0 | - | Bridge outputs contain 4012 HML263 predictions, 4012 SMPL `motion_135` files, and 4012 MotionStreamer-272 files. The SMPL IK shard summaries report zero conversion failures and mean joint-fit MPJPE around 15.2-15.6 mm. ## Implementation Notes - **Architecture**: MoGenTS generates a 1D auxiliary token stream and a 2D spatial-temporal token grid, then decodes both streams together with the dual RVQ-VAE. - **Runtime package**: `motius/models/motion/mogents/network/` contains only the inference-time model components from the MIT-licensed upstream code. - **Artifact loading**: `MoGenTSBundle.from_pretrained` consumes local/HF-style artifacts and stores CLIP once as `clip.safetensors`; raw upstream `.tar` checkpoints are supported only through explicit converter/debug paths. - **Native representation**: generated outputs are HumanML3D-263. Any MotionStreamer-272 or SMPL `motion_135` comparison should be produced by the existing representation-conversion pipeline after generation. ## Direct Loading ```python from motius import Pipeline pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-MoGenTS-HumanML3D") ```