ZeyuLing/Motius-Language-of-Motion

Language of Motion Model Card

Language of Motion (LoM) is a multimodal language model for text-conditioned motion and co-speech gesture generation. Motius currently wraps the paper's v1 release: HuBERT audio units, a Flan-T5 encoder-decoder, four body-part VQ tokenizers, and a global translation decoder.

• Paper: https://arxiv.org/abs/2412.10523
• Official source: https://github.com/Juzezhang/language_of_motion
• Official checkpoints: https://huggingface.co/JuzeZhang/language_of_motion
• Motius release: https://huggingface.co/ZeyuLing/Motius-Language-of-Motion
• Motius pipeline: motius.pipelines.language_of_motion.LanguageOfMotionPipeline
• Output: neutral SMPL-X, 55 axis-angle joints, 100 expression coefficients, root translation, 30 FPS

Integration status

The model bundle and both inference paths are integrated. The shared BEAT2 speech-to-gesture manifest and prediction contract are also available.

The released BEAT2 feature encoder, onset detector, velocity statistics, and SMPL-X metric path are integrated. On the fixed 15-clip speaker-2 population, the public demo checkpoint measures FGD = 5.334, BC = 7.860, and Diversity = 15.001 on the paper display scale. Table 1 reports 5.301 / 7.780 / 15.167, so the relative differences are +0.62% / +1.02% / -1.09%.

This is near numerical parity, not an exact paper reproduction. The released checkpoint is identified by the upstream README as a visualization demo rather than the paper's best/table checkpoint. The demo performs independent non-overlapping 120-frame chunk generation, while the paper evaluation entry point (test.py → val_a2m_forward) uses 116-frame advances with four-frame overlap and drops non-eight-frame tails. Motius also resets seed 1234 for every independently sharded clip. Until the paper checkpoint and its complete evaluation entry point are available and matched, the public checkpoint remains paper_table_eligible=false.

The official source remains an optional runtime rather than copied model code. Motius owns the adapter under motius/models/language_of_motion/ and the multi-task inference pipeline under motius/pipelines/language_of_motion/.

Evaluation Results

Task	Evaluator	Clips	FGD ↓	BC ↑	Diversity	uTMR FID ↓	uTMR Paired Dist. ↓
Speech-to-Gesture	BEAT2 Official	15	5.334	7.860	15.001	0.0626	20.3129

The uTMR columns use only canonical 30 FPS SMPL-22 joint positions; no text or expression coefficients are encoded, and metric FK sets SMPL-X translation to zero. FID is L2-normalized; paired distance uses native uTMR embeddings.

The T2M path is released for inference, but HumanML3D benchmark rows have not yet been materialized; this multi-task release must not be interpreted as a complete T2M leaderboard reproduction.

Release Snapshot

Item	Value
Tasks	T2M, Speech-to-Gesture
Native motion	SMPL-X 55-joint axis-angle pose + expression + translation
Motion rate	30 FPS
Motius artifact	ZeyuLing/Motius-Language-of-Motion
Upstream weights	JuzeZhang/language_of_motion@c15eb3a2240844c4b29abf1f71f62bf03160c335
Pipeline	motius.pipelines.language_of_motion.LanguageOfMotionPipeline
Leaderboard	BEAT2 Speech-to-Gesture
License	Apache-2.0 code; upstream checkpoint and SMPL-X terms also apply

The Motius artifact is an immutable release descriptor that resolves the four original upstream weight files by pinned revision and SHA-256 instead of duplicating them. This keeps the original authors' download and license record authoritative while making the Motius pipeline and inference profile explicit.

Preview

Open the audio-synchronized 15-case Three.js comparison. The browser compares BEAT2 GT with the released LoM output, supports camera control and timeline seeking, and exports the native full-SMPL-X NPZ.

Setup

git clone https://github.com/Juzezhang/language_of_motion \
  ref_repo/language_of_motion

pip install -e ".[language-of-motion]"

Run the official build_resources.sh inside that checkout to obtain Flan-T5, HuBERT, and the HuBERT quantizer. SMPL-X is only required for mesh rendering and mesh-based metrics, but its license still applies. Co-speech inference also requires the official fairseq checkout described by the LoM README. Motius defaults to vanilla Hugging Face T5 (flash_attention=False) for broad library compatibility. The LoM v1 demo-parity profile uses Python 3.10 or newer, bf16, TEST_LENGTH=120, the TurboT5 flash backend, and transformers==4.40.2. TurboT5 0.0.2 still uses the legacy tuple KV-cache; Transformers 4.42+ can pass the newer empty Cache object and fail before the first generated token:

PYTHON_BIN=python3.10 \
RUNTIME_ROOT=ref_repo/language_of_motion \
bash tools/setup_language_of_motion_v1_env.sh

On PyTorch 2.5, fairseq's unused legacy C++ extensions may fail to compile. HuBERT inference works with the pure-Python install:

pip install bitarray sacrebleu
READTHEDOCS=1 pip install --no-deps --no-build-isolation -e \
  ref_repo/language_of_motion/third_party/fairseq

export MOTIUS_LANGUAGE_OF_MOTION_ROOT="$PWD/ref_repo/language_of_motion"

Text-to-motion

from motius.pipelines.language_of_motion import LanguageOfMotionPipeline

pipeline = LanguageOfMotionPipeline.from_pretrained(
    "ZeyuLing/Motius-Language-of-Motion",
    bundle_kwargs={
        "runtime_root": "ref_repo/language_of_motion",
        "task": "text2motion",
        "device": "cuda",
    },
)
motion = pipeline.infer_t2m(["A person swings a golf club."])[0]
pipeline.save_npz(motion, "outputs/inference/language_of_motion/t2m/golf.npz")

LoM predicts its own duration. Passing lengths applies an output cap but does not pad a short generation. The released result is rotation-based SMPL-X, not HumanML3D-263; it must be evaluated through an explicit SMPL-X conversion or a cross-representation evaluator.

Speech-to-gesture

from motius.pipelines.language_of_motion import LanguageOfMotionPipeline

pipeline = LanguageOfMotionPipeline.from_pretrained(
    "ZeyuLing/Motius-Language-of-Motion",
    bundle_kwargs={
        "runtime_root": "ref_repo/language_of_motion",
        "task": "cospeech",
        "device": "cuda",
        "load_audio_tokenizer": True,
    },
)
motion = pipeline.infer_s2g(["speech.wav"])[0]
pipeline.save_npz(
    motion,
    "outputs/inference/language_of_motion/speech_to_gesture/speech.npz",
)

One task-specific bundle loads one task-specific language-model checkpoint. Loading both tasks simultaneously would duplicate the 220M language model and the compositional decoders, so applications should create separate pipelines.