| --- |
| license: other |
| license_name: research-only |
| tags: |
| - text-to-motion |
| - motion-generation |
| - humanml3d |
| - mixture-of-experts |
| language: en |
| --- |
| |
| # ViBES-T2M (Qwen-MoME): frozen-LLM text-to-motion at specialist-SOTA quality |
|
|
| Text-to-motion model in the ViBES AR-MoME (mixture-of-modality-experts) architecture: a **fully |
| frozen Qwen3-1.7B** text backbone plus a narrow trainable motion expert (~0.27B). The text LLM is |
| untouched — a visibility rule (text attends only to text; motion attends causally to text+motion) |
| makes freezing exact, so the base model's language ability is fully preserved. |
|
|
| ## Results (HumanML3D test, Guo et al. protocol, 20 replications) |
|
|
| | Model | Params (trainable) | Frozen text LLM | R@1 | R@3 | FID | |
| |---|---|---|---|---|---| |
| | T2M-GPT | full | no | 0.492 | 0.679 | 0.141 | |
| | MoMask | full | no | 0.521 | 0.807 | 0.045 | |
| | **ViBES-T2M (this)** | **0.27B** | **yes** | **0.489** | **0.769** | **0.265** | |
|
|
| R@1 on par with T2M-GPT while keeping the entire conversational LLM intact. |
|
|
| ## Demos |
|
|
| GT | ViBES-T2M (frozen 1.7B) | ViBES-9B: |
|
|
| *"a man walks forward, then squats to pick something up with both hands, stands back up, and resumes walking"* |
|
|
|  |
|
|
| *"the man runs backwards"* |
|
|
|  |
|
|
| ## Files |
|
|
| - `vibes_t2m_qwen_mome_1p7b.pt` — motion-expert weights (`mot_state`), training config, metrics. |
| The frozen base is downloaded from `Qwen/Qwen3-1.7B` at load time. |
| - `modeling_qwen_mome.py` — self-contained model definition (needs `transformers` with Qwen3). |
|
|
| ## Motion tokenizer |
|
|
| MotionGPT's public HumanML3D VQ-VAE (`motiongpt_s3_h3d.tar`, 512 codes, 4x temporal downsample, |
| 263-D HumanML3D features). Motion vocab = 514 (512 codes + BOS/EOS). Decode generated codes with |
| that VQ-VAE + the Comp_v6_KLD01 mean/std. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from modeling_qwen_mome import QwenMoMEForT2M |
| |
| ck = torch.load("vibes_t2m_qwen_mome_1p7b.pt", map_location="cpu", weights_only=False) |
| model = QwenMoMEForT2M("Qwen/Qwen3-1.7B", d_mot=512, ffn_mot=4096, mot_vocab=514) |
| model.load_state_dict(ck["mot_state"], strict=False) # base.* comes from the HF download |
| model = model.cuda().eval() |
| |
| from transformers import AutoTokenizer |
| tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B") |
| text = "Give me a gesture that corresponds to a person walks forward then squats down" |
| ids = torch.tensor([tok(text, add_special_tokens=False)["input_ids"]]).cuda() |
| codes = model.generate_motion(ids, max_new=60, greedy=False, |
| temperature=0.7, top_p=0.9, cfg_scale=2.5) |
| # codes: MotionGPT-512 VQ indices -> decode with motiongpt_s3_h3d.tar to 263-D motion |
| ``` |
|
|
| Recommended decoding: sampling `temperature=0.7, top_p=0.9` with classifier-free guidance |
| `cfg_scale` 2.5–3 (the model was trained with 10% caption dropout). |
|
|
| ## Training recipe (for reproduction) |
|
|
| Frozen Qwen3-1.7B; per-layer motion expert (q/k/v/o + q/k-norm + RMSNorms + SwiGLU 512→4096), |
| own 514-token embedding and untied head; plain-arange positions; motion-only shifted CE; |
| bs 16, lr 2e-4 cosine, wd 0.05, clip 1.0, caption-dropout 0.1; peak at ~10k steps |
| (select checkpoints by generation quality on the official val split, not teacher-forced loss). |
| Plain PyTorch bf16 (no DeepSpeed). |
|
|
| ## License |
|
|
| Research use only. The frozen backbone follows the Qwen3 license; the motion tokenizer follows |
| MotionGPT's license; HumanML3D data terms apply. |
|
|
| Part of the ViBES project (speech-language-behavior modeling). |
|
|
