---
license: other
license_name: research-only
tags:
- text-to-motion
- motion-generation
- humanml3d
- mixture-of-experts
language: en
---

# ViBES-T2M (Qwen-MoME): frozen-LLM text-to-motion at specialist-SOTA quality

Text-to-motion model in the ViBES AR-MoME (mixture-of-modality-experts) architecture: a **fully
frozen Qwen3-1.7B** text backbone plus a narrow trainable motion expert (~0.27B). The text LLM is
untouched — a visibility rule (text attends only to text; motion attends causally to text+motion)
makes freezing exact, so the base model's language ability is fully preserved.

## Results (HumanML3D test, Guo et al. protocol, 20 replications)

| Model | Params (trainable) | Frozen text LLM | R@1 | R@3 | FID |
|---|---|---|---|---|---|
| T2M-GPT | full | no | 0.492 | 0.679 | 0.141 |
| MoMask | full | no | 0.521 | 0.807 | 0.045 |
| **ViBES-T2M (this)** | **0.27B** | **yes** | **0.489** | **0.769** | **0.265** |

R@1 on par with T2M-GPT while keeping the entire conversational LLM intact.

## Demos

GT | ViBES-T2M (frozen 1.7B) | ViBES-9B:

*"a man walks forward, then squats to pick something up with both hands, stands back up, and resumes walking"*

![demo1](assets/demo_008463.gif)

*"the man runs backwards"*

![demo2](assets/demo_009613.gif)

## Files

- `vibes_t2m_qwen_mome_1p7b.pt` — motion-expert weights (`mot_state`), training config, metrics.
  The frozen base is downloaded from `Qwen/Qwen3-1.7B` at load time.
- `modeling_qwen_mome.py` — self-contained model definition (needs `transformers` with Qwen3).

## Motion tokenizer

MotionGPT's public HumanML3D VQ-VAE (`motiongpt_s3_h3d.tar`, 512 codes, 4x temporal downsample,
263-D HumanML3D features). Motion vocab = 514 (512 codes + BOS/EOS). Decode generated codes with
that VQ-VAE + the Comp_v6_KLD01 mean/std.

## Usage

```python
import torch
from modeling_qwen_mome import QwenMoMEForT2M

ck = torch.load("vibes_t2m_qwen_mome_1p7b.pt", map_location="cpu", weights_only=False)
model = QwenMoMEForT2M("Qwen/Qwen3-1.7B", d_mot=512, ffn_mot=4096, mot_vocab=514)
model.load_state_dict(ck["mot_state"], strict=False)   # base.* comes from the HF download
model = model.cuda().eval()

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
text = "Give me a gesture that corresponds to a person walks forward then squats down"
ids = torch.tensor([tok(text, add_special_tokens=False)["input_ids"]]).cuda()
codes = model.generate_motion(ids, max_new=60, greedy=False,
                              temperature=0.7, top_p=0.9, cfg_scale=2.5)
# codes: MotionGPT-512 VQ indices -> decode with motiongpt_s3_h3d.tar to 263-D motion
```

Recommended decoding: sampling `temperature=0.7, top_p=0.9` with classifier-free guidance
`cfg_scale` 2.5–3 (the model was trained with 10% caption dropout).

## Training recipe (for reproduction)

Frozen Qwen3-1.7B; per-layer motion expert (q/k/v/o + q/k-norm + RMSNorms + SwiGLU 512→4096),
own 514-token embedding and untied head; plain-arange positions; motion-only shifted CE;
bs 16, lr 2e-4 cosine, wd 0.05, clip 1.0, caption-dropout 0.1; peak at ~10k steps
(select checkpoints by generation quality on the official val split, not teacher-forced loss).
Plain PyTorch bf16 (no DeepSpeed).

## License

Research use only. The frozen backbone follows the Qwen3 license; the motion tokenizer follows
MotionGPT's license; HumanML3D data terms apply.

Part of the ViBES project (speech-language-behavior modeling).