README.md

---
license: apache-2.0
language:
- en
- zh
tags:
- motion-generation
- vision-language
- robotics
- qwen
- dual-stream
datasets:
- MotionVLA-Dataset
---

# MotionVLA

**MotionVLA** is an end-to-end vision-language-action model for humanoid motion generation. It combines a **Qwen3.5** autoregressive backbone (conditioned on a scene image and a text instruction) with **DSFT (Dual-Stream Frequency-domain Tokenizer)**, which decouples low-frequency pose semantics from high-frequency physical dynamics.

## Repository Contents

This HuggingFace repository contains:

| Path | Description |
|------|-------------|
| `tokenizer/` | DSFT tokenizer checkpoints |
| `tokenizer/base/` | Base stream BPE tokenizer (4096 vocab, 201-dim DCT) |
| `tokenizer/phys/` | Phys stream BPE tokenizer (4096 vocab, 75-dim DCT) |
| `dataset/` | Dataset index files (motion_path → relative paths) |

**Motion data files** (`.pt`) and **images** are stored in the companion dataset repo: `[your-hf-username]/MotionVLA-Dataset`

## Tokenizer Design

The DSFT tokenizer decomposes 276-dim ViMoGen motion into two streams:

```
276-dim motion (T frames)
    ↓ split by dimension
Base (201-dim): body_pose_6d + joints + root_orient + root_trans   ← low-freq semantic
Phys  (75-dim): joints_vel + root_vel + root_trans_vel             ← high-freq dynamics
    ↓ DCT along time axis, keep top K coefficients
    ↓ BPE encoding
Base tokens: ~477/sequence  (K=5,  vocab=4096)
Phys tokens: ~40/sequence   (K=15, vocab=4096)
```

Each motion sample is laid out as a unified autoregressive sequence:

```
[ M_BOS, b_1, ..., b_N, M_SEP, p_1, ..., p_M, M_EOS ]
```

where `b_i` are Base tokens and `p_j` are Phys tokens. A phase-aware logit mask
enforces the order `BASE → SEP → PHYS → EOS` at inference, so semantic pose
structure is generated before high-frequency physical dynamics.

## Token Vocabulary

The Qwen3.5 backbone vocabulary is extended with motion tokens (used in the
ms-swift training pipeline):

| Token type | ID range | Count |
|------------|----------|-------|
| Base motion tokens | 248320 – 252415 | 4096 |
| Phys motion tokens | 252416 – 256511 | 4096 |
| MOTION_BOS | 256512 | 1 |
| MOTION_SEP | 256513 | 1 |
| MOTION_EOS | 256514 | 1 |

## Usage

```python
from tokenizer.ds_fast_tokenizer import DSFTTokenizer
import numpy as np

# Load tokenizer
tok = DSFTTokenizer.load("tokenizer/checkpoints")

# Encode 276-dim motion
motion = np.load("motion.npy")  # shape: (T, 276)
result = tok.encode(motion)
# result["base_tokens"]: list of int (BPE IDs for base stream)
# result["phys_tokens"]: list of int (BPE IDs for phys stream)
# result["T"]: number of frames

# Decode back
base_recon, phys_recon = tok.decode(
    result["base_tokens"], result["phys_tokens"], result["T"])
# base_recon: (T, 201), phys_recon: (T, 75)
```

## Code

Training code and model architecture: [GitHub](https://github.com/AIGeeksGroup/MotionVLA)

## Citation

```bibtex
@article{motionvla2026,
  title={MotionVLA: Vision-Language-Action Model for Humanoid Motion},
  author={Zhang, Nonghai and Zhai, Siyu and Zhang, Zeyu and Tang, Hao},
  year={2026}
}
```