ONNX
GGUF
English
text-to-motion
motion-generation
animation
3d-animation
character-animation
skeletal-animation
game-development
flow-matching
cpu-inference
qtmesheditor
feature-extraction
Instructions to use fernandotonon/QtMeshEditor-t2m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use fernandotonon/QtMeshEditor-t2m with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible embeddings server: llama serve -hf fernandotonon/QtMeshEditor-t2m:Q8_0 --embeddings
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible embeddings server: llama serve -hf fernandotonon/QtMeshEditor-t2m:Q8_0 --embeddings
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible embeddings server: ./build/bin/llama serve -hf fernandotonon/QtMeshEditor-t2m:Q8_0 --embeddings
- Llama new
- LM Studio
- Jan
- Ollama
How to use fernandotonon/QtMeshEditor-t2m with Ollama:
ollama run hf.co/fernandotonon/QtMeshEditor-t2m:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use fernandotonon/QtMeshEditor-t2m with Docker Model Runner:
docker model run hf.co/fernandotonon/QtMeshEditor-t2m:Q8_0
- Lemonade
How to use fernandotonon/QtMeshEditor-t2m with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull fernandotonon/QtMeshEditor-t2m:Q8_0
Run and chat with the model
lemonade run user.QtMeshEditor-t2m-Q8_0
List all available models
lemonade list
- Atomic Chat
Model card: QtMesh T2M v9 rewrite — benchmark methodology, seed failure case, per-file licences, v8 docs moved to PREVIOUS_VERSIONS.md
Browse files- .gitattributes +1 -0
- LICENSES.md +50 -0
- PREVIOUS_VERSIONS.md +155 -0
- README.md +338 -186
- media/compare_jump_seeds.gif +3 -0
.gitattributes
CHANGED
|
@@ -39,3 +39,4 @@ media/compare_jump.gif filter=lfs diff=lfs merge=lfs -text
|
|
| 39 |
media/compare_run.gif filter=lfs diff=lfs merge=lfs -text
|
| 40 |
media/compare_walk.gif filter=lfs diff=lfs merge=lfs -text
|
| 41 |
media/compare_waveR.gif filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 39 |
media/compare_run.gif filter=lfs diff=lfs merge=lfs -text
|
| 40 |
media/compare_walk.gif filter=lfs diff=lfs merge=lfs -text
|
| 41 |
media/compare_waveR.gif filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
media/compare_jump_seeds.gif filter=lfs diff=lfs merge=lfs -text
|
LICENSES.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Licences in this repository
|
| 2 |
+
|
| 3 |
+
`fernandotonon/QtMeshEditor-t2m` holds files under several licences. That is why the model card metadata says
|
| 4 |
+
`license: other` (`mixed-per-file`) instead of a single licence. This page lists every file.
|
| 5 |
+
|
| 6 |
+
| Path | Licence | Notes |
|
| 7 |
+
|---|---|---|
|
| 8 |
+
| `qtmesh_t2m/t2m-text.onnx` | CC0-1.0, with CC-BY-4.0 credits | QtMesh T2M v9 weights, trained from scratch by the QtMeshEditor project. See [Model weights](#model-weights). |
|
| 9 |
+
| `qtmesh_t2m/t2m-text.json` | CC0-1.0 | The model contract (shapes, embedding instruction, canonical rest pose). |
|
| 10 |
+
| `qtmesh_t2m/Qwen3-Embedding-0.6B-Q8_0.gguf` | Apache-2.0 | Qwen3-Embedding-0.6B by the Qwen team, Alibaba Cloud. Byte-identical to [Qwen/Qwen3-Embedding-0.6B-GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF). |
|
| 11 |
+
| `qtmesh_t2m/LICENSE-Qwen3-Embedding.txt` | — | The Apache-2.0 text for the file above. The upstream repository declares Apache-2.0 on its card but ships no LICENSE file, so this copy carries an attribution header. |
|
| 12 |
+
| `qtmesh_t2m/ATTRIBUTION.md` | — | The credits for the CC-BY-4.0 works behind `t2m-text.onnx`. **Keep it with the model.** |
|
| 13 |
+
| `t2m.onnx`, `t2m-vocab.json` | CC0-1.0, with CC-BY-4.0 credits | The previous v8.0 action-label model. Its credits are in `ATTRIBUTION-v2.md` and `ATTRIBUTION.md`. |
|
| 14 |
+
| `motion-library-v2.json` | Mixed: CC0-1.0 and CC-BY-4.0, per clip | The curated template clip library and the v8 training source. Credits are in `ATTRIBUTION-v2.md`; per-asset provenance is in [QtMeshEditor-motion-corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus). |
|
| 15 |
+
| `motion-library.json` | Mixed: CC0-1.0 and CC-BY-4.0, per clip | The older v1 template library. Credits are in `ATTRIBUTION.md`. |
|
| 16 |
+
| `ATTRIBUTION.md`, `ATTRIBUTION-v2.md` | — | The credits for the CC-BY-4.0 works. **Keep them with the data or anything derived from it.** |
|
| 17 |
+
| `media/*.gif` | CC0-1.0 | Renders made by the QtMeshEditor project of the two models' outputs on a QtMeshEditor-generated character. |
|
| 18 |
+
| `README.md`, `LICENSES.md`, `PREVIOUS_VERSIONS.md` | CC0-1.0 | Documentation. |
|
| 19 |
+
|
| 20 |
+
## Model weights
|
| 21 |
+
|
| 22 |
+
The QtMeshEditor project dedicates its own work in the model files to the public domain under
|
| 23 |
+
[CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/), as this repository has declared since it was
|
| 24 |
+
created. Part of the training data is licensed CC-BY-4.0, and the project treats the trained weights as
|
| 25 |
+
derived from that data. The credits in `ATTRIBUTION.md` / `ATTRIBUTION-v2.md` therefore travel with the weights.
|
| 26 |
+
If you redistribute the models, include those files.
|
| 27 |
+
|
| 28 |
+
## Training data that is not in this repository
|
| 29 |
+
|
| 30 |
+
**Clips generated with NVIDIA Kimodo.**
|
| 31 |
+
|
| 32 |
+
- About 3,400 clips generated with `Kimodo-SOMA-RP-v1.2` were used as training data.
|
| 33 |
+
- They are not redistributed here.
|
| 34 |
+
- Kimodo's outputs are not "Derivative Models" under the
|
| 35 |
+
[NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
| 36 |
+
(§1.1), and NVIDIA claims no ownership of outputs (§2.4).
|
| 37 |
+
- No Kimodo weights, code or Llama 3 text embeddings are in this repository or in the model.
|
| 38 |
+
|
| 39 |
+
**The CC0 / CC-BY-4.0 motion corpus.**
|
| 40 |
+
|
| 41 |
+
- It is published separately as
|
| 42 |
+
[fernandotonon/QtMeshEditor-motion-corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus),
|
| 43 |
+
with per-asset licences and credits.
|
| 44 |
+
- Non-commercial sources (AMASS, HumanML3D, KIT-ML, LAFAN1) and Mixamo were excluded.
|
| 45 |
+
|
| 46 |
+
## QtMeshEditor
|
| 47 |
+
|
| 48 |
+
The editor that runs these models is MIT-licensed: [github.com/fernandotonon/QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor).
|
| 49 |
+
Its third-party model licences are documented in
|
| 50 |
+
[`THIRD_PARTY_AI_MODELS.md`](https://github.com/fernandotonon/QtMeshEditor/blob/master/THIRD_PARTY_AI_MODELS.md).
|
PREVIOUS_VERSIONS.md
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Previous versions
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
This page keeps the documentation of the models that preceded **QtMesh T2M (v9)**, the current model described in
|
| 5 |
+
the [main model card](README.md). The v8 files are still in this repository: QtMeshEditor uses `t2m.onnx` for its
|
| 6 |
+
opt-in action-label path (`qtmesh anim <rig> --generate "<prompt>" --model`) and `motion-library-v2.json` as its
|
| 7 |
+
template clip library.
|
| 8 |
+
|
| 9 |
+
The text below is the v8.0 model card as published on 2026-08-29, unchanged except for the file list.
|
| 10 |
+
|
| 11 |
+
---
|
| 12 |
+
## v8.0 action-label model
|
| 13 |
+
|
| 14 |
+
Generative text→skeletal-animation model shipped with
|
| 15 |
+
[QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor)
|
| 16 |
+
(`qtmesh anim <rig> --generate "<prompt>" --model`).
|
| 17 |
+
|
| 18 |
+
**v8.0** — flow-matching DiT, 21.7M params, 30 actions.
|
| 19 |
+
|
| 20 |
+
## Files
|
| 21 |
+
|
| 22 |
+
- `t2m.onnx` + `t2m-vocab.json` — the generative model + its vocab/contract.
|
| 23 |
+
- Earlier checkpoints (v4 and later) — in the repository's
|
| 24 |
+
[commit history](https://huggingface.co/fernandotonon/QtMeshEditor-t2m/commits/main).
|
| 25 |
+
- `motion-library-v2.json` — the curated **template clip library** (150 clips).
|
| 26 |
+
It is both the fallback for prompts outside the model vocab *and* — since
|
| 27 |
+
v8.0 — the model's training data. This is the exact library v8.0 trained on.
|
| 28 |
+
- `ATTRIBUTION-v2.md` — **required credits** for the CC-BY clips in that
|
| 29 |
+
library. It must ship with the data and with anything derived from it.
|
| 30 |
+
|
| 31 |
+
## Dataset
|
| 32 |
+
|
| 33 |
+
The published library is the full v8.0 training source: 150 clips drawn from 31
|
| 34 |
+
source assets — 27 CC-BY-4.0, plus CC0 material (largely Quaternius packs) and
|
| 35 |
+
17 clips the author owns outright. Every CC-BY asset is credited in
|
| 36 |
+
`ATTRIBUTION-v2.md`; no clip in the library lacks a credit where one is due.
|
| 37 |
+
|
| 38 |
+
The raw source assets the library was built from — plus per-asset provenance
|
| 39 |
+
and the build scripts — live in the companion dataset repo
|
| 40 |
+
[**fernandotonon/QtMeshEditor-motion-corpus**](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus),
|
| 41 |
+
which also carries a copy of this library under `derived/`.
|
| 42 |
+
|
| 43 |
+
To rebuild the training cache from it, see **Reproducing** below. The cache
|
| 44 |
+
itself is not published — it is fully regenerable from this library, and it
|
| 45 |
+
stamps its own `buildArgv`/`buildFlags` so any `.npz` can state how it was
|
| 46 |
+
built.
|
| 47 |
+
|
| 48 |
+
## What changed in v8.0
|
| 49 |
+
|
| 50 |
+
Earlier versions trained on the **CMU MoCap** corpus and inherited a facing
|
| 51 |
+
problem: generated walks moved convincingly but oriented backwards. Measuring
|
| 52 |
+
against a real reference walk showed why — CMU "walk" windows sit 1.73–2.07 rad
|
| 53 |
+
away from it, *further than a real punch, dance or jump*, and the real-vs-real
|
| 54 |
+
floor is 0.437. The corpus never carried the correct convention, so the model
|
| 55 |
+
could not learn it. v6.x models scored 1.89 while their own training data
|
| 56 |
+
scored 1.73: they were faithfully reproducing bad data.
|
| 57 |
+
|
| 58 |
+
v8.0 trains on the **curated template clips** instead (`--library-repeat 40`),
|
| 59 |
+
which measure 0.303–0.462 against the real walk. Plus two data gates and one
|
| 60 |
+
new loss:
|
| 61 |
+
|
| 62 |
+
- **Anatomical ankle gate** — rejects clips whose mean knee→foot angle exceeds
|
| 63 |
+
70°. Caught `march` (106°), `throw` (101°) and `hey` (90°): the foot folded
|
| 64 |
+
perpendicular to the shin in *100%* of their windows.
|
| 65 |
+
- **Minimum resolved roles (16/22)** — `hey` resolved only 9, with all four
|
| 66 |
+
knee/ankle roles invalid, so the model had no leg data and invented legs.
|
| 67 |
+
- **Leg-chain loss** — keeps a walk's bend in the **knee** rather than the
|
| 68 |
+
ankle. The v7.x walk bent the knee 25° and the ankle 31° (real: 38°/10°),
|
| 69 |
+
which renders as a near-straight knee with a false second joint below it.
|
| 70 |
+
|
| 71 |
+
### Measured against a real reference walk
|
| 72 |
+
|
| 73 |
+
| | v7.x | **v8.0** | real |
|
| 74 |
+
|---|---|---|---|
|
| 75 |
+
| knee/ankle split | 25°/31° | **41°/8°** | 38°/10° |
|
| 76 |
+
| leg-chain penalty | 0.359 | **0.0003** | 0.014 |
|
| 77 |
+
| walk refDist | 0.287 | **0.243** | 0.437 (real-vs-real floor) |
|
| 78 |
+
| run refDist | 0.472 | **0.324** | — |
|
| 79 |
+
|
| 80 |
+
Scored against the training distribution across 20 actions, v8.0 is closer to
|
| 81 |
+
ground truth on 8, further on 3, and tied on 9. Largest gains: `wave`
|
| 82 |
+
0.710→0.082 (the arm now actually goes overhead), `idle`, `climb`, `salute`,
|
| 83 |
+
`punch`, `sit`.
|
| 84 |
+
|
| 85 |
+
## Vocabulary (30 actions)
|
| 86 |
+
|
| 87 |
+
```
|
| 88 |
+
attack boxing buildloop cartgiveloop climb cough crouch cut dance death
|
| 89 |
+
fall farmloop hit idle jump kick pickup punch run salute shake shoot
|
| 90 |
+
sit stonegiveloop strafeleft straferight walk wave woodgiveloop working
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
`march`, `throw`, `hey` and `confession` were **removed** — their source clips
|
| 94 |
+
are anatomically broken (see the ankle gate above). Those prompts fall back to
|
| 95 |
+
the curated template library, which for `march`/`throw` is strictly better than
|
| 96 |
+
what the model produced. `march`/`marching` resolves to `walk`.
|
| 97 |
+
|
| 98 |
+
Quality is uneven across actions: `walk`, `run`, `wave`, `cough`, `death`,
|
| 99 |
+
`pickup` and `attack` are the strongest; `dance` and `idle` are weak (`dance`
|
| 100 |
+
averages five genuinely different dances under one label; `idle` is nearly
|
| 101 |
+
static). This is a **template-quality** ceiling, not an architecture one ��� more
|
| 102 |
+
curated clips is the improvement path.
|
| 103 |
+
|
| 104 |
+
## Training data — permissive only
|
| 105 |
+
|
| 106 |
+
The curated library is CC0/CC-BY content (Sketchfab, OpenGameArt, Quaternius
|
| 107 |
+
packs) plus clips the author owns; credits are in `ATTRIBUTION-v2.md`, **which
|
| 108 |
+
must ship with the library and with anything derived from it**. Earlier CMU
|
| 109 |
+
MoCap windows (commercial-OK) remain in the corpus path but no longer dominate
|
| 110 |
+
training. AMASS / HumanML3D / KIT-ML are **excluded** (non-commercial).
|
| 111 |
+
Zombie/fruit/non-humanoid characters are filtered out, as are clips failing the
|
| 112 |
+
anatomical ankle gate or resolving fewer than 16 of the 22 canonical roles.
|
| 113 |
+
|
| 114 |
+
## Architecture (v8.0)
|
| 115 |
+
|
| 116 |
+
- **6D-rotation** representation (Zhou et al. 2019), column-packed.
|
| 117 |
+
- **Flow-matching DiT**, dim 384 × 8 layers, ~21.7M params — the model predicts
|
| 118 |
+
a velocity field; the host runs a fixed Euler schedule baked into the graph.
|
| 119 |
+
- Auxiliary losses: contralateral gait phase, spine-twist penalty, travel
|
| 120 |
+
direction, limb amplitude band, jitter band, gait periodicity, and the v8.0
|
| 121 |
+
leg-chain term — each gated to the action classes where it is meaningful.
|
| 122 |
+
- Exports to ONNX as a single forward pass.
|
| 123 |
+
|
| 124 |
+
## I/O contract
|
| 125 |
+
|
| 126 |
+
```
|
| 127 |
+
input "tokens" float32 [1, V] one-hot over the fixed action vocab (see t2m-vocab.json)
|
| 128 |
+
input "seed" float32 [1, Z] latent noise (host samples ~N(0,0.5) and does best-of-N)
|
| 129 |
+
output "motion" float32 [1, T, C] C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
`t2m-vocab.json` ships the `{vocab, Z, T, C, J, fps, frame, restDir}` the host
|
| 133 |
+
needs. `frame: "world"` marks the WORLD-frame convention (the retarget takes a
|
| 134 |
+
world delta); `fps: 30`, `T: 60` (2.0 s).
|
| 135 |
+
|
| 136 |
+
## Reproducing
|
| 137 |
+
|
| 138 |
+
`scripts/prep-t2m-v6.py` + `scripts/train-t2m-flow-v5.py` in the QtMeshEditor
|
| 139 |
+
repo (one-time, offline dev tools — the app never runs Python):
|
| 140 |
+
|
| 141 |
+
`--corpus` is optional and points at the raw CMU-derived corpus, which is NOT
|
| 142 |
+
published here; with `--library-repeat 40` the curated library dominates
|
| 143 |
+
training either way, so omitting it reproduces the same shape of dataset from
|
| 144 |
+
the published files alone:
|
| 145 |
+
|
| 146 |
+
```
|
| 147 |
+
python3 scripts/prep-t2m-v6.py --library motion-library-v2.json \
|
| 148 |
+
--library-repeat 40 --min-roles 16 --out t2m.npz
|
| 149 |
+
python3 scripts/train-t2m-flow-v5.py --data t2m.npz --out flow --dim 384 --layers 8 \
|
| 150 |
+
--phase-weight 0.08 --twist-weight 0.5 --travel-weight 0.2 --amp-weight 0.4 \
|
| 151 |
+
--jitter-weight 0.4 --period-weight 0.5 --legchain-weight 0.35 --loco-boost 6.0
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
The cache stamps its own `buildArgv`/`buildFlags`, so any `.npz` can report how
|
| 155 |
+
it was built.
|
README.md
CHANGED
|
@@ -1,221 +1,373 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
tags:
|
| 4 |
- text-to-motion
|
|
|
|
| 5 |
- animation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
- onnx
|
| 7 |
- gguf
|
|
|
|
| 8 |
- qtmesheditor
|
| 9 |
---
|
| 10 |
|
| 11 |
-
#
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
**
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
It
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|---|---|---|
|
| 41 |
-
| Download | ~
|
| 42 |
-
| Text
|
| 43 |
-
|
|
| 44 |
-
|
|
| 45 |
-
|
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
-
|
| 55 |
-
|
| 56 |
-
prompt
|
| 57 |
-
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
##
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
[
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
**v8.0** — flow-matching DiT, 21.7M params, 30 actions.
|
| 86 |
-
|
| 87 |
-
## Files
|
| 88 |
-
|
| 89 |
-
- `t2m.onnx` + `t2m-vocab.json` — the generative model + its vocab/contract.
|
| 90 |
-
- `t2m-v61.onnx` / `t2m-v4.onnx` — previous versions, kept for rollback.
|
| 91 |
-
- `motion-library-v2.json` — the curated **template clip library** (150 clips).
|
| 92 |
-
It is both the fallback for prompts outside the model vocab *and* — since
|
| 93 |
-
v8.0 — the model's training data. This is the exact library v8.0 trained on.
|
| 94 |
-
- `ATTRIBUTION-v2.md` — **required credits** for the CC-BY clips in that
|
| 95 |
-
library. It must ship with the data and with anything derived from it.
|
| 96 |
-
|
| 97 |
-
## Dataset
|
| 98 |
|
| 99 |
-
|
| 100 |
-
source assets — 27 CC-BY-4.0, plus CC0 material (largely Quaternius packs) and
|
| 101 |
-
17 clips the author owns outright. Every CC-BY asset is credited in
|
| 102 |
-
`ATTRIBUTION-v2.md`; no clip in the library lacks a credit where one is due.
|
| 103 |
|
| 104 |
-
The
|
| 105 |
-
|
| 106 |
-
[**fernandotonon/QtMeshEditor-motion-corpus**](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus),
|
| 107 |
-
which also carries a copy of this library under `derived/`.
|
| 108 |
|
| 109 |
-
|
| 110 |
-
itself is not published — it is fully regenerable from this library, and it
|
| 111 |
-
stamps its own `buildArgv`/`buildFlags` so any `.npz` can state how it was
|
| 112 |
-
built.
|
| 113 |
|
| 114 |
-
|
|
|
|
|
|
|
| 115 |
|
| 116 |
-
|
| 117 |
-
problem: generated walks moved convincingly but oriented backwards. Measuring
|
| 118 |
-
against a real reference walk showed why — CMU "walk" windows sit 1.73–2.07 rad
|
| 119 |
-
away from it, *further than a real punch, dance or jump*, and the real-vs-real
|
| 120 |
-
floor is 0.437. The corpus never carried the correct convention, so the model
|
| 121 |
-
could not learn it. v6.x models scored 1.89 while their own training data
|
| 122 |
-
scored 1.73: they were faithfully reproducing bad data.
|
| 123 |
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 127 |
|
| 128 |
-
- **
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
|
| 137 |
-
|
| 138 |
|
| 139 |
-
|
|
| 140 |
-
|---|---|
|
| 141 |
-
|
|
| 142 |
-
|
|
| 143 |
-
|
|
| 144 |
-
|
|
|
|
|
| 145 |
|
| 146 |
-
|
| 147 |
-
ground truth on 8, further on 3, and tied on 9. Largest gains: `wave`
|
| 148 |
-
0.710→0.082 (the arm now actually goes overhead), `idle`, `climb`, `salute`,
|
| 149 |
-
`punch`, `sit`.
|
| 150 |
|
| 151 |
-
|
| 152 |
|
| 153 |
-
```
|
| 154 |
-
|
| 155 |
-
fall farmloop hit idle jump kick pickup punch run salute shake shoot
|
| 156 |
-
sit stonegiveloop strafeleft straferight walk wave woodgiveloop working
|
| 157 |
```
|
| 158 |
|
| 159 |
-
|
| 160 |
-
are anatomically broken (see the ankle gate above). Those prompts fall back to
|
| 161 |
-
the curated template library, which for `march`/`throw` is strictly better than
|
| 162 |
-
what the model produced. `march`/`marching` resolves to `walk`.
|
| 163 |
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
static). This is a **template-quality** ceiling, not an architecture one — more
|
| 168 |
-
curated clips is the improvement path.
|
| 169 |
|
| 170 |
-
|
|
|
|
|
|
|
|
|
|
| 171 |
|
| 172 |
-
|
| 173 |
-
packs) plus clips the author owns; credits are in `ATTRIBUTION-v2.md`, **which
|
| 174 |
-
must ship with the library and with anything derived from it**. Earlier CMU
|
| 175 |
-
MoCap windows (commercial-OK) remain in the corpus path but no longer dominate
|
| 176 |
-
training. AMASS / HumanML3D / KIT-ML are **excluded** (non-commercial).
|
| 177 |
-
Zombie/fruit/non-humanoid characters are filtered out, as are clips failing the
|
| 178 |
-
anatomical ankle gate or resolving fewer than 16 of the 22 canonical roles.
|
| 179 |
|
| 180 |
-
|
| 181 |
|
| 182 |
-
|
| 183 |
-
-
|
| 184 |
-
|
| 185 |
-
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
## I/O contract
|
| 191 |
-
|
| 192 |
-
```
|
| 193 |
-
input "tokens" float32 [1, V] one-hot over the fixed action vocab (see t2m-vocab.json)
|
| 194 |
-
input "seed" float32 [1, Z] latent noise (host samples ~N(0,0.5) and does best-of-N)
|
| 195 |
-
output "motion" float32 [1, T, C] C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]
|
| 196 |
-
```
|
| 197 |
-
|
| 198 |
-
`t2m-vocab.json` ships the `{vocab, Z, T, C, J, fps, frame, restDir}` the host
|
| 199 |
-
needs. `frame: "world"` marks the WORLD-frame convention (the retarget takes a
|
| 200 |
-
world delta); `fps: 30`, `T: 60` (2.0 s).
|
| 201 |
|
| 202 |
-
|
| 203 |
|
| 204 |
-
|
| 205 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 206 |
|
| 207 |
-
`-
|
| 208 |
-
published here; with `--library-repeat 40` the curated library dominates
|
| 209 |
-
training either way, so omitting it reproduces the same shape of dataset from
|
| 210 |
-
the published files alone:
|
| 211 |
|
| 212 |
-
```
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
python3 scripts/train-t2m-flow-v5.py --data t2m.npz --out flow --dim 384 --layers 8 \
|
| 216 |
-
--phase-weight 0.08 --twist-weight 0.5 --travel-weight 0.2 --amp-weight 0.4 \
|
| 217 |
-
--jitter-weight 0.4 --period-weight 0.5 --legchain-weight 0.35 --loco-boost 6.0
|
| 218 |
```
|
| 219 |
|
| 220 |
-
The
|
| 221 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: mixed-per-file
|
| 4 |
+
license_link: https://huggingface.co/fernandotonon/QtMeshEditor-t2m/blob/main/LICENSES.md
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
library_name: onnx
|
| 8 |
+
datasets:
|
| 9 |
+
- fernandotonon/QtMeshEditor-motion-corpus
|
| 10 |
tags:
|
| 11 |
- text-to-motion
|
| 12 |
+
- motion-generation
|
| 13 |
- animation
|
| 14 |
+
- 3d-animation
|
| 15 |
+
- character-animation
|
| 16 |
+
- skeletal-animation
|
| 17 |
+
- game-development
|
| 18 |
+
- flow-matching
|
| 19 |
- onnx
|
| 20 |
- gguf
|
| 21 |
+
- cpu-inference
|
| 22 |
- qtmesheditor
|
| 23 |
---
|
| 24 |
|
| 25 |
+
# QtMesh T2M — Lightweight Text-to-3D-Motion Generation
|
| 26 |
+
|
| 27 |
+
QtMesh T2M turns a sentence such as *"a person waves with the right hand"* into a skeletal animation for a
|
| 28 |
+
rigged 3D character. The [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor) project trained it from
|
| 29 |
+
scratch. It is **not** a quantized or distilled copy of NVIDIA Kimodo.
|
| 30 |
+
|
| 31 |
+
- **Two models, loaded separately.** The motion generator is a custom **22.6 M-parameter flow-matching
|
| 32 |
+
transformer**, run with ONNX Runtime. The prompt is read by **Qwen3-Embedding-0.6B** (0.6 B parameters, Q8_0
|
| 33 |
+
GGUF), run with llama.cpp.
|
| 34 |
+
- **No Python at inference.** Inside QtMeshEditor everything runs natively in C++.
|
| 35 |
+
- **Small enough for modest hardware.** It is about a 740 MB download and peaks at under 1 GB of memory.
|
| 36 |
+
It runs on the CPU: ONNX Runtime's CPU provider, and llama.cpp on Apple Silicon.
|
| 37 |
+
- **Integrated with the editor.** QtMeshEditor maps the output onto your character's rig, can add root travel,
|
| 38 |
+
and exports the result as glTF/GLB or FBX.
|
| 39 |
+
- **Experimental.** Its motion quality is clearly below [NVIDIA Kimodo](https://github.com/nv-tlabs/kimodo), which
|
| 40 |
+
stays QtMeshEditor's default. QtMesh T2M is the option for machines that cannot run Kimodo comfortably.
|
| 41 |
+
|
| 42 |
+
## Contents
|
| 43 |
+
|
| 44 |
+
- [Benchmark: QtMesh T2M vs NVIDIA Kimodo](#benchmark-qtmesh-t2m-vs-nvidia-kimodo)
|
| 45 |
+
- [Visual results](#visual-results--qtmesh-t2m-vs-nvidia-kimodo)
|
| 46 |
+
- [How it works](#how-it-works)
|
| 47 |
+
- [Training](#training)
|
| 48 |
+
- [Usage](#usage)
|
| 49 |
+
- [Files and tensor contract](#files-and-tensor-contract)
|
| 50 |
+
- [Limitations](#limitations)
|
| 51 |
+
- [License and attribution](#license-and-attribution)
|
| 52 |
+
- [Previous versions](#previous-versions)
|
| 53 |
+
- [Links](#links)
|
| 54 |
+
|
| 55 |
+
## Benchmark: QtMesh T2M vs NVIDIA Kimodo
|
| 56 |
+
|
| 57 |
+
These are the QtMeshEditor project's own measurements, not an independent benchmark. Each run generated one
|
| 58 |
+
3-second clip for the same character.
|
| 59 |
+
|
| 60 |
+
| | **QtMesh T2M** | NVIDIA Kimodo (native runtime) |
|
| 61 |
|---|---|---|
|
| 62 |
+
| Download | **~740 MB** | ~9.2 GB |
|
| 63 |
+
| Text encoder | Qwen3-Embedding-0.6B (Q8_0) | Llama-3-LLM2Vec-8B (Q8_0) |
|
| 64 |
+
| Text encoding | **~0.2–0.4 s** | ~1.1 s, plus loading the 8 B encoder |
|
| 65 |
+
| Motion sampling | **~2.7 s** (24 flow steps) | ~14 s (100 diffusion steps) |
|
| 66 |
+
| Whole CLI run (import, generate, export) | **~5 s** | ~18–21 s |
|
| 67 |
+
| Peak resident memory | **~0.87 GB** | ~7.6–8.2 GB |
|
| 68 |
+
|
| 69 |
+
**How it was measured**
|
| 70 |
+
|
| 71 |
+
- **Hardware:** Apple M5, 24 GB unified memory, macOS.
|
| 72 |
+
- **Runtime:** QtMeshEditor 3.60 with ONNX Runtime 1.31.1 on the CPU execution provider. Both text encoders run as
|
| 73 |
+
Q8_0 GGUFs through the bundled llama.cpp. Kimodo used QtMeshEditor's native (Python-free) port of
|
| 74 |
+
`Kimodo-SOMA-RP-v1.2`, with its post-processing on.
|
| 75 |
+
- **Method:**
|
| 76 |
+
- The command was `qtmesh anim char.fbx --generate "<prompt>" --backend <kimodo|qtmesh_t2m> --duration 3 --seed 1 --json -o out.fbx`.
|
| 77 |
+
- Each backend ran once per prompt, for four prompts: walk, run, wave with the right hand, jump.
|
| 78 |
+
- Wall time and maximum resident set size come from `/usr/bin/time -l`.
|
| 79 |
+
- The text-encoding and sampling times are the ones the CLI reports in `--json`.
|
| 80 |
+
- The "whole CLI run" covers process start, importing the character, generating, retargeting and exporting the
|
| 81 |
+
FBX.
|
| 82 |
+
- The models were already downloaded.
|
| 83 |
+
- **Not controlled or not recorded:**
|
| 84 |
+
- File-cache warmth.
|
| 85 |
+
- Background load.
|
| 86 |
+
- Repeated trials. Each figure is one run per prompt; the ranges span the four prompts.
|
| 87 |
+
- **Root motion differed:** Kimodo ran with `--no-root-motion`, while QtMesh T2M used its default foot-driven root
|
| 88 |
+
motion.
|
| 89 |
+
|
| 90 |
+
Smaller and faster does **not** mean equivalent. Kimodo's motions are clearly more detailed and reliable; compare
|
| 91 |
+
the clips below.
|
| 92 |
+
|
| 93 |
+
## Visual results — QtMesh T2M vs NVIDIA Kimodo
|
| 94 |
+
|
| 95 |
+
The same generated character and prompt are used on both sides. Each clip is 3 seconds, rendered in place
|
| 96 |
+
(root motion off), with one sample per model. **Left: NVIDIA Kimodo. Right: QtMesh T2M.** The caption in each GIF
|
| 97 |
+
gives the seed.
|
| 98 |
+
|
| 99 |
+
| Prompt | Seeds (Kimodo / QtMesh T2M) | Comparison | What to look for |
|
| 100 |
+
|---|---|---|---|
|
| 101 |
+
| "a person walks forward" | 1 / 1 |  | Both read as a walk cycle. QtMesh T2M's arms swing less. |
|
| 102 |
+
| "a person runs forward" | 1 / 1 |  | Both read as a run. Kimodo keeps the elbows bent in a running position; QtMesh T2M's arms hang straighter. |
|
| 103 |
+
| "a person waves with the right hand" | 1 / 1 |  | Both raise the character's **right** hand, the side named in the prompt; the text embedding carries left/right. QtMesh T2M waves with a large overhead arm, Kimodo with a smaller wave near the head. |
|
| 104 |
+
| "a person jumps" | 1 / **2** |  | **Cherry-picked:** the QtMesh T2M clip uses seed 2, not seed 1. See the failure case below. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
+
### Failure case: seed sensitivity
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
+
The two clips below come from the same model and prompt with different seeds. The left one is what you get with
|
| 109 |
+
seed 1, the seed used for the other comparisons.
|
|
|
|
|
|
|
| 110 |
|
| 111 |
+

|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
+
Across ten seeds of "a person jumps", four bent the knees 70–86° into a clear jump, and four barely moved. The
|
| 114 |
+
showcased clips are therefore not a statistically representative sample. Treat them as examples of what the
|
| 115 |
+
model *can* produce. In practice, generate a few variations and keep the best one.
|
| 116 |
|
| 117 |
+
## How it works
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
|
| 119 |
+
```
|
| 120 |
+
Text prompt
|
| 121 |
+
→ Qwen3-Embedding-0.6B (llama.cpp) 1024-d, L2-normalised sentence embedding
|
| 122 |
+
→ Flow-matching transformer (ONNX) 22.6 M params, 24 Euler steps, classifier-free guidance
|
| 123 |
+
→ Canonical 22-joint motion joint rotations, 30 fps, up to 150 frames (5 s)
|
| 124 |
+
→ QtMeshEditor retargeting mapped onto your character's skeleton
|
| 125 |
+
→ glTF/GLB or FBX animation
|
| 126 |
+
```
|
| 127 |
|
| 128 |
+
- **Text conditioning.**
|
| 129 |
+
- The prompt is prefixed with a fixed instruction:
|
| 130 |
+
`Instruct: Given a description of a human body motion, retrieve motions that match it\nQuery:`.
|
| 131 |
+
- Qwen3-Embedding then encodes it with last-token pooling, and the vector is L2-normalised.
|
| 132 |
+
- Because the model sees a sentence embedding rather than an action label, wording, body side ("with the right
|
| 133 |
+
hand"), speed and style all reach it.
|
| 134 |
+
- **Motion model.**
|
| 135 |
+
- A transformer of width 384 and 8 layers, conditioned on the text and the flow time through **AdaLN-Zero**.
|
| 136 |
+
- It is trained with **rectified flow (flow matching)**: it learns a velocity field that carries random noise to
|
| 137 |
+
a motion.
|
| 138 |
+
- The ONNX graph contains the whole sampler, unrolled: **24 Euler steps with classifier-free guidance
|
| 139 |
+
(scale 2.0)**.
|
| 140 |
+
- The model generates **rotations for 22 canonical joints** (pelvis, spine, neck, head, arms and legs), in the
|
| 141 |
+
world frame, at 30 fps.
|
| 142 |
+
- Clips are up to 150 frames (5 s). The length is an input, and frames past it are padding.
|
| 143 |
+
- **What the model does not generate.**
|
| 144 |
+
- It produces no root translation and no vertical root displacement. A jump therefore stays on the spot.
|
| 145 |
+
- When **Root motion** is on, QtMeshEditor *estimates* horizontal travel from the planted foot. Measured: walk
|
| 146 |
+
~1.4 m/s, run ~1.9 m/s. The character never rises.
|
| 147 |
+
- **Editor post-processing.**
|
| 148 |
+
- Long clips in the training data start from rest, so the editor generates up to 1 s of extra motion and trims a
|
| 149 |
+
still start.
|
| 150 |
+
- The canonical motion is then retargeted onto the character with QtMeshEditor's incompatible-skeleton retargeter,
|
| 151 |
+
the same one used for Kimodo output. Humanoid rigs are mapped automatically from their bone names (for
|
| 152 |
+
example Mixamo-style rigs and the characters QtMeshEditor generates).
|
| 153 |
+
|
| 154 |
+
## Training
|
| 155 |
+
|
| 156 |
+
The training is best described as **synthetic-data-assisted**: part of the motion data was generated by a stronger
|
| 157 |
+
teacher model (Kimodo). It is not formal knowledge distillation. QtMesh T2M never sees Kimodo's weights, logits or
|
| 158 |
+
text features; it only sees the finished clips.
|
| 159 |
+
|
| 160 |
+
- **Data:** 30,488 training samples and 8,028 distinct text descriptions, including paraphrases. These figures were
|
| 161 |
+
checked against the training cache. The samples come from:
|
| 162 |
+
- **3,400 clips generated with NVIDIA Kimodo** from 3,470 prompts. The prompts were written with
|
| 163 |
+
`make-kimodo-t2m-prompts.py`, combining gaits, directions, styles, gestures, combat, sports, dance, reactions and
|
| 164 |
+
body-side variants.
|
| 165 |
+
- Every clip was **mirrored** left↔right, with "left" and "right" swapped in its text. That gives 6,800 samples.
|
| 166 |
+
- **23,688 windows** from QtMeshEditor's earlier v8 data, captioned per action: the curated template clip
|
| 167 |
+
library (`motion-library-v2.json`) and the **CC0 / CC-BY-4.0** [motion corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus).
|
| 168 |
+
- Every text gets paraphrases: the person word varies ("a person" / "someone" / "the character" / none), and
|
| 169 |
+
the action words are kept.
|
| 170 |
+
- **Quality control.**
|
| 171 |
+
- Kimodo clips pass automatic gates (`check-kimodo-t2m-corpus.py`): finite values, foot sliding during contact,
|
| 172 |
+
feet below the floor, implausible hip height, and joint-acceleration spikes. 3,400 of the 3,470 generated clips
|
| 173 |
+
passed.
|
| 174 |
+
- Samples that resolve fewer than 16 of the 22 joints are dropped.
|
| 175 |
+
- The v8 data had already been filtered with an anatomical ankle gate and a minimum-joints gate, and non-humanoid
|
| 176 |
+
characters were excluded.
|
| 177 |
+
- **Text features.**
|
| 178 |
+
- The Qwen3 embeddings were computed separately for this model with `embed-t2m-text.py`, through the same
|
| 179 |
+
llama.cpp build the editor uses. On the reference prompts, the app's embeddings match the training ones with
|
| 180 |
+
cosine 1.000000.
|
| 181 |
+
- **Kimodo's Llama 3 text embeddings were never stored or used as training inputs.**
|
| 182 |
+
- **Recipe** (`train-t2m-text.py`):
|
| 183 |
+
- 200 epochs of 20,000 samples each, batch 128.
|
| 184 |
+
- AdamW at 2e-4 (weight decay 1e-4) with a cosine schedule.
|
| 185 |
+
- 70% of each batch is drawn from the Kimodo clips, balanced per base prompt.
|
| 186 |
+
- 10% of the texts are dropped (zero vector) to train classifier-free guidance.
|
| 187 |
+
- Clips are padded to 150 frames with a length mask.
|
| 188 |
+
- The final flow-matching loss is 0.0478, and the ONNX export matches PyTorch to 2.4e-6.
|
| 189 |
+
- The export pins two canonical joints the data never supervises (the left and right buttock roles) to identity.
|
| 190 |
+
- **Scripts** (offline development tools; the editor never runs Python):
|
| 191 |
+
- [`make-kimodo-t2m-prompts.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/make-kimodo-t2m-prompts.py)
|
| 192 |
+
writes the prompt list.
|
| 193 |
+
- [`build-kimodo-t2m-corpus.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/build-kimodo-t2m-corpus.py)
|
| 194 |
+
generates the Kimodo clips.
|
| 195 |
+
- [`check-kimodo-t2m-corpus.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/check-kimodo-t2m-corpus.py)
|
| 196 |
+
applies the quality gates.
|
| 197 |
+
- [`prep-t2m-text.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/prep-t2m-text.py)
|
| 198 |
+
builds the training cache: mirroring, paraphrases, embeddings.
|
| 199 |
+
- [`embed-t2m-text.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/embed-t2m-text.py)
|
| 200 |
+
computes the Qwen3 embeddings.
|
| 201 |
+
- [`train-t2m-text.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/train-t2m-text.py)
|
| 202 |
+
trains the model and exports the ONNX graph (`--export-only` re-exports from a checkpoint).
|
| 203 |
+
|
| 204 |
+
The Kimodo-generated training clips and the training cache are not published.
|
| 205 |
+
|
| 206 |
+
## Usage
|
| 207 |
+
|
| 208 |
+
### Recommended: the QtMeshEditor GUI
|
| 209 |
+
|
| 210 |
+
1. **Install QtMeshEditor.**
|
| 211 |
+
|
| 212 |
+
| Platform | Command |
|
| 213 |
+
|---|---|
|
| 214 |
+
| Windows | `winget install FernandoTonon.QtMeshEditor` |
|
| 215 |
+
| macOS | `brew tap fernandotonon/qtmesheditor && brew trust fernandotonon/qtmesheditor && brew install qtmesheditor` |
|
| 216 |
+
| Linux | `sudo snap install qtmesheditor` |
|
| 217 |
+
|
| 218 |
+
Installers are also on the [GitHub releases page](https://github.com/fernandotonon/QtMeshEditor/releases).
|
| 219 |
+
QtMesh T2M needs version **3.60.0** or later.
|
| 220 |
+
2. **Import a rigged humanoid character** (FBX, glTF/GLB and other formats). You can also make one with the
|
| 221 |
+
built-in character generator.
|
| 222 |
+
3. **Switch to Animation mode** and find **Generate from text** in the Inspector's Animations section.
|
| 223 |
+
4. **Pick a model:** set **Model** to **QtMesh T2M (experimental, small)**.
|
| 224 |
+
5. **Download the model:** press **Download QtMesh T2M**. This happens once, about 740 MB. Nothing downloads
|
| 225 |
+
until you ask.
|
| 226 |
+
6. **Write a prompt**, for example *"a person waves with the right hand"*. Set the duration (up to 5 s) and the
|
| 227 |
+
number of variations. Optionally turn on **Root motion** or set a **Fixed seed**.
|
| 228 |
+
7. **Generate.** The clips appear as `generated_qtmesh_t2m`, `generated_qtmesh_t2m_v2`, … All variations are added
|
| 229 |
+
in one undo step. Preview them on the timeline.
|
| 230 |
+
8. **Export** the character with its animations (for example to `.glb` or `.fbx`).
|
| 231 |
+
|
| 232 |
+
### Advanced: the command line
|
| 233 |
+
|
| 234 |
+
```bash
|
| 235 |
+
qtmesh anim rigged.glb \
|
| 236 |
+
--generate "a person waves with the right hand" \
|
| 237 |
+
--backend qtmesh_t2m \
|
| 238 |
+
--duration 3 \
|
| 239 |
+
--download \
|
| 240 |
+
-o animated.glb
|
| 241 |
+
```
|
| 242 |
|
| 243 |
+
`--download` fetches the model files the first time. Other options:
|
| 244 |
|
| 245 |
+
| Option | What it does |
|
| 246 |
+
|---|---|
|
| 247 |
+
| `--variations N` | Generates N clips, 1–16. |
|
| 248 |
+
| `--seed N` | Makes the result reproducible. |
|
| 249 |
+
| `--no-root-motion` | Keeps the clip in place. |
|
| 250 |
+
| `--name NAME` | Names the clip. |
|
| 251 |
+
| `--json` | Prints a machine-readable report, including the timings. |
|
| 252 |
|
| 253 |
+
### MCP (AI agents)
|
|
|
|
|
|
|
|
|
|
| 254 |
|
| 255 |
+
QtMeshEditor's MCP server exposes the same thing through the `generate_motion` tool:
|
| 256 |
|
| 257 |
+
```json
|
| 258 |
+
{"backend": "qtmesh_t2m", "prompt": "a person jumps", "duration": 3, "variations": 2, "seed": 2, "download": true}
|
|
|
|
|
|
|
| 259 |
```
|
| 260 |
|
| 261 |
+
### Using the files directly
|
|
|
|
|
|
|
|
|
|
| 262 |
|
| 263 |
+
You can run the ONNX model in your own pipeline with ONNX Runtime and any llama.cpp build that supports Qwen3
|
| 264 |
+
embeddings. See [Files and tensor contract](#files-and-tensor-contract). You then need your own retargeting from
|
| 265 |
+
the 22-joint canonical skeleton, which is described in `t2m-text.json`.
|
|
|
|
|
|
|
| 266 |
|
| 267 |
+
> **About the GGUF file.** `Qwen3-Embedding-0.6B-Q8_0.gguf` is a **text-embedding model**. It is **not** a chat
|
| 268 |
+
> model and **not** the animation generator. Hugging Face may show "Use this model with llama.cpp / Ollama"
|
| 269 |
+
> buttons because of it. Those apply only to the embedding model, not to the text-to-motion pipeline, and chatting
|
| 270 |
+
> with it will not produce anything useful.
|
| 271 |
|
| 272 |
+
## Files and tensor contract
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 273 |
|
| 274 |
+
The current model lives in [`qtmesh_t2m/`](qtmesh_t2m):
|
| 275 |
|
| 276 |
+
| File | Size | What it is |
|
| 277 |
+
|---|---|---|
|
| 278 |
+
| `t2m-text.onnx` | 98 MB | The motion model (22.6 M parameters), with the 24-step Euler + CFG sampler unrolled in the graph. |
|
| 279 |
+
| `t2m-text.json` | small | The contract: shapes, fps, the embedding instruction and pooling, canonical rest directions. |
|
| 280 |
+
| `Qwen3-Embedding-0.6B-Q8_0.gguf` | ~640 MB | The text encoder. It is byte-identical to Qwen's official [Q8_0 GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF) (SHA-256 `06507c7b…e439`). |
|
| 281 |
+
| `LICENSE-Qwen3-Embedding.txt` | | The Apache-2.0 licence for the encoder. |
|
| 282 |
+
| `ATTRIBUTION.md` | | The required credits for the CC-BY-4.0 motion data the model was trained on. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 283 |
|
| 284 |
+
**`t2m-text.onnx` I/O** (checked against the graph):
|
| 285 |
|
| 286 |
+
| Tensor | Shape / type | Meaning |
|
| 287 |
+
|---|---|---|
|
| 288 |
+
| input `text` | `float32 [1, 1024]` | The L2-normalised Qwen3 embedding of `instruct + prompt`. A zero vector means unconditional. |
|
| 289 |
+
| input `seed` | `float32 [1, 19800]` | Starting noise, sampled from N(0, 0.5). 19800 = 150 frames × 132. |
|
| 290 |
+
| input `length` | `float32 [1]` | The number of frames wanted, up to 150. |
|
| 291 |
+
| output `motion` | `float32 [1, 150, 220]` | Per frame, 22 joints × 10 values `[tx, ty, tz, qx, qy, qz, qw, sx, sy, sz]`: world-frame rotation quaternions, with t = 0 and s = 1. Frames from `length` onward are padding. |
|
| 292 |
|
| 293 |
+
From `t2m-text.json`:
|
|
|
|
|
|
|
|
|
|
| 294 |
|
| 295 |
+
```text
|
| 296 |
+
version v9-text T 150 J 22 C 220 Z 19800 fps 30 frame world
|
| 297 |
+
flowSteps 24 guidance 2.0 embedDim 1024 pooling last
|
|
|
|
|
|
|
|
|
|
| 298 |
```
|
| 299 |
|
| 300 |
+
The joint order and the canonical rest pose (`restWorld`, `restDir`) are listed in the JSON.
|
| 301 |
+
|
| 302 |
+
## Limitations
|
| 303 |
+
|
| 304 |
+
- **Lower motion fidelity than Kimodo.** Motions are simpler, less expressive and less physically convincing.
|
| 305 |
+
- **Occasionally wrong or too subtle.** Some prompts give a weak or only partly correct motion.
|
| 306 |
+
- **Sensitive to the seed.** Quality varies a lot between seeds (see the jump example), so generate several
|
| 307 |
+
variations and pick the best one.
|
| 308 |
+
- **One action prompt per clip.** There are no multi-segment timelines; QtMeshEditor rejects a multi-line prompt
|
| 309 |
+
for this model. Kimodo supports timelines.
|
| 310 |
+
- **At most 5 seconds** per clip (150 frames at 30 fps).
|
| 311 |
+
- **No root height or root translation.** Jumps stay on the ground plane. Horizontal travel is only an estimate
|
| 312 |
+
made by the editor.
|
| 313 |
+
- **Weaker on uncommon movements.** Actions that are rare in the training data come out worse.
|
| 314 |
+
- **Body only.** There are no fingers, face or props; the model drives 22 body joints. In QtMeshEditor, Kimodo
|
| 315 |
+
drives the fingers.
|
| 316 |
+
- **English prompts.** The training text was English only.
|
| 317 |
+
- **Experimental** and under active development. Expect the model to change.
|
| 318 |
+
|
| 319 |
+
## License and attribution
|
| 320 |
+
|
| 321 |
+
This repository contains files under **different licences**, so its metadata says `license: other` (mixed, per
|
| 322 |
+
file) instead of claiming a single licence. The full per-file breakdown is in [`LICENSES.md`](LICENSES.md).
|
| 323 |
+
In short:
|
| 324 |
+
|
| 325 |
+
| Files | Licence |
|
| 326 |
+
|---|---|
|
| 327 |
+
| QtMesh T2M weights and contracts (`qtmesh_t2m/t2m-text.onnx`, `t2m-text.json`), the v8 model (`t2m.onnx`, `t2m-vocab.json`), the GIFs in `media/` | **CC0-1.0**, released by the QtMeshEditor project. The models were trained on CC-BY-4.0 data, so please keep the attribution files with them. |
|
| 328 |
+
| `qtmesh_t2m/Qwen3-Embedding-0.6B-Q8_0.gguf` | **Apache-2.0**, © Qwen team, Alibaba Cloud ([`LICENSE-Qwen3-Embedding.txt`](qtmesh_t2m/LICENSE-Qwen3-Embedding.txt)). |
|
| 329 |
+
| `motion-library.json`, `motion-library-v2.json` | **Mixed CC0-1.0 and CC-BY-4.0**, per clip. Credits are in [`ATTRIBUTION-v2.md`](ATTRIBUTION-v2.md) and [`ATTRIBUTION.md`](ATTRIBUTION.md). |
|
| 330 |
+
| `ATTRIBUTION.md`, `ATTRIBUTION-v2.md`, `qtmesh_t2m/ATTRIBUTION.md` | The credits for the CC-BY-4.0 works. **Ship them with the models or the data.** |
|
| 331 |
+
|
| 332 |
+
**About the Kimodo-generated training clips:**
|
| 333 |
+
|
| 334 |
+
- Kimodo's outputs are not "Derivative Models" under the
|
| 335 |
+
[NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
| 336 |
+
(§1.1), and NVIDIA claims no ownership of outputs (§2.4).
|
| 337 |
+
- The QtMeshEditor project therefore used Kimodo outputs as training data.
|
| 338 |
+
- No Kimodo weights, code, generated clips or Llama 3 embeddings are distributed in this repository. Kimodo itself
|
| 339 |
+
is a separate download inside QtMeshEditor, under its own licences.
|
| 340 |
+
|
| 341 |
+
QtMeshEditor itself is MIT-licensed.
|
| 342 |
+
|
| 343 |
+
## Previous versions
|
| 344 |
+
|
| 345 |
+
QtMesh T2M (v9, in `qtmesh_t2m/`) is the **current** model. The files at the repository root belong to earlier
|
| 346 |
+
releases and are kept for compatibility and reproducibility:
|
| 347 |
+
|
| 348 |
+
- `t2m.onnx` + `t2m-vocab.json` is the **v8.0 action-label model**: a 21.7 M-parameter flow-matching transformer
|
| 349 |
+
conditioned on one of 30 action words. It is QtMeshEditor's opt-in `--model` path.
|
| 350 |
+
- `motion-library-v2.json` (150 clips) is the curated template clip library. It is also the v8 training source.
|
| 351 |
+
- `motion-library.json` is the older v1 template library.
|
| 352 |
+
|
| 353 |
+
The full v8 model card (dataset, measurements, architecture, I/O contract, reproduction) is in
|
| 354 |
+
[`PREVIOUS_VERSIONS.md`](PREVIOUS_VERSIONS.md). Earlier checkpoints remain in this repository's
|
| 355 |
+
[commit history](https://huggingface.co/fernandotonon/QtMeshEditor-t2m/commits/main).
|
| 356 |
+
|
| 357 |
+
## Links
|
| 358 |
+
|
| 359 |
+
- **QtMeshEditor:** [GitHub](https://github.com/fernandotonon/QtMeshEditor).
|
| 360 |
+
- **Documentation:**
|
| 361 |
+
- [Kimodo and QtMesh T2M user guide](https://github.com/fernandotonon/QtMeshEditor/blob/master/docs/KIMODO.md).
|
| 362 |
+
- [Third-party model licences](https://github.com/fernandotonon/QtMeshEditor/blob/master/THIRD_PARTY_AI_MODELS.md).
|
| 363 |
+
- **Inference code:** [`QtMeshT2M.cpp`](https://github.com/fernandotonon/QtMeshEditor/blob/master/src/AIMotion/QtMeshT2M.cpp)
|
| 364 |
+
and [`MotionSceneApply.cpp`](https://github.com/fernandotonon/QtMeshEditor/blob/master/src/AIMotion/MotionSceneApply.cpp)
|
| 365 |
+
(retargeting and root motion).
|
| 366 |
+
- **Training and export scripts:** see [Training](#training).
|
| 367 |
+
- **Model files:** this repository, plus the
|
| 368 |
+
[QtMeshEditor-models](https://huggingface.co/fernandotonon/QtMeshEditor-models/tree/main/qtmesh_t2m) mirror that
|
| 369 |
+
the app downloads from (pinned commit and SHA-256 digests).
|
| 370 |
+
- **Training motion corpus:** [fernandotonon/QtMeshEditor-motion-corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus).
|
| 371 |
+
- **NVIDIA Kimodo:** [github.com/nv-tlabs/kimodo](https://github.com/nv-tlabs/kimodo).
|
| 372 |
+
- **Qwen3-Embedding-0.6B:** [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)
|
| 373 |
+
([GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF)).
|
media/compare_jump_seeds.gif
ADDED
|
Git LFS Details
|