fernandotonon commited on
Commit
1cbe103
·
verified ·
1 Parent(s): cd97e39

Model card: QtMesh T2M v9 rewrite — benchmark methodology, seed failure case, per-file licences, v8 docs moved to PREVIOUS_VERSIONS.md

Browse files
Files changed (5) hide show
  1. .gitattributes +1 -0
  2. LICENSES.md +50 -0
  3. PREVIOUS_VERSIONS.md +155 -0
  4. README.md +338 -186
  5. media/compare_jump_seeds.gif +3 -0
.gitattributes CHANGED
@@ -39,3 +39,4 @@ media/compare_jump.gif filter=lfs diff=lfs merge=lfs -text
39
  media/compare_run.gif filter=lfs diff=lfs merge=lfs -text
40
  media/compare_walk.gif filter=lfs diff=lfs merge=lfs -text
41
  media/compare_waveR.gif filter=lfs diff=lfs merge=lfs -text
 
 
39
  media/compare_run.gif filter=lfs diff=lfs merge=lfs -text
40
  media/compare_walk.gif filter=lfs diff=lfs merge=lfs -text
41
  media/compare_waveR.gif filter=lfs diff=lfs merge=lfs -text
42
+ media/compare_jump_seeds.gif filter=lfs diff=lfs merge=lfs -text
LICENSES.md ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Licences in this repository
2
+
3
+ `fernandotonon/QtMeshEditor-t2m` holds files under several licences. That is why the model card metadata says
4
+ `license: other` (`mixed-per-file`) instead of a single licence. This page lists every file.
5
+
6
+ | Path | Licence | Notes |
7
+ |---|---|---|
8
+ | `qtmesh_t2m/t2m-text.onnx` | CC0-1.0, with CC-BY-4.0 credits | QtMesh T2M v9 weights, trained from scratch by the QtMeshEditor project. See [Model weights](#model-weights). |
9
+ | `qtmesh_t2m/t2m-text.json` | CC0-1.0 | The model contract (shapes, embedding instruction, canonical rest pose). |
10
+ | `qtmesh_t2m/Qwen3-Embedding-0.6B-Q8_0.gguf` | Apache-2.0 | Qwen3-Embedding-0.6B by the Qwen team, Alibaba Cloud. Byte-identical to [Qwen/Qwen3-Embedding-0.6B-GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF). |
11
+ | `qtmesh_t2m/LICENSE-Qwen3-Embedding.txt` | — | The Apache-2.0 text for the file above. The upstream repository declares Apache-2.0 on its card but ships no LICENSE file, so this copy carries an attribution header. |
12
+ | `qtmesh_t2m/ATTRIBUTION.md` | — | The credits for the CC-BY-4.0 works behind `t2m-text.onnx`. **Keep it with the model.** |
13
+ | `t2m.onnx`, `t2m-vocab.json` | CC0-1.0, with CC-BY-4.0 credits | The previous v8.0 action-label model. Its credits are in `ATTRIBUTION-v2.md` and `ATTRIBUTION.md`. |
14
+ | `motion-library-v2.json` | Mixed: CC0-1.0 and CC-BY-4.0, per clip | The curated template clip library and the v8 training source. Credits are in `ATTRIBUTION-v2.md`; per-asset provenance is in [QtMeshEditor-motion-corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus). |
15
+ | `motion-library.json` | Mixed: CC0-1.0 and CC-BY-4.0, per clip | The older v1 template library. Credits are in `ATTRIBUTION.md`. |
16
+ | `ATTRIBUTION.md`, `ATTRIBUTION-v2.md` | — | The credits for the CC-BY-4.0 works. **Keep them with the data or anything derived from it.** |
17
+ | `media/*.gif` | CC0-1.0 | Renders made by the QtMeshEditor project of the two models' outputs on a QtMeshEditor-generated character. |
18
+ | `README.md`, `LICENSES.md`, `PREVIOUS_VERSIONS.md` | CC0-1.0 | Documentation. |
19
+
20
+ ## Model weights
21
+
22
+ The QtMeshEditor project dedicates its own work in the model files to the public domain under
23
+ [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/), as this repository has declared since it was
24
+ created. Part of the training data is licensed CC-BY-4.0, and the project treats the trained weights as
25
+ derived from that data. The credits in `ATTRIBUTION.md` / `ATTRIBUTION-v2.md` therefore travel with the weights.
26
+ If you redistribute the models, include those files.
27
+
28
+ ## Training data that is not in this repository
29
+
30
+ **Clips generated with NVIDIA Kimodo.**
31
+
32
+ - About 3,400 clips generated with `Kimodo-SOMA-RP-v1.2` were used as training data.
33
+ - They are not redistributed here.
34
+ - Kimodo's outputs are not "Derivative Models" under the
35
+ [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
36
+ (§1.1), and NVIDIA claims no ownership of outputs (§2.4).
37
+ - No Kimodo weights, code or Llama 3 text embeddings are in this repository or in the model.
38
+
39
+ **The CC0 / CC-BY-4.0 motion corpus.**
40
+
41
+ - It is published separately as
42
+ [fernandotonon/QtMeshEditor-motion-corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus),
43
+ with per-asset licences and credits.
44
+ - Non-commercial sources (AMASS, HumanML3D, KIT-ML, LAFAN1) and Mixamo were excluded.
45
+
46
+ ## QtMeshEditor
47
+
48
+ The editor that runs these models is MIT-licensed: [github.com/fernandotonon/QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor).
49
+ Its third-party model licences are documented in
50
+ [`THIRD_PARTY_AI_MODELS.md`](https://github.com/fernandotonon/QtMeshEditor/blob/master/THIRD_PARTY_AI_MODELS.md).
PREVIOUS_VERSIONS.md ADDED
@@ -0,0 +1,155 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Previous versions
2
+
3
+
4
+ This page keeps the documentation of the models that preceded **QtMesh T2M (v9)**, the current model described in
5
+ the [main model card](README.md). The v8 files are still in this repository: QtMeshEditor uses `t2m.onnx` for its
6
+ opt-in action-label path (`qtmesh anim <rig> --generate "<prompt>" --model`) and `motion-library-v2.json` as its
7
+ template clip library.
8
+
9
+ The text below is the v8.0 model card as published on 2026-08-29, unchanged except for the file list.
10
+
11
+ ---
12
+ ## v8.0 action-label model
13
+
14
+ Generative text→skeletal-animation model shipped with
15
+ [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor)
16
+ (`qtmesh anim <rig> --generate "<prompt>" --model`).
17
+
18
+ **v8.0** — flow-matching DiT, 21.7M params, 30 actions.
19
+
20
+ ## Files
21
+
22
+ - `t2m.onnx` + `t2m-vocab.json` — the generative model + its vocab/contract.
23
+ - Earlier checkpoints (v4 and later) — in the repository's
24
+ [commit history](https://huggingface.co/fernandotonon/QtMeshEditor-t2m/commits/main).
25
+ - `motion-library-v2.json` — the curated **template clip library** (150 clips).
26
+ It is both the fallback for prompts outside the model vocab *and* — since
27
+ v8.0 — the model's training data. This is the exact library v8.0 trained on.
28
+ - `ATTRIBUTION-v2.md` — **required credits** for the CC-BY clips in that
29
+ library. It must ship with the data and with anything derived from it.
30
+
31
+ ## Dataset
32
+
33
+ The published library is the full v8.0 training source: 150 clips drawn from 31
34
+ source assets — 27 CC-BY-4.0, plus CC0 material (largely Quaternius packs) and
35
+ 17 clips the author owns outright. Every CC-BY asset is credited in
36
+ `ATTRIBUTION-v2.md`; no clip in the library lacks a credit where one is due.
37
+
38
+ The raw source assets the library was built from — plus per-asset provenance
39
+ and the build scripts — live in the companion dataset repo
40
+ [**fernandotonon/QtMeshEditor-motion-corpus**](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus),
41
+ which also carries a copy of this library under `derived/`.
42
+
43
+ To rebuild the training cache from it, see **Reproducing** below. The cache
44
+ itself is not published — it is fully regenerable from this library, and it
45
+ stamps its own `buildArgv`/`buildFlags` so any `.npz` can state how it was
46
+ built.
47
+
48
+ ## What changed in v8.0
49
+
50
+ Earlier versions trained on the **CMU MoCap** corpus and inherited a facing
51
+ problem: generated walks moved convincingly but oriented backwards. Measuring
52
+ against a real reference walk showed why — CMU "walk" windows sit 1.73–2.07 rad
53
+ away from it, *further than a real punch, dance or jump*, and the real-vs-real
54
+ floor is 0.437. The corpus never carried the correct convention, so the model
55
+ could not learn it. v6.x models scored 1.89 while their own training data
56
+ scored 1.73: they were faithfully reproducing bad data.
57
+
58
+ v8.0 trains on the **curated template clips** instead (`--library-repeat 40`),
59
+ which measure 0.303–0.462 against the real walk. Plus two data gates and one
60
+ new loss:
61
+
62
+ - **Anatomical ankle gate** — rejects clips whose mean knee→foot angle exceeds
63
+ 70°. Caught `march` (106°), `throw` (101°) and `hey` (90°): the foot folded
64
+ perpendicular to the shin in *100%* of their windows.
65
+ - **Minimum resolved roles (16/22)** — `hey` resolved only 9, with all four
66
+ knee/ankle roles invalid, so the model had no leg data and invented legs.
67
+ - **Leg-chain loss** — keeps a walk's bend in the **knee** rather than the
68
+ ankle. The v7.x walk bent the knee 25° and the ankle 31° (real: 38°/10°),
69
+ which renders as a near-straight knee with a false second joint below it.
70
+
71
+ ### Measured against a real reference walk
72
+
73
+ | | v7.x | **v8.0** | real |
74
+ |---|---|---|---|
75
+ | knee/ankle split | 25°/31° | **41°/8°** | 38°/10° |
76
+ | leg-chain penalty | 0.359 | **0.0003** | 0.014 |
77
+ | walk refDist | 0.287 | **0.243** | 0.437 (real-vs-real floor) |
78
+ | run refDist | 0.472 | **0.324** | — |
79
+
80
+ Scored against the training distribution across 20 actions, v8.0 is closer to
81
+ ground truth on 8, further on 3, and tied on 9. Largest gains: `wave`
82
+ 0.710→0.082 (the arm now actually goes overhead), `idle`, `climb`, `salute`,
83
+ `punch`, `sit`.
84
+
85
+ ## Vocabulary (30 actions)
86
+
87
+ ```
88
+ attack boxing buildloop cartgiveloop climb cough crouch cut dance death
89
+ fall farmloop hit idle jump kick pickup punch run salute shake shoot
90
+ sit stonegiveloop strafeleft straferight walk wave woodgiveloop working
91
+ ```
92
+
93
+ `march`, `throw`, `hey` and `confession` were **removed** — their source clips
94
+ are anatomically broken (see the ankle gate above). Those prompts fall back to
95
+ the curated template library, which for `march`/`throw` is strictly better than
96
+ what the model produced. `march`/`marching` resolves to `walk`.
97
+
98
+ Quality is uneven across actions: `walk`, `run`, `wave`, `cough`, `death`,
99
+ `pickup` and `attack` are the strongest; `dance` and `idle` are weak (`dance`
100
+ averages five genuinely different dances under one label; `idle` is nearly
101
+ static). This is a **template-quality** ceiling, not an architecture one ��� more
102
+ curated clips is the improvement path.
103
+
104
+ ## Training data — permissive only
105
+
106
+ The curated library is CC0/CC-BY content (Sketchfab, OpenGameArt, Quaternius
107
+ packs) plus clips the author owns; credits are in `ATTRIBUTION-v2.md`, **which
108
+ must ship with the library and with anything derived from it**. Earlier CMU
109
+ MoCap windows (commercial-OK) remain in the corpus path but no longer dominate
110
+ training. AMASS / HumanML3D / KIT-ML are **excluded** (non-commercial).
111
+ Zombie/fruit/non-humanoid characters are filtered out, as are clips failing the
112
+ anatomical ankle gate or resolving fewer than 16 of the 22 canonical roles.
113
+
114
+ ## Architecture (v8.0)
115
+
116
+ - **6D-rotation** representation (Zhou et al. 2019), column-packed.
117
+ - **Flow-matching DiT**, dim 384 × 8 layers, ~21.7M params — the model predicts
118
+ a velocity field; the host runs a fixed Euler schedule baked into the graph.
119
+ - Auxiliary losses: contralateral gait phase, spine-twist penalty, travel
120
+ direction, limb amplitude band, jitter band, gait periodicity, and the v8.0
121
+ leg-chain term — each gated to the action classes where it is meaningful.
122
+ - Exports to ONNX as a single forward pass.
123
+
124
+ ## I/O contract
125
+
126
+ ```
127
+ input "tokens" float32 [1, V] one-hot over the fixed action vocab (see t2m-vocab.json)
128
+ input "seed" float32 [1, Z] latent noise (host samples ~N(0,0.5) and does best-of-N)
129
+ output "motion" float32 [1, T, C] C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]
130
+ ```
131
+
132
+ `t2m-vocab.json` ships the `{vocab, Z, T, C, J, fps, frame, restDir}` the host
133
+ needs. `frame: "world"` marks the WORLD-frame convention (the retarget takes a
134
+ world delta); `fps: 30`, `T: 60` (2.0 s).
135
+
136
+ ## Reproducing
137
+
138
+ `scripts/prep-t2m-v6.py` + `scripts/train-t2m-flow-v5.py` in the QtMeshEditor
139
+ repo (one-time, offline dev tools — the app never runs Python):
140
+
141
+ `--corpus` is optional and points at the raw CMU-derived corpus, which is NOT
142
+ published here; with `--library-repeat 40` the curated library dominates
143
+ training either way, so omitting it reproduces the same shape of dataset from
144
+ the published files alone:
145
+
146
+ ```
147
+ python3 scripts/prep-t2m-v6.py --library motion-library-v2.json \
148
+ --library-repeat 40 --min-roles 16 --out t2m.npz
149
+ python3 scripts/train-t2m-flow-v5.py --data t2m.npz --out flow --dim 384 --layers 8 \
150
+ --phase-weight 0.08 --twist-weight 0.5 --travel-weight 0.2 --amp-weight 0.4 \
151
+ --jitter-weight 0.4 --period-weight 0.5 --legchain-weight 0.35 --loco-boost 6.0
152
+ ```
153
+
154
+ The cache stamps its own `buildArgv`/`buildFlags`, so any `.npz` can report how
155
+ it was built.
README.md CHANGED
@@ -1,221 +1,373 @@
1
  ---
2
- license: cc0-1.0
 
 
 
 
 
 
 
3
  tags:
4
  - text-to-motion
 
5
  - animation
 
 
 
 
 
6
  - onnx
7
  - gguf
 
8
  - qtmesheditor
9
  ---
10
 
11
- # QtMeshEditor — text-to-motion (t2m)
12
-
13
- Text→skeletal-animation models shipped with [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor).
14
-
15
- ## QtMesh T2M (v9, text-conditioned) — experimental
16
-
17
- QtMeshEditor's own text-to-motion model, offered next to NVIDIA Kimodo in the editor (Animation panel →
18
- **Model: QtMesh T2M**, CLI `--backend qtmesh_t2m`, MCP `generate_motion {backend: "qtmesh_t2m"}`). Kimodo stays the
19
- default; QtMesh T2M is **much smaller and faster, intended for lower-end machines**.
20
-
21
- It conditions on a sentence embedding of the prompt (Qwen3-Embedding-0.6B), so wording, sides ("with the right
22
- hand"), styles and speeds reach the model — unlike the v8 action-label model below.
23
-
24
- ### Kimodo vs QtMesh T2M
25
-
26
- Same generated character, same prompt, 3 s, in place, one sample each (seed in the caption).
27
-
28
- ![walk](media/compare_walk.gif)
29
- ![run](media/compare_run.gif)
30
- ![wave with the right hand](media/compare_waveR.gif)
31
- ![jump](media/compare_jump.gif)
32
-
33
- Jumps vary a lot with the seed: of ten seeds, four bend the knees 70–86° (the one shown), four barely move.
34
-
35
- ### Memory and speed
36
-
37
- One 3 s clip on an Apple M-series Mac; the CLI run includes importing the rig and exporting the result.
38
-
39
- | | NVIDIA Kimodo (native) | QtMesh T2M |
 
 
 
 
 
 
 
40
  |---|---|---|
41
- | Download | ~9.2 GB | ~740 MB |
42
- | Text encoding | ~1.1 s + loading an 8 GB encoder | ~0.2–0.4 s |
43
- | Motion sampling | ~14 s | ~2.7 s |
44
- | Whole CLI run | 18–21 s | ~5 s |
45
- | Peak resident memory | 7.6–8.2 GB | ~0.87 GB |
46
-
47
- ### Files (`qtmesh_t2m/`)
48
-
49
- - `t2m-text.onnx` (98 MB) — 22.6 M-parameter flow-matching transformer with AdaLN-Zero conditioning; the 24-step
50
- Euler sampler with classifier-free guidance (2.0) is unrolled in the graph. Inputs `text[1,1024]` (L2-normalised
51
- embedding), `seed[1,19800]` ~ N(0, 0.5), `length[1]` frames (≤ 150, 30 fps); output `motion[1,150,220]`: per
52
- canonical role (22 joints) `t, quat x y z w, s`, world frame, rotation only.
53
- - `t2m-text.json` — contract: shapes, fps, the embedding instruction, canonical rest directions.
54
- - `Qwen3-Embedding-0.6B-Q8_0.gguf` — the text encoder (Apache-2.0, `LICENSE-Qwen3-Embedding.txt`), run through
55
- llama.cpp: `"Instruct: Given a description of a human body motion, retrieve motions that match it\nQuery:" +
56
- prompt`, last-token pooling, L2-normalised.
57
- - `ATTRIBUTION.md` — required credits for the CC-BY-4.0 motion the model was trained on.
58
-
59
- ### Training
60
-
61
- - **Data:** 30,488 samples — 3,400 clips we generated with NVIDIA Kimodo (each also mirrored left↔right, with
62
- "left"/"right" swapped in the text), plus the v8 windows (curated template library + CC0/CC-BY corpus) captioned
63
- per action; 8,028 distinct texts including paraphrases. Kimodo outputs are not Derivative Models under the NVIDIA
64
- Open Model License; Kimodo's Llama 3 text embeddings were never stored or trained on.
65
- - **Recipe:** `scripts/prep-t2m-text.py` + `scripts/train-t2m-text.py` in the QtMeshEditor repo; 200 epochs, batch
66
- 128, AdamW 2e-4 cosine, 10 % text dropout for CFG, padded to 150 frames with a length mask. The export pins the
67
- never-supervised canonical buttock roles (14/18) to identity.
68
- - **Inference details in the editor:** long clips start from rest in the Kimodo data, so the editor generates up to
69
- 1 s extra and trims the still start; root travel is estimated from the planted foot; the clip is retargeted with
70
- the same incompatible-skeleton retargeter as Kimodo.
71
-
72
- ### Limitations
73
-
74
- One prompt per clip (no multi-segment timelines), up to 5 s, no root height (jumps stay on the spot), weaker on
75
- rare actions; quality varies with the seed — generate a few variations.
76
-
77
- ---
78
-
79
- ## v8.0 action-label model (previous)
80
-
81
- Generative text→skeletal-animation model shipped with
82
- [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor)
83
- (`qtmesh anim <rig> --generate "<prompt>" --model`).
84
-
85
- **v8.0** — flow-matching DiT, 21.7M params, 30 actions.
86
-
87
- ## Files
88
-
89
- - `t2m.onnx` + `t2m-vocab.json` — the generative model + its vocab/contract.
90
- - `t2m-v61.onnx` / `t2m-v4.onnx` — previous versions, kept for rollback.
91
- - `motion-library-v2.json` — the curated **template clip library** (150 clips).
92
- It is both the fallback for prompts outside the model vocab *and* — since
93
- v8.0 — the model's training data. This is the exact library v8.0 trained on.
94
- - `ATTRIBUTION-v2.md` — **required credits** for the CC-BY clips in that
95
- library. It must ship with the data and with anything derived from it.
96
-
97
- ## Dataset
98
 
99
- The published library is the full v8.0 training source: 150 clips drawn from 31
100
- source assets — 27 CC-BY-4.0, plus CC0 material (largely Quaternius packs) and
101
- 17 clips the author owns outright. Every CC-BY asset is credited in
102
- `ATTRIBUTION-v2.md`; no clip in the library lacks a credit where one is due.
103
 
104
- The raw source assets the library was built from — plus per-asset provenance
105
- and the build scripts — live in the companion dataset repo
106
- [**fernandotonon/QtMeshEditor-motion-corpus**](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus),
107
- which also carries a copy of this library under `derived/`.
108
 
109
- To rebuild the training cache from it, see **Reproducing** below. The cache
110
- itself is not published — it is fully regenerable from this library, and it
111
- stamps its own `buildArgv`/`buildFlags` so any `.npz` can state how it was
112
- built.
113
 
114
- ## What changed in v8.0
 
 
115
 
116
- Earlier versions trained on the **CMU MoCap** corpus and inherited a facing
117
- problem: generated walks moved convincingly but oriented backwards. Measuring
118
- against a real reference walk showed why — CMU "walk" windows sit 1.73–2.07 rad
119
- away from it, *further than a real punch, dance or jump*, and the real-vs-real
120
- floor is 0.437. The corpus never carried the correct convention, so the model
121
- could not learn it. v6.x models scored 1.89 while their own training data
122
- scored 1.73: they were faithfully reproducing bad data.
123
 
124
- v8.0 trains on the **curated template clips** instead (`--library-repeat 40`),
125
- which measure 0.303–0.462 against the real walk. Plus two data gates and one
126
- new loss:
 
 
 
 
 
127
 
128
- - **Anatomical ankle gate** — rejects clips whose mean knee→foot angle exceeds
129
- 70°. Caught `march` (106°), `throw` (101°) and `hey` (90°): the foot folded
130
- perpendicular to the shin in *100%* of their windows.
131
- - **Minimum resolved roles (16/22)** — `hey` resolved only 9, with all four
132
- knee/ankle roles invalid, so the model had no leg data and invented legs.
133
- - **Leg-chain loss** — keeps a walk's bend in the **knee** rather than the
134
- ankle. The v7.x walk bent the knee 25° and the ankle 31° (real: 38°/10°),
135
- which renders as a near-straight knee with a false second joint below it.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
 
137
- ### Measured against a real reference walk
138
 
139
- | | v7.x | **v8.0** | real |
140
- |---|---|---|---|
141
- | knee/ankle split | 25°/31° | **41°/8°** | 38°/10° |
142
- | leg-chain penalty | 0.359 | **0.0003** | 0.014 |
143
- | walk refDist | 0.287 | **0.243** | 0.437 (real-vs-real floor) |
144
- | run refDist | 0.472 | **0.324** | — |
 
145
 
146
- Scored against the training distribution across 20 actions, v8.0 is closer to
147
- ground truth on 8, further on 3, and tied on 9. Largest gains: `wave`
148
- 0.710→0.082 (the arm now actually goes overhead), `idle`, `climb`, `salute`,
149
- `punch`, `sit`.
150
 
151
- ## Vocabulary (30 actions)
152
 
153
- ```
154
- attack boxing buildloop cartgiveloop climb cough crouch cut dance death
155
- fall farmloop hit idle jump kick pickup punch run salute shake shoot
156
- sit stonegiveloop strafeleft straferight walk wave woodgiveloop working
157
  ```
158
 
159
- `march`, `throw`, `hey` and `confession` were **removed** — their source clips
160
- are anatomically broken (see the ankle gate above). Those prompts fall back to
161
- the curated template library, which for `march`/`throw` is strictly better than
162
- what the model produced. `march`/`marching` resolves to `walk`.
163
 
164
- Quality is uneven across actions: `walk`, `run`, `wave`, `cough`, `death`,
165
- `pickup` and `attack` are the strongest; `dance` and `idle` are weak (`dance`
166
- averages five genuinely different dances under one label; `idle` is nearly
167
- static). This is a **template-quality** ceiling, not an architecture one — more
168
- curated clips is the improvement path.
169
 
170
- ## Training data — permissive only
 
 
 
171
 
172
- The curated library is CC0/CC-BY content (Sketchfab, OpenGameArt, Quaternius
173
- packs) plus clips the author owns; credits are in `ATTRIBUTION-v2.md`, **which
174
- must ship with the library and with anything derived from it**. Earlier CMU
175
- MoCap windows (commercial-OK) remain in the corpus path but no longer dominate
176
- training. AMASS / HumanML3D / KIT-ML are **excluded** (non-commercial).
177
- Zombie/fruit/non-humanoid characters are filtered out, as are clips failing the
178
- anatomical ankle gate or resolving fewer than 16 of the 22 canonical roles.
179
 
180
- ## Architecture (v8.0)
181
 
182
- - **6D-rotation** representation (Zhou et al. 2019), column-packed.
183
- - **Flow-matching DiT**, dim 384 × 8 layers, ~21.7M params — the model predicts
184
- a velocity field; the host runs a fixed Euler schedule baked into the graph.
185
- - Auxiliary losses: contralateral gait phase, spine-twist penalty, travel
186
- direction, limb amplitude band, jitter band, gait periodicity, and the v8.0
187
- leg-chain term — each gated to the action classes where it is meaningful.
188
- - Exports to ONNX as a single forward pass.
189
-
190
- ## I/O contract
191
-
192
- ```
193
- input "tokens" float32 [1, V] one-hot over the fixed action vocab (see t2m-vocab.json)
194
- input "seed" float32 [1, Z] latent noise (host samples ~N(0,0.5) and does best-of-N)
195
- output "motion" float32 [1, T, C] C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]
196
- ```
197
-
198
- `t2m-vocab.json` ships the `{vocab, Z, T, C, J, fps, frame, restDir}` the host
199
- needs. `frame: "world"` marks the WORLD-frame convention (the retarget takes a
200
- world delta); `fps: 30`, `T: 60` (2.0 s).
201
 
202
- ## Reproducing
203
 
204
- `scripts/prep-t2m-v6.py` + `scripts/train-t2m-flow-v5.py` in the QtMeshEditor
205
- repo (one-time, offline dev tools — the app never runs Python):
 
 
 
 
206
 
207
- `--corpus` is optional and points at the raw CMU-derived corpus, which is NOT
208
- published here; with `--library-repeat 40` the curated library dominates
209
- training either way, so omitting it reproduces the same shape of dataset from
210
- the published files alone:
211
 
212
- ```
213
- python3 scripts/prep-t2m-v6.py --library motion-library-v2.json \
214
- --library-repeat 40 --min-roles 16 --out t2m.npz
215
- python3 scripts/train-t2m-flow-v5.py --data t2m.npz --out flow --dim 384 --layers 8 \
216
- --phase-weight 0.08 --twist-weight 0.5 --travel-weight 0.2 --amp-weight 0.4 \
217
- --jitter-weight 0.4 --period-weight 0.5 --legchain-weight 0.35 --loco-boost 6.0
218
  ```
219
 
220
- The cache stamps its own `buildArgv`/`buildFlags`, so any `.npz` can report how
221
- it was built.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: other
3
+ license_name: mixed-per-file
4
+ license_link: https://huggingface.co/fernandotonon/QtMeshEditor-t2m/blob/main/LICENSES.md
5
+ language:
6
+ - en
7
+ library_name: onnx
8
+ datasets:
9
+ - fernandotonon/QtMeshEditor-motion-corpus
10
  tags:
11
  - text-to-motion
12
+ - motion-generation
13
  - animation
14
+ - 3d-animation
15
+ - character-animation
16
+ - skeletal-animation
17
+ - game-development
18
+ - flow-matching
19
  - onnx
20
  - gguf
21
+ - cpu-inference
22
  - qtmesheditor
23
  ---
24
 
25
+ # QtMesh T2M — Lightweight Text-to-3D-Motion Generation
26
+
27
+ QtMesh T2M turns a sentence such as *"a person waves with the right hand"* into a skeletal animation for a
28
+ rigged 3D character. The [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor) project trained it from
29
+ scratch. It is **not** a quantized or distilled copy of NVIDIA Kimodo.
30
+
31
+ - **Two models, loaded separately.** The motion generator is a custom **22.6 M-parameter flow-matching
32
+ transformer**, run with ONNX Runtime. The prompt is read by **Qwen3-Embedding-0.6B** (0.6 B parameters, Q8_0
33
+ GGUF), run with llama.cpp.
34
+ - **No Python at inference.** Inside QtMeshEditor everything runs natively in C++.
35
+ - **Small enough for modest hardware.** It is about a 740 MB download and peaks at under 1 GB of memory.
36
+ It runs on the CPU: ONNX Runtime's CPU provider, and llama.cpp on Apple Silicon.
37
+ - **Integrated with the editor.** QtMeshEditor maps the output onto your character's rig, can add root travel,
38
+ and exports the result as glTF/GLB or FBX.
39
+ - **Experimental.** Its motion quality is clearly below [NVIDIA Kimodo](https://github.com/nv-tlabs/kimodo), which
40
+ stays QtMeshEditor's default. QtMesh T2M is the option for machines that cannot run Kimodo comfortably.
41
+
42
+ ## Contents
43
+
44
+ - [Benchmark: QtMesh T2M vs NVIDIA Kimodo](#benchmark-qtmesh-t2m-vs-nvidia-kimodo)
45
+ - [Visual results](#visual-results--qtmesh-t2m-vs-nvidia-kimodo)
46
+ - [How it works](#how-it-works)
47
+ - [Training](#training)
48
+ - [Usage](#usage)
49
+ - [Files and tensor contract](#files-and-tensor-contract)
50
+ - [Limitations](#limitations)
51
+ - [License and attribution](#license-and-attribution)
52
+ - [Previous versions](#previous-versions)
53
+ - [Links](#links)
54
+
55
+ ## Benchmark: QtMesh T2M vs NVIDIA Kimodo
56
+
57
+ These are the QtMeshEditor project's own measurements, not an independent benchmark. Each run generated one
58
+ 3-second clip for the same character.
59
+
60
+ | | **QtMesh T2M** | NVIDIA Kimodo (native runtime) |
61
  |---|---|---|
62
+ | Download | **~740 MB** | ~9.2 GB |
63
+ | Text encoder | Qwen3-Embedding-0.6B (Q8_0) | Llama-3-LLM2Vec-8B (Q8_0) |
64
+ | Text encoding | **~0.2–0.4 s** | ~1.1 s, plus loading the 8 B encoder |
65
+ | Motion sampling | **~2.7 s** (24 flow steps) | ~14 s (100 diffusion steps) |
66
+ | Whole CLI run (import, generate, export) | **~5 s** | ~18–21 s |
67
+ | Peak resident memory | **~0.87 GB** | ~7.6–8.2 GB |
68
+
69
+ **How it was measured**
70
+
71
+ - **Hardware:** Apple M5, 24 GB unified memory, macOS.
72
+ - **Runtime:** QtMeshEditor 3.60 with ONNX Runtime 1.31.1 on the CPU execution provider. Both text encoders run as
73
+ Q8_0 GGUFs through the bundled llama.cpp. Kimodo used QtMeshEditor's native (Python-free) port of
74
+ `Kimodo-SOMA-RP-v1.2`, with its post-processing on.
75
+ - **Method:**
76
+ - The command was `qtmesh anim char.fbx --generate "<prompt>" --backend <kimodo|qtmesh_t2m> --duration 3 --seed 1 --json -o out.fbx`.
77
+ - Each backend ran once per prompt, for four prompts: walk, run, wave with the right hand, jump.
78
+ - Wall time and maximum resident set size come from `/usr/bin/time -l`.
79
+ - The text-encoding and sampling times are the ones the CLI reports in `--json`.
80
+ - The "whole CLI run" covers process start, importing the character, generating, retargeting and exporting the
81
+ FBX.
82
+ - The models were already downloaded.
83
+ - **Not controlled or not recorded:**
84
+ - File-cache warmth.
85
+ - Background load.
86
+ - Repeated trials. Each figure is one run per prompt; the ranges span the four prompts.
87
+ - **Root motion differed:** Kimodo ran with `--no-root-motion`, while QtMesh T2M used its default foot-driven root
88
+ motion.
89
+
90
+ Smaller and faster does **not** mean equivalent. Kimodo's motions are clearly more detailed and reliable; compare
91
+ the clips below.
92
+
93
+ ## Visual results — QtMesh T2M vs NVIDIA Kimodo
94
+
95
+ The same generated character and prompt are used on both sides. Each clip is 3 seconds, rendered in place
96
+ (root motion off), with one sample per model. **Left: NVIDIA Kimodo. Right: QtMesh T2M.** The caption in each GIF
97
+ gives the seed.
98
+
99
+ | Prompt | Seeds (Kimodo / QtMesh T2M) | Comparison | What to look for |
100
+ |---|---|---|---|
101
+ | "a person walks forward" | 1 / 1 | ![walk](media/compare_walk.gif) | Both read as a walk cycle. QtMesh T2M's arms swing less. |
102
+ | "a person runs forward" | 1 / 1 | ![run](media/compare_run.gif) | Both read as a run. Kimodo keeps the elbows bent in a running position; QtMesh T2M's arms hang straighter. |
103
+ | "a person waves with the right hand" | 1 / 1 | ![wave with the right hand](media/compare_waveR.gif) | Both raise the character's **right** hand, the side named in the prompt; the text embedding carries left/right. QtMesh T2M waves with a large overhead arm, Kimodo with a smaller wave near the head. |
104
+ | "a person jumps" | 1 / **2** | ![jump](media/compare_jump.gif) | **Cherry-picked:** the QtMesh T2M clip uses seed 2, not seed 1. See the failure case below. |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
 
106
+ ### Failure case: seed sensitivity
 
 
 
107
 
108
+ The two clips below come from the same model and prompt with different seeds. The left one is what you get with
109
+ seed 1, the seed used for the other comparisons.
 
 
110
 
111
+ ![QtMesh T2M jump, seed 1 vs seed 2](media/compare_jump_seeds.gif)
 
 
 
112
 
113
+ Across ten seeds of "a person jumps", four bent the knees 70–86° into a clear jump, and four barely moved. The
114
+ showcased clips are therefore not a statistically representative sample. Treat them as examples of what the
115
+ model *can* produce. In practice, generate a few variations and keep the best one.
116
 
117
+ ## How it works
 
 
 
 
 
 
118
 
119
+ ```
120
+ Text prompt
121
+ → Qwen3-Embedding-0.6B (llama.cpp) 1024-d, L2-normalised sentence embedding
122
+ → Flow-matching transformer (ONNX) 22.6 M params, 24 Euler steps, classifier-free guidance
123
+ → Canonical 22-joint motion joint rotations, 30 fps, up to 150 frames (5 s)
124
+ → QtMeshEditor retargeting mapped onto your character's skeleton
125
+ → glTF/GLB or FBX animation
126
+ ```
127
 
128
+ - **Text conditioning.**
129
+ - The prompt is prefixed with a fixed instruction:
130
+ `Instruct: Given a description of a human body motion, retrieve motions that match it\nQuery:`.
131
+ - Qwen3-Embedding then encodes it with last-token pooling, and the vector is L2-normalised.
132
+ - Because the model sees a sentence embedding rather than an action label, wording, body side ("with the right
133
+ hand"), speed and style all reach it.
134
+ - **Motion model.**
135
+ - A transformer of width 384 and 8 layers, conditioned on the text and the flow time through **AdaLN-Zero**.
136
+ - It is trained with **rectified flow (flow matching)**: it learns a velocity field that carries random noise to
137
+ a motion.
138
+ - The ONNX graph contains the whole sampler, unrolled: **24 Euler steps with classifier-free guidance
139
+ (scale 2.0)**.
140
+ - The model generates **rotations for 22 canonical joints** (pelvis, spine, neck, head, arms and legs), in the
141
+ world frame, at 30 fps.
142
+ - Clips are up to 150 frames (5 s). The length is an input, and frames past it are padding.
143
+ - **What the model does not generate.**
144
+ - It produces no root translation and no vertical root displacement. A jump therefore stays on the spot.
145
+ - When **Root motion** is on, QtMeshEditor *estimates* horizontal travel from the planted foot. Measured: walk
146
+ ~1.4 m/s, run ~1.9 m/s. The character never rises.
147
+ - **Editor post-processing.**
148
+ - Long clips in the training data start from rest, so the editor generates up to 1 s of extra motion and trims a
149
+ still start.
150
+ - The canonical motion is then retargeted onto the character with QtMeshEditor's incompatible-skeleton retargeter,
151
+ the same one used for Kimodo output. Humanoid rigs are mapped automatically from their bone names (for
152
+ example Mixamo-style rigs and the characters QtMeshEditor generates).
153
+
154
+ ## Training
155
+
156
+ The training is best described as **synthetic-data-assisted**: part of the motion data was generated by a stronger
157
+ teacher model (Kimodo). It is not formal knowledge distillation. QtMesh T2M never sees Kimodo's weights, logits or
158
+ text features; it only sees the finished clips.
159
+
160
+ - **Data:** 30,488 training samples and 8,028 distinct text descriptions, including paraphrases. These figures were
161
+ checked against the training cache. The samples come from:
162
+ - **3,400 clips generated with NVIDIA Kimodo** from 3,470 prompts. The prompts were written with
163
+ `make-kimodo-t2m-prompts.py`, combining gaits, directions, styles, gestures, combat, sports, dance, reactions and
164
+ body-side variants.
165
+ - Every clip was **mirrored** left↔right, with "left" and "right" swapped in its text. That gives 6,800 samples.
166
+ - **23,688 windows** from QtMeshEditor's earlier v8 data, captioned per action: the curated template clip
167
+ library (`motion-library-v2.json`) and the **CC0 / CC-BY-4.0** [motion corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus).
168
+ - Every text gets paraphrases: the person word varies ("a person" / "someone" / "the character" / none), and
169
+ the action words are kept.
170
+ - **Quality control.**
171
+ - Kimodo clips pass automatic gates (`check-kimodo-t2m-corpus.py`): finite values, foot sliding during contact,
172
+ feet below the floor, implausible hip height, and joint-acceleration spikes. 3,400 of the 3,470 generated clips
173
+ passed.
174
+ - Samples that resolve fewer than 16 of the 22 joints are dropped.
175
+ - The v8 data had already been filtered with an anatomical ankle gate and a minimum-joints gate, and non-humanoid
176
+ characters were excluded.
177
+ - **Text features.**
178
+ - The Qwen3 embeddings were computed separately for this model with `embed-t2m-text.py`, through the same
179
+ llama.cpp build the editor uses. On the reference prompts, the app's embeddings match the training ones with
180
+ cosine 1.000000.
181
+ - **Kimodo's Llama 3 text embeddings were never stored or used as training inputs.**
182
+ - **Recipe** (`train-t2m-text.py`):
183
+ - 200 epochs of 20,000 samples each, batch 128.
184
+ - AdamW at 2e-4 (weight decay 1e-4) with a cosine schedule.
185
+ - 70% of each batch is drawn from the Kimodo clips, balanced per base prompt.
186
+ - 10% of the texts are dropped (zero vector) to train classifier-free guidance.
187
+ - Clips are padded to 150 frames with a length mask.
188
+ - The final flow-matching loss is 0.0478, and the ONNX export matches PyTorch to 2.4e-6.
189
+ - The export pins two canonical joints the data never supervises (the left and right buttock roles) to identity.
190
+ - **Scripts** (offline development tools; the editor never runs Python):
191
+ - [`make-kimodo-t2m-prompts.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/make-kimodo-t2m-prompts.py)
192
+ writes the prompt list.
193
+ - [`build-kimodo-t2m-corpus.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/build-kimodo-t2m-corpus.py)
194
+ generates the Kimodo clips.
195
+ - [`check-kimodo-t2m-corpus.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/check-kimodo-t2m-corpus.py)
196
+ applies the quality gates.
197
+ - [`prep-t2m-text.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/prep-t2m-text.py)
198
+ builds the training cache: mirroring, paraphrases, embeddings.
199
+ - [`embed-t2m-text.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/embed-t2m-text.py)
200
+ computes the Qwen3 embeddings.
201
+ - [`train-t2m-text.py`](https://github.com/fernandotonon/QtMeshEditor/blob/master/scripts/train-t2m-text.py)
202
+ trains the model and exports the ONNX graph (`--export-only` re-exports from a checkpoint).
203
+
204
+ The Kimodo-generated training clips and the training cache are not published.
205
+
206
+ ## Usage
207
+
208
+ ### Recommended: the QtMeshEditor GUI
209
+
210
+ 1. **Install QtMeshEditor.**
211
+
212
+ | Platform | Command |
213
+ |---|---|
214
+ | Windows | `winget install FernandoTonon.QtMeshEditor` |
215
+ | macOS | `brew tap fernandotonon/qtmesheditor && brew trust fernandotonon/qtmesheditor && brew install qtmesheditor` |
216
+ | Linux | `sudo snap install qtmesheditor` |
217
+
218
+ Installers are also on the [GitHub releases page](https://github.com/fernandotonon/QtMeshEditor/releases).
219
+ QtMesh T2M needs version **3.60.0** or later.
220
+ 2. **Import a rigged humanoid character** (FBX, glTF/GLB and other formats). You can also make one with the
221
+ built-in character generator.
222
+ 3. **Switch to Animation mode** and find **Generate from text** in the Inspector's Animations section.
223
+ 4. **Pick a model:** set **Model** to **QtMesh T2M (experimental, small)**.
224
+ 5. **Download the model:** press **Download QtMesh T2M**. This happens once, about 740 MB. Nothing downloads
225
+ until you ask.
226
+ 6. **Write a prompt**, for example *"a person waves with the right hand"*. Set the duration (up to 5 s) and the
227
+ number of variations. Optionally turn on **Root motion** or set a **Fixed seed**.
228
+ 7. **Generate.** The clips appear as `generated_qtmesh_t2m`, `generated_qtmesh_t2m_v2`, … All variations are added
229
+ in one undo step. Preview them on the timeline.
230
+ 8. **Export** the character with its animations (for example to `.glb` or `.fbx`).
231
+
232
+ ### Advanced: the command line
233
+
234
+ ```bash
235
+ qtmesh anim rigged.glb \
236
+ --generate "a person waves with the right hand" \
237
+ --backend qtmesh_t2m \
238
+ --duration 3 \
239
+ --download \
240
+ -o animated.glb
241
+ ```
242
 
243
+ `--download` fetches the model files the first time. Other options:
244
 
245
+ | Option | What it does |
246
+ |---|---|
247
+ | `--variations N` | Generates N clips, 1–16. |
248
+ | `--seed N` | Makes the result reproducible. |
249
+ | `--no-root-motion` | Keeps the clip in place. |
250
+ | `--name NAME` | Names the clip. |
251
+ | `--json` | Prints a machine-readable report, including the timings. |
252
 
253
+ ### MCP (AI agents)
 
 
 
254
 
255
+ QtMeshEditor's MCP server exposes the same thing through the `generate_motion` tool:
256
 
257
+ ```json
258
+ {"backend": "qtmesh_t2m", "prompt": "a person jumps", "duration": 3, "variations": 2, "seed": 2, "download": true}
 
 
259
  ```
260
 
261
+ ### Using the files directly
 
 
 
262
 
263
+ You can run the ONNX model in your own pipeline with ONNX Runtime and any llama.cpp build that supports Qwen3
264
+ embeddings. See [Files and tensor contract](#files-and-tensor-contract). You then need your own retargeting from
265
+ the 22-joint canonical skeleton, which is described in `t2m-text.json`.
 
 
266
 
267
+ > **About the GGUF file.** `Qwen3-Embedding-0.6B-Q8_0.gguf` is a **text-embedding model**. It is **not** a chat
268
+ > model and **not** the animation generator. Hugging Face may show "Use this model with llama.cpp / Ollama"
269
+ > buttons because of it. Those apply only to the embedding model, not to the text-to-motion pipeline, and chatting
270
+ > with it will not produce anything useful.
271
 
272
+ ## Files and tensor contract
 
 
 
 
 
 
273
 
274
+ The current model lives in [`qtmesh_t2m/`](qtmesh_t2m):
275
 
276
+ | File | Size | What it is |
277
+ |---|---|---|
278
+ | `t2m-text.onnx` | 98 MB | The motion model (22.6 M parameters), with the 24-step Euler + CFG sampler unrolled in the graph. |
279
+ | `t2m-text.json` | small | The contract: shapes, fps, the embedding instruction and pooling, canonical rest directions. |
280
+ | `Qwen3-Embedding-0.6B-Q8_0.gguf` | ~640 MB | The text encoder. It is byte-identical to Qwen's official [Q8_0 GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF) (SHA-256 `06507c7b…e439`). |
281
+ | `LICENSE-Qwen3-Embedding.txt` | | The Apache-2.0 licence for the encoder. |
282
+ | `ATTRIBUTION.md` | | The required credits for the CC-BY-4.0 motion data the model was trained on. |
 
 
 
 
 
 
 
 
 
 
 
 
283
 
284
+ **`t2m-text.onnx` I/O** (checked against the graph):
285
 
286
+ | Tensor | Shape / type | Meaning |
287
+ |---|---|---|
288
+ | input `text` | `float32 [1, 1024]` | The L2-normalised Qwen3 embedding of `instruct + prompt`. A zero vector means unconditional. |
289
+ | input `seed` | `float32 [1, 19800]` | Starting noise, sampled from N(0, 0.5). 19800 = 150 frames × 132. |
290
+ | input `length` | `float32 [1]` | The number of frames wanted, up to 150. |
291
+ | output `motion` | `float32 [1, 150, 220]` | Per frame, 22 joints × 10 values `[tx, ty, tz, qx, qy, qz, qw, sx, sy, sz]`: world-frame rotation quaternions, with t = 0 and s = 1. Frames from `length` onward are padding. |
292
 
293
+ From `t2m-text.json`:
 
 
 
294
 
295
+ ```text
296
+ version v9-text T 150 J 22 C 220 Z 19800 fps 30 frame world
297
+ flowSteps 24 guidance 2.0 embedDim 1024 pooling last
 
 
 
298
  ```
299
 
300
+ The joint order and the canonical rest pose (`restWorld`, `restDir`) are listed in the JSON.
301
+
302
+ ## Limitations
303
+
304
+ - **Lower motion fidelity than Kimodo.** Motions are simpler, less expressive and less physically convincing.
305
+ - **Occasionally wrong or too subtle.** Some prompts give a weak or only partly correct motion.
306
+ - **Sensitive to the seed.** Quality varies a lot between seeds (see the jump example), so generate several
307
+ variations and pick the best one.
308
+ - **One action prompt per clip.** There are no multi-segment timelines; QtMeshEditor rejects a multi-line prompt
309
+ for this model. Kimodo supports timelines.
310
+ - **At most 5 seconds** per clip (150 frames at 30 fps).
311
+ - **No root height or root translation.** Jumps stay on the ground plane. Horizontal travel is only an estimate
312
+ made by the editor.
313
+ - **Weaker on uncommon movements.** Actions that are rare in the training data come out worse.
314
+ - **Body only.** There are no fingers, face or props; the model drives 22 body joints. In QtMeshEditor, Kimodo
315
+ drives the fingers.
316
+ - **English prompts.** The training text was English only.
317
+ - **Experimental** and under active development. Expect the model to change.
318
+
319
+ ## License and attribution
320
+
321
+ This repository contains files under **different licences**, so its metadata says `license: other` (mixed, per
322
+ file) instead of claiming a single licence. The full per-file breakdown is in [`LICENSES.md`](LICENSES.md).
323
+ In short:
324
+
325
+ | Files | Licence |
326
+ |---|---|
327
+ | QtMesh T2M weights and contracts (`qtmesh_t2m/t2m-text.onnx`, `t2m-text.json`), the v8 model (`t2m.onnx`, `t2m-vocab.json`), the GIFs in `media/` | **CC0-1.0**, released by the QtMeshEditor project. The models were trained on CC-BY-4.0 data, so please keep the attribution files with them. |
328
+ | `qtmesh_t2m/Qwen3-Embedding-0.6B-Q8_0.gguf` | **Apache-2.0**, © Qwen team, Alibaba Cloud ([`LICENSE-Qwen3-Embedding.txt`](qtmesh_t2m/LICENSE-Qwen3-Embedding.txt)). |
329
+ | `motion-library.json`, `motion-library-v2.json` | **Mixed CC0-1.0 and CC-BY-4.0**, per clip. Credits are in [`ATTRIBUTION-v2.md`](ATTRIBUTION-v2.md) and [`ATTRIBUTION.md`](ATTRIBUTION.md). |
330
+ | `ATTRIBUTION.md`, `ATTRIBUTION-v2.md`, `qtmesh_t2m/ATTRIBUTION.md` | The credits for the CC-BY-4.0 works. **Ship them with the models or the data.** |
331
+
332
+ **About the Kimodo-generated training clips:**
333
+
334
+ - Kimodo's outputs are not "Derivative Models" under the
335
+ [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
336
+ (§1.1), and NVIDIA claims no ownership of outputs (§2.4).
337
+ - The QtMeshEditor project therefore used Kimodo outputs as training data.
338
+ - No Kimodo weights, code, generated clips or Llama 3 embeddings are distributed in this repository. Kimodo itself
339
+ is a separate download inside QtMeshEditor, under its own licences.
340
+
341
+ QtMeshEditor itself is MIT-licensed.
342
+
343
+ ## Previous versions
344
+
345
+ QtMesh T2M (v9, in `qtmesh_t2m/`) is the **current** model. The files at the repository root belong to earlier
346
+ releases and are kept for compatibility and reproducibility:
347
+
348
+ - `t2m.onnx` + `t2m-vocab.json` is the **v8.0 action-label model**: a 21.7 M-parameter flow-matching transformer
349
+ conditioned on one of 30 action words. It is QtMeshEditor's opt-in `--model` path.
350
+ - `motion-library-v2.json` (150 clips) is the curated template clip library. It is also the v8 training source.
351
+ - `motion-library.json` is the older v1 template library.
352
+
353
+ The full v8 model card (dataset, measurements, architecture, I/O contract, reproduction) is in
354
+ [`PREVIOUS_VERSIONS.md`](PREVIOUS_VERSIONS.md). Earlier checkpoints remain in this repository's
355
+ [commit history](https://huggingface.co/fernandotonon/QtMeshEditor-t2m/commits/main).
356
+
357
+ ## Links
358
+
359
+ - **QtMeshEditor:** [GitHub](https://github.com/fernandotonon/QtMeshEditor).
360
+ - **Documentation:**
361
+ - [Kimodo and QtMesh T2M user guide](https://github.com/fernandotonon/QtMeshEditor/blob/master/docs/KIMODO.md).
362
+ - [Third-party model licences](https://github.com/fernandotonon/QtMeshEditor/blob/master/THIRD_PARTY_AI_MODELS.md).
363
+ - **Inference code:** [`QtMeshT2M.cpp`](https://github.com/fernandotonon/QtMeshEditor/blob/master/src/AIMotion/QtMeshT2M.cpp)
364
+ and [`MotionSceneApply.cpp`](https://github.com/fernandotonon/QtMeshEditor/blob/master/src/AIMotion/MotionSceneApply.cpp)
365
+ (retargeting and root motion).
366
+ - **Training and export scripts:** see [Training](#training).
367
+ - **Model files:** this repository, plus the
368
+ [QtMeshEditor-models](https://huggingface.co/fernandotonon/QtMeshEditor-models/tree/main/qtmesh_t2m) mirror that
369
+ the app downloads from (pinned commit and SHA-256 digests).
370
+ - **Training motion corpus:** [fernandotonon/QtMeshEditor-motion-corpus](https://huggingface.co/datasets/fernandotonon/QtMeshEditor-motion-corpus).
371
+ - **NVIDIA Kimodo:** [github.com/nv-tlabs/kimodo](https://github.com/nv-tlabs/kimodo).
372
+ - **Qwen3-Embedding-0.6B:** [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)
373
+ ([GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF)).
media/compare_jump_seeds.gif ADDED

Git LFS Details

  • SHA256: 09adfd9f0ff6232c61f0b603530d39f1aeb6e020129e377a13bbcbb178170e7a
  • Pointer size: 131 Bytes
  • Size of remote file: 254 kB