| --- |
| library_name: pytorch |
| license: other |
| license_name: upstream-and-data-licenses-apply |
| tags: |
| - motion-generation |
| - text-to-motion |
| - motion-to-motion |
| - smpl-x |
| - vimogen |
| --- |
| |
| # ViMoGen-Hand-456 checkpoints |
|
|
| This model repository stores the selected large checkpoints for the |
| ViMoGen-Hand-456 experiments. The project extends the original ViMoGen 276D |
| body representation with 180D articulated hand rotations: |
|
|
| ```text |
| motion456 = body276 || left_hand90 || right_hand90 |
| ``` |
|
|
| The complete source, experiment configs, training records, reports and |
| evaluation artifacts are distributed separately in the `ViMoGen_Hand` source |
| repository. These weights require the original ViMoGen implementation and its |
| licensed dependencies; they are not standalone Transformers checkpoints. |
|
|
| ## Representation |
|
|
| | Channels | Meaning | |
| |---|---| |
| | `0:126` | 21 non-root body-joint rotations in Rot6D | |
| | `126:192` | 22 global body-joint XYZ positions | |
| | `192:258` | 22 body-joint XYZ velocities | |
| | `258:264` | root/global orientation in Rot6D | |
| | `264:270` | root orientation velocity in Rot6D | |
| | `270:273` | root translation XYZ | |
| | `273:276` | root translation velocity XYZ | |
| | `276:366` | 15 left-hand joint rotations in Rot6D | |
| | `366:456` | 15 right-hand joint rotations in Rot6D | |
|
|
| The hand block contains local rotations only. It does not add hand joint |
| positions, hand velocities, face parameters or body shape parameters. |
|
|
| TM2M body conditioning uses a 138D reference consisting of body channels |
| `0:126` and `258:270`. Hand conditioning uses the full 180D hand pose. Unified |
| models concatenate both conditions into a 318D reference. |
|
|
| ## Files |
|
|
| | Remote path | Purpose | Size | SHA256 | |
| |---|---|---:|---| |
| | `hand180/step_00014000/model.pt` | Independent Hand180 model used by Concat, as the hand teacher, and by Two-Expert | 9,093,836,032 B | `daa66c4d29d2b94b33da392e3d4920f067fcd37965daadcadd47cf0966066f7b` | |
| | `unified_distill/stage2/step_00015000/model.pt` | Selected single-network Stage2 unified 456D model | 9,098,076,480 B | `443843959903f4a20639ce9d50e3cd023de694f51628c88f0998da191a381492` | |
| | `direct456/step_00020000/model.pt` | Selected Direct456 all-parameter ablation | 9,098,076,480 B | `e63221693337d4b2b07e8dd124ea992d5a0b798884bb6ec51e383f1e3a8a69ec` | |
|
|
| All three are RAW FP32 model states used by the archived primary comparisons. |
| EMA weights and optimizer states are intentionally not included. |
|
|
| The original ViMoGen Body276 checkpoint must be downloaded from the official |
| ViMoGen release. The compact Two-Expert fusion state is stored in the source |
| repository because it is only 7.2 MB. The 18.2 GB full Two-Expert checkpoint is |
| not uploaded because it duplicates the frozen Body276 and Hand180 experts. |
|
|
| ## Architectures represented here |
|
|
| ### Hand180 |
|
|
| The original ViMoGen TM2M topology is retained but its motion/reference/output |
| domains are changed to `180/180/180`. It contains 30 main Transformer blocks, |
| 15 reference self-attention blocks, hidden width 1536, 12 heads and FFN width |
| 8960. All Hand180 parameters were trained with hand-only flow matching. |
|
|
| ### Stage2 unified |
|
|
| A single original-topology ViMoGen Student uses `456/318/456` I/O. It was first |
| distilled from frozen Body276 and Hand180 teachers, then trained with equal |
| body/hand GT flow losses plus a 0.1 teacher anchor. The selected raw checkpoint |
| is step 15000 of Stage2. |
|
|
| ### Direct456 |
|
|
| A single original-topology `456/318/456` ViMoGen model was initialized from |
| all shape-compatible Body276 weights and then trained end-to-end on equal |
| body/hand GT flow losses without teachers. This checkpoint is retained as the |
| direct all-parameter ablation. |
|
|
| ## Training data and common settings |
|
|
| - 25,861 processed Motion-X/Motion-X++ motions; |
| - 3,194,032 frames at 20 FPS; |
| - source SMPL-X 322D converted to `[T,456]`; |
| - T2M/TM2M sampling ratio 20/80; |
| - AdamW, betas `(0.9, 0.95)`, weight decay `1e-4`; |
| - BF16 compute with FP32 master parameters, optimizer and EMA; |
| - 7-GPU FSDP, global batch 112; |
| - 50 flow sampling steps and denoising strength 0.7 for primary evaluation. |
|
|
| The dataset and SMPL-X body model are not redistributed here. |
|
|
| ## Download |
|
|
| ```bash |
| hf download Agentc1/ViMoGen_Hand_456 \ |
| hand180/step_00014000/model.pt \ |
| unified_distill/stage2/step_00015000/model.pt \ |
| direct456/step_00020000/model.pt \ |
| checkpoint_manifest.json \ |
| --local-dir checkpoints_external |
| ``` |
|
|
| Verify SHA256 values against `checkpoint_manifest.json` before evaluation. |
|
|
| ## Main benchmark result |
|
|
| Mean raw-motion MSE on the archived 16-case strict GRAB OOD TM2M suite: |
|
|
| | Method | Full | Body | Hand | |
| |---|---:|---:|---:| |
| | Frozen Two-Expert | 0.017182 | 0.026319 | 0.003171 | |
| | Body276 + Hand180 Concat | 0.017617 | 0.026920 | 0.003353 | |
| | Stage2 unified | 0.021346 | 0.032927 | 0.003589 | |
| | Direct456 | 0.021307 | 0.031046 | 0.006373 | |
|
|
| The historical method runs did not use a perfectly identical seed map, so the |
| small Two-Expert-versus-Concat difference should be treated as a trend rather |
| than a statistical significance claim. |
|
|
| ## Licenses and attribution |
|
|
| Use of these checkpoints is subject to the licenses and terms of the original |
| ViMoGen/Wan components, Motion-X/Motion-X++ data, SMPL-X models and any source |
| datasets. This repository does not grant redistribution rights for those |
| dependencies. Please cite the original ViMoGen, Motion-X and applicable source |
| dataset papers when using the models. |
|
|
