File size: 3,644 Bytes
338c3c5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
language:
- en
license: other
license_name: nvidia-open-model-agreement
license_link: LICENSE
base_model:
- nvidia/MotionBricks
library_name: diffusers
tags:
- motionbricks
- modular-diffusers
- motion-generation
- motion-inbetweening
- soma
---
# MotionBricks SOMA
<video controls loop muted playsinline src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/motionbricks_soma_diffusers/motionbricks_soma_run_stop.mp4"></video>
*The start of a Kimodo run, then the 30 frames generated with the code below, then the standing end of another Kimodo take, placed 1.4 m ahead. The generated frames are drawn on SOMA77 with a relaxed finger pose. At normal playback speed; the cameras follow the root.*
This is the SOMA model of [nvidia/MotionBricks](https://huggingface.co/nvidia/MotionBricks), repacked into the Diffusers format in float32. It runs as a Modular Diffusers pipeline with the [MotionBricks SOMA custom blocks](https://huggingface.co/OzzyGT/motionbricks_soma_custom_blocks) and generates the transition from the last poses of one motion to target poses in one forward pass, choosing the length of the transition itself.
The three components work together:
- `root_model` (40M parameters) chooses the length and predicts the root trajectory.
- `pose_model` (137M parameters) predicts the pose codes of every token of four frames.
- `pose_decoder` (12M parameters) decodes the codes to frames.
Note: You need `diffusers>=0.41.0`.
## Usage
The custom blocks load the model code shipped in this repository, hence `trust_remote_code` and `trust_components_code`.
**Note:** both flags run Python code downloaded from the Hub: `trust_remote_code=True` runs the code of the blocks repository, and `trust_components_code=True` runs the code shipped in the `components_repo`. Only use them with repositories you trust, and review their code before passing different ones.
```python
import numpy as np
import torch
from diffusers import ModularPipelineBlocks
from huggingface_hub import hf_hub_download
blocks = ModularPipelineBlocks.from_pretrained(
"OzzyGT/motionbricks_soma_custom_blocks",
trust_remote_code=True,
components_repo="OzzyGT/motionbricks_soma_diffusers",
trust_components_code=True,
)
pipe = blocks.init_pipeline()
pipe.load_components(dtype=torch.float32)
pipe.to("cuda")
first = hf_hub_download("OzzyGT/diffusers-examples", "motionbricks_soma_diffusers/first.npz", repo_type="dataset")
second = hf_hub_download("OzzyGT/diffusers-examples", "motionbricks_soma_diffusers/second.npz", repo_type="dataset")
context = dict(np.load(first))
target = dict(np.load(second))
motion = pipe(context=context, target=target, target_offset=(1.4, 0.0), num_target_poses=4, output="motion")[0]
np.savez_compressed("transition.npz", **{key: value.cpu().numpy() for key, value in motion.items()})
```
Both clips come from [Kimodo SOMA Modular](https://huggingface.co/OzzyGT/Kimodo_SOMA_Modular): the start of a run, and a standing pose that `target_offset` places 1.4 m ahead of it. The model accepts SOMA30 or SOMA77 poses at 30 FPS and returns a SOMA30 motion; here it brings the run to a stop in 30 frames. See the [custom blocks](https://huggingface.co/OzzyGT/motionbricks_soma_custom_blocks) for the options.
Float32 reproduces NVIDIA's ONNX graph with PyTorch's default settings; float16 halves the memory with small drift, while bfloat16 changes the motion noticeably.
## Licenses
The model weights are licensed under the [NVIDIA Open Model Agreement](LICENSE), the same as the original. The model code in this repository is Apache-2.0. See [NOTICE](NOTICE) for what was changed.
|