| --- |
| language: |
| - en |
| license: other |
| license_name: nvidia-open-model-agreement |
| license_link: LICENSE |
| base_model: |
| - nvidia/MotionBricks |
| library_name: diffusers |
| tags: |
| - motionbricks |
| - modular-diffusers |
| - motion-generation |
| - motion-inbetweening |
| - soma |
| --- |
| |
| # MotionBricks SOMA |
|
|
| <video controls loop muted playsinline src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/motionbricks_soma_diffusers/motionbricks_soma_run_stop.mp4"></video> |
|
|
| *The start of a Kimodo run, then the 30 frames generated with the code below, then the standing end of another Kimodo take, placed 1.4 m ahead. The generated frames are drawn on SOMA77 with a relaxed finger pose. At normal playback speed; the cameras follow the root.* |
|
|
| This is the SOMA model of [nvidia/MotionBricks](https://huggingface.co/nvidia/MotionBricks), repacked into the Diffusers format in float32. It runs as a Modular Diffusers pipeline with the [MotionBricks SOMA custom blocks](https://huggingface.co/OzzyGT/motionbricks_soma_custom_blocks) and generates the transition from the last poses of one motion to target poses in one forward pass, choosing the length of the transition itself. |
|
|
| The three components work together: |
|
|
| - `root_model` (40M parameters) chooses the length and predicts the root trajectory. |
| - `pose_model` (137M parameters) predicts the pose codes of every token of four frames. |
| - `pose_decoder` (12M parameters) decodes the codes to frames. |
|
|
| Note: You need `diffusers>=0.41.0`. |
|
|
| ## Usage |
|
|
| The custom blocks load the model code shipped in this repository, hence `trust_remote_code` and `trust_components_code`. |
|
|
| **Note:** both flags run Python code downloaded from the Hub: `trust_remote_code=True` runs the code of the blocks repository, and `trust_components_code=True` runs the code shipped in the `components_repo`. Only use them with repositories you trust, and review their code before passing different ones. |
|
|
| ```python |
| import numpy as np |
| import torch |
| from diffusers import ModularPipelineBlocks |
| from huggingface_hub import hf_hub_download |
| |
| blocks = ModularPipelineBlocks.from_pretrained( |
| "OzzyGT/motionbricks_soma_custom_blocks", |
| trust_remote_code=True, |
| components_repo="OzzyGT/motionbricks_soma_diffusers", |
| trust_components_code=True, |
| ) |
| pipe = blocks.init_pipeline() |
| pipe.load_components(dtype=torch.float32) |
| pipe.to("cuda") |
| |
| first = hf_hub_download("OzzyGT/diffusers-examples", "motionbricks_soma_diffusers/first.npz", repo_type="dataset") |
| second = hf_hub_download("OzzyGT/diffusers-examples", "motionbricks_soma_diffusers/second.npz", repo_type="dataset") |
| context = dict(np.load(first)) |
| target = dict(np.load(second)) |
| |
| motion = pipe(context=context, target=target, target_offset=(1.4, 0.0), num_target_poses=4, output="motion")[0] |
| np.savez_compressed("transition.npz", **{key: value.cpu().numpy() for key, value in motion.items()}) |
| ``` |
|
|
| Both clips come from [Kimodo SOMA Modular](https://huggingface.co/OzzyGT/Kimodo_SOMA_Modular): the start of a run, and a standing pose that `target_offset` places 1.4 m ahead of it. The model accepts SOMA30 or SOMA77 poses at 30 FPS and returns a SOMA30 motion; here it brings the run to a stop in 30 frames. See the [custom blocks](https://huggingface.co/OzzyGT/motionbricks_soma_custom_blocks) for the options. |
|
|
| Float32 reproduces NVIDIA's ONNX graph with PyTorch's default settings; float16 halves the memory with small drift, while bfloat16 changes the motion noticeably. |
|
|
| ## Licenses |
|
|
| The model weights are licensed under the [NVIDIA Open Model Agreement](LICENSE), the same as the original. The model code in this repository is Apache-2.0. See [NOTICE](NOTICE) for what was changed. |
|
|
