| --- |
| language: |
| - en |
| license: apache-2.0 |
| base_model: |
| - nvidia/Kimodo-SOMA-RP-v1.2 |
| library_name: diffusers |
| tags: |
| - text-to-motion |
| - motion-generation |
| - kimodo |
| - modular-diffusers |
| - soma |
| --- |
| |
| # Kimodo SOMA Modular Diffusers |
|
|
| <video controls loop muted playsinline src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/Kimodo_SOMA_Modular/Kimodo_SOMA_RP_v1_2_run_seed42.mp4"></video> |
|
|
| *A four-second motion generated with the code below, seed `42`, at normal playback speed. The cameras follow the original root motion.* |
|
|
| Modular Diffusers blocks for [NVIDIA Kimodo SOMA RP v1.2](https://huggingface.co/nvidia/Kimodo-SOMA-RP-v1.2): generate human motion from a text prompt. The model code and weights are in [OzzyGT/Kimodo_SOMA_RP_v1_2](https://huggingface.co/OzzyGT/Kimodo_SOMA_RP_v1_2). |
|
|
| The same blocks also support [Kimodo SOMA RP v1.1](https://huggingface.co/OzzyGT/Kimodo_SOMA_RP_v1_1); set `components_repo` to that repository to use the earlier model. |
|
|
| Requires PyTorch, `diffusers==0.41.0`, `transformers==5.20.2`, `peft==0.21.2`, `accelerate>=1.15`, `safetensors>=0.8`, and `numpy`. |
|
|
| ## Licensing |
|
|
| The blocks are [Apache-2.0](LICENSE), adapted from [Kimodo](https://github.com/nv-tlabs/kimodo), with [LLM2Vec](https://github.com/McGill-NLP/llm2vec) helpers under [MIT](LICENSE-LLM2VEC). See [NOTICE](NOTICE). This repository contains no model weights. Motion weights use the NVIDIA Open Model License; the text encoder uses the Meta Llama 3 Community License. See the [model repository](https://huggingface.co/OzzyGT/Kimodo_SOMA_RP_v1_2#licenses) for those terms. Built with Meta Llama 3. |
|
|
| ## Usage |
|
|
| Generate a four-second motion and save it as NPZ: |
|
|
| ```python |
| import numpy as np |
| import torch |
| from diffusers import ModularPipelineBlocks |
| |
| blocks_path = "OzzyGT/Kimodo_SOMA_Modular" |
| model_id = "OzzyGT/Kimodo_SOMA_RP_v1_2" |
| |
| blocks = ModularPipelineBlocks.from_pretrained( |
| blocks_path, |
| trust_remote_code=True, |
| components_repo=model_id, |
| trust_components_code=True, |
| ) |
| pipe = blocks.init_pipeline() |
| pipe.load_components(dtype={"text_encoder": torch.bfloat16, "default": torch.float32}) |
| pipe.to("cuda") |
| |
| motion = pipe( |
| prompt="A person runs forward at a steady pace.", |
| num_frames=120, |
| seed=42, |
| num_inference_steps=100, |
| output="motion", |
| ) |
| np.savez_compressed( |
| "motion.npz", |
| **{key: value[0].cpu().numpy() for key, value in motion.items()}, |
| ) |
| ``` |
|
|
| Output is 30 FPS, meters, +Y up, initial +Z forward, SOMA77, with root motion preserved. Vary `seed` for another take. Negative prompts are not supported. |
|
|
