Add model card with image-to-video pipeline tag and quick start guide
#1
by nielsr HF Staff - opened
README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: image-to-video
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
# HunyuanVideo-HOMA
|
| 6 |
+
|
| 7 |
+
HOMA (Towards Generic Human-Object Interaction in Multimodal Driven Human Animation with Weak Conditions) is a weakly conditioned multimodal-driven framework that turns a reference person image, an object image, speech audio, sparse motion, and a text prompt into a physically plausible human-object interaction video.
|
| 8 |
+
|
| 9 |
+
* **Paper:** [HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation](https://huggingface.co/papers/2506.08797)
|
| 10 |
+
* **Project Page:** [https://bone-11.github.io/homa-page/](https://bone-11.github.io/homa-page/)
|
| 11 |
+
* **Repository:** [https://github.com/bone-11/homa](https://github.com/bone-11/homa)
|
| 12 |
+
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
## Quick Start (Inference)
|
| 16 |
+
|
| 17 |
+
To use these checkpoints, please clone the official GitHub repository and install the dependencies.
|
| 18 |
+
|
| 19 |
+
### 1. Environment Setup
|
| 20 |
+
|
| 21 |
+
```bash
|
| 22 |
+
conda create -n homa python=3.10 -y
|
| 23 |
+
conda activate homa
|
| 24 |
+
|
| 25 |
+
conda install -c conda-forge ffmpeg -y
|
| 26 |
+
|
| 27 |
+
# Match the CUDA wheel to your machine (e.g., CUDA 12.4)
|
| 28 |
+
pip install "torch>=2.4.0" torchvision --index-url https://download.pytorch.org/whl/cu124
|
| 29 |
+
|
| 30 |
+
pip install flash-attn --no-build-isolation
|
| 31 |
+
pip install -r requirements-infer.txt
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
### 2. Download Model Weights
|
| 35 |
+
|
| 36 |
+
First, download the public dependencies:
|
| 37 |
+
```bash
|
| 38 |
+
bash download_models.sh
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
Then, download the HOMA checkpoint from this repository:
|
| 42 |
+
```bash
|
| 43 |
+
huggingface-cli download ProAudience/homa_checkpoint --local-dir checkpoints/homa_checkpoint
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
### 3. Run Inference
|
| 47 |
+
|
| 48 |
+
Configure the paths in `run_inference.sh`:
|
| 49 |
+
```bash
|
| 50 |
+
CHECKPOINT_DIR="checkpoints/homa_checkpoint"
|
| 51 |
+
MODEL_BASE="checkpoints/other_models/pretrained_models"
|
| 52 |
+
MODEL_AUX_PATH="checkpoints/other_models/aux"
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
Then run the demo:
|
| 56 |
+
```bash
|
| 57 |
+
bash run_inference.sh
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
For more detailed setup and running customized inference cases, please refer to the [GitHub repository](https://github.com/bone-11/homa).
|
| 61 |
+
|
| 62 |
+
## Citation
|
| 63 |
+
|
| 64 |
+
If you find HOMA useful for your research, please cite the paper:
|
| 65 |
+
|
| 66 |
+
```bibtex
|
| 67 |
+
@inproceedings{huang2025homa,
|
| 68 |
+
title = {HOMA: Towards Generic Human-Object Interaction in Multimodal Driven Human Animation with Weak Conditions},
|
| 69 |
+
author = {Huang, Ziyao and Zhou, Zixiang and Cao, Juan and Ma, Yifeng and Chen, Yi and Rao, Zejing and Xu, Zhiyong and Wang, Hongmei and Lin, Qin and Zhou, Yuan folding and Lu, Qinglin and Tang, Fan},
|
| 70 |
+
booktitle = {SIGGRAPH Asia 2025 Conference Papers},
|
| 71 |
+
year = {2025},
|
| 72 |
+
}
|
| 73 |
+
```
|