MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model
Paper • 2405.20222 • Published • 11
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image, export_to_video
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("MyNiuuu/MOFA-Video-Hybrid", dtype=torch.bfloat16, device_map="cuda")
pipe.to("cuda")
prompt = "A man with short gray hair plays a red electric guitar."
image = load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png"
)
output = pipe(image=image, prompt=prompt).frames[0]
export_to_video(output, "output.mp4")Paper
arxiv.org/abs/2405.20222
Introduction
This repo provides the inference Gradio demo for Hybrid (Trajectory + Landmark) Control of MOFA-Video.
Environment Setup
cd MOFA-Hybrid
conda create -n mofa python==3.10
conda activate mofa
pip install -r requirements.txt
pip install opencv-python-headless
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
IMPORTANT: Gradio Version of 4.5.0 should be used since other versions may cause errors.
Checkpoints Download
Download the checkpoint of CMP from here and put it into
./models/cmp/experiments/semiauto_annot/resnet50_vip+mpii_liteflow/checkpoints.Downloading the necessary pretrained checkpoints from huggingface. It is recommended to directly using git lfs to clone the huggingface repo. The checkpoints should be orgnized as
./ckpt_tree.md(they will be automatically organized if you use git lfs to clone the huggingface repo).
Run Gradio Demo
Using audio to animate the facial part
python run_gradio_audio_driven.py
Using refernce video to animate the facial part
python run_gradio_video_driven.py
IMPORTANT: Please refer to the instructions on the gradio interface during the inference process.
- Downloads last month
- -