MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model
Paper • 2405.20222 • Published • 11
import torch
from diffusers import DiffusionPipeline
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("MyNiuuu/MOFA-Video-Traj", dtype=torch.bfloat16, device_map="cuda")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]Updates 🔥🔥🔥
We have released the Gradio demo for Hybrid (Trajectory + Landmark) Controls HERE!
Introduction
This repo provides the inference Gradio demo for Trajectory Control of MOFA-Video.
Environment Setup
pip install -r requirements.txt
Download checkpoints
Download the pretrained checkpoints of SVD_xt from huggingface to
./ckpts.Download the checkpint of MOFA-Adapter from huggingface to
./ckpts.
The final structure of checkpoints should be:
./ckpts/
|-- controlnet
| |-- config.json
| `-- diffusion_pytorch_model.safetensors
|-- stable-video-diffusion-img2vid-xt-1-1
| |-- feature_extractor
| |-- ...
| |-- image_encoder
| |-- ...
| |-- scheduler
| |-- ...
| |-- unet
| |-- ...
| |-- vae
| |-- ...
| |-- svd_xt_1_1.safetensors
| `-- model_index.json
Run Gradio Demo
python run_gradio.py
Please refer to the instructions on the gradio interface during the inference process.
Paper
arxiv.org/abs/2405.20222
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support