Instructions to use stabilityai/stable-video-diffusion-img2vid-xt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use stabilityai/stable-video-diffusion-img2vid-xt with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image, export_to_video # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-video-diffusion-img2vid-xt", dtype=torch.bfloat16, device_map="cuda") pipe.to("cuda") prompt = "A man with short gray hair plays a red electric guitar." image = load_image( "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png" ) output = pipe(image=image, prompt=prompt).frames[0] export_to_video(output, "output.mp4") - Notebooks
- Google Colab
- Kaggle
Docker Image/Runpod.IO setup
Wanted to share my work on containerising/templating SVD deployment for runpod.io instances
Here's a Dockehub image, it has all of the dependencies required to run SVD and automatically downloads the weights to /workspace directory if it can not find them there (this will be configurable soon)
latest or svd-base-xt tag auto-downloads both models, if you only want a specific model use svd-base/svd-xt tags respectively
https://hub.docker.com/repository/docker/mnb3000/runpod-stable-video-diffusion/
And here are automatic preconfigured templates for runpod instances
Base model:
https://runpod.io/gsc?template=dove05wvcv
XT model:
https://runpod.io/gsc?template=0yuqqd2v24
Streamlit UI is available on port 3000
SVD-XT runs fine on RTX A6000 instances for $0.8/hr with discrete_t set to 8
Update 29.11:
- Dockerfile/shell source is now available in GitHub Repo
- Added
$MODEL_MOUNTPOINTand$PORTenv variables - Removed nginx reverse proxying
- Added detailed instructions to run the image locally with directory mounting examples in README on Dockerhub/GitHub
Update 30.11:
- Replaced git clone with direct wget download for weights, as the repo was updated with diffusers weights, which are not used to run streamlit demo
- Added build instructions to GitHub README
- Fixed weights symlinks to
checkpoints/folder
The image is available on (Dockerhub)[https://hub.docker.com/repository/docker/mnb3000/runpod-stable-video-diffusion/] with detailed instructions for running it locally or with a cloud template
Hey @mnb3000 , is it possible to run it with 3060ti 12GB and a 16GB RAM? What should be the configuration for these specs?