VIRES: Video Instance Repainting with Sketch and Text Guidance
Paper • 2411.16199 • Published • 1
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image, export_to_video
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("suimu/VIRES_VAE", dtype=torch.bfloat16, device_map="cuda")
pipe.to("cuda")
prompt = "A man with short gray hair plays a red electric guitar."
image = load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png"
)
output = pipe(image=image, prompt=prompt).frames[0]
export_to_video(output, "output.mp4")VIRES_VAE model card
Model Page: VIRES
This is a component of the VIRES model, as presented in VIRES: Video Instance Repainting with Sketch and Text Guidance. VIRES is a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. It leverages the generative priors of text-to-video models to maintain temporal consistency and produce visually pleasing results.
Usage
Ref to our GitHub page: https://github.com/suimuc/VIRES/
Citation
@article{vires,
title={VIRES: Video Instance Repainting via Sketch and Text Guided Generation},
author={Weng, Shuchen and Zheng, Haojie and Zhang, Peixuan and Hong, Yuchen and Jiang, Han and Li, Si and Shi, Boxin},
journal={arXiv preprint arXiv:2411.16199},
year={2024}
}
- Downloads last month
- 4