Update README.md
Browse files
README.md
CHANGED
|
@@ -179,7 +179,101 @@ Install diffusers from source:
|
|
| 179 |
pip install git+https://github.com/huggingface/diffusers.git
|
| 180 |
```
|
| 181 |
|
| 182 |
-
For example, let's take Helios-Distilled.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
|
| 184 |
<details>
|
| 185 |
<summary>Click to expand the code</summary>
|
|
|
|
| 179 |
pip install git+https://github.com/huggingface/diffusers.git
|
| 180 |
```
|
| 181 |
|
| 182 |
+
For example, let's take Helios-Distilled (**Standard Pipeline**).
|
| 183 |
+
|
| 184 |
+
<details>
|
| 185 |
+
<summary>Click to expand the code</summary>
|
| 186 |
+
|
| 187 |
+
```bash
|
| 188 |
+
import torch
|
| 189 |
+
from diffusers import AutoModel, HeliosPyramidPipeline
|
| 190 |
+
from diffusers.utils import export_to_video, load_video, load_image
|
| 191 |
+
|
| 192 |
+
vae = AutoModel.from_pretrained("BestWishYsh/Helios-Distilled", subfolder="vae", torch_dtype=torch.float32)
|
| 193 |
+
|
| 194 |
+
pipeline = HeliosPyramidPipeline.from_pretrained(
|
| 195 |
+
"BestWishYsh/Helios-Distilled",
|
| 196 |
+
vae=vae,
|
| 197 |
+
torch_dtype=torch.bfloat16
|
| 198 |
+
)
|
| 199 |
+
pipeline.to("cuda")
|
| 200 |
+
|
| 201 |
+
negative_prompt = """
|
| 202 |
+
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
|
| 203 |
+
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
|
| 204 |
+
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
|
| 205 |
+
"""
|
| 206 |
+
|
| 207 |
+
# --- T2V ---
|
| 208 |
+
prompt = """
|
| 209 |
+
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue
|
| 210 |
+
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with
|
| 211 |
+
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear,
|
| 212 |
+
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades
|
| 213 |
+
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and
|
| 214 |
+
the vivid colors of its surroundings. A close-up shot with dynamic movement.
|
| 215 |
+
"""
|
| 216 |
+
|
| 217 |
+
output = pipeline(
|
| 218 |
+
prompt=prompt,
|
| 219 |
+
negative_prompt=negative_prompt,
|
| 220 |
+
num_frames=240,
|
| 221 |
+
pyramid_num_inference_steps_list=[2, 2, 2],
|
| 222 |
+
guidance_scale=1.0,
|
| 223 |
+
is_amplify_first_chunk=True,
|
| 224 |
+
generator=torch.Generator("cuda").manual_seed(42),
|
| 225 |
+
).frames[0]
|
| 226 |
+
export_to_video(output, "helios_distilled_t2v_output.mp4", fps=24)
|
| 227 |
+
|
| 228 |
+
# --- I2V ---
|
| 229 |
+
i2v_prompt = """
|
| 230 |
+
A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water,
|
| 231 |
+
illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest,
|
| 232 |
+
casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes
|
| 233 |
+
apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and
|
| 234 |
+
relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and
|
| 235 |
+
respect for nature’s might.
|
| 236 |
+
"""
|
| 237 |
+
image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/wave.jpg"
|
| 238 |
+
|
| 239 |
+
output = pipeline(
|
| 240 |
+
prompt=i2v_prompt,
|
| 241 |
+
negative_prompt=negative_prompt,
|
| 242 |
+
image=load_image(image_path).resize((640, 384)),
|
| 243 |
+
num_frames=240,
|
| 244 |
+
pyramid_num_inference_steps_list=[2, 2, 2],
|
| 245 |
+
guidance_scale=1.0,
|
| 246 |
+
is_amplify_first_chunk=True,
|
| 247 |
+
generator=torch.Generator("cuda").manual_seed(42),
|
| 248 |
+
).frames[0]
|
| 249 |
+
export_to_video(output, "helios_distilled_i2v_output.mp4", fps=24)
|
| 250 |
+
|
| 251 |
+
# --- V2V ---
|
| 252 |
+
v2v_prompt = """
|
| 253 |
+
A bright yellow Lamborghini Huracn Tecnica speeds along a curving mountain road, surrounded by lush green trees
|
| 254 |
+
under a partly cloudy sky. The car's sleek design and vibrant color stand out against the natural backdrop,
|
| 255 |
+
emphasizing its dynamic movement. The road curves gently, with a guardrail visible on one side, adding depth to
|
| 256 |
+
the scene. The motion blur captures the sense of speed and energy, creating a thrilling and exhilarating atmosphere.
|
| 257 |
+
A front-facing shot from a slightly elevated angle, highlighting the car's aggressive stance and the surrounding greenery.
|
| 258 |
+
"""
|
| 259 |
+
video_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/car.mp4"
|
| 260 |
+
|
| 261 |
+
output = pipeline(
|
| 262 |
+
prompt=v2v_prompt,
|
| 263 |
+
negative_prompt=negative_prompt,
|
| 264 |
+
video=load_video(video_path),
|
| 265 |
+
num_frames=240,
|
| 266 |
+
pyramid_num_inference_steps_list=[2, 2, 2],
|
| 267 |
+
guidance_scale=1.0,
|
| 268 |
+
is_amplify_first_chunk=True,
|
| 269 |
+
generator=torch.Generator("cuda").manual_seed(42),
|
| 270 |
+
).frames[0]
|
| 271 |
+
export_to_video(output, "helios_distilled_v2v_output.mp4", fps=24)
|
| 272 |
+
```
|
| 273 |
+
|
| 274 |
+
</details>
|
| 275 |
+
|
| 276 |
+
For example, let's take Helios-Distilled (**Modular Pipeline**).
|
| 277 |
|
| 278 |
<details>
|
| 279 |
<summary>Click to expand the code</summary>
|