Update README.md
#2
by kimhyunwoo - opened
README.md
CHANGED
|
@@ -4,11 +4,55 @@ emoji: 🎬
|
|
| 4 |
colorFrom: red
|
| 5 |
colorTo: purple
|
| 6 |
sdk: gradio
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
sdk_version: 6.17.3
|
| 8 |
app_file: app.py
|
| 9 |
pinned: false
|
| 10 |
license: apache-2.0
|
| 11 |
short_description: Fast Image Edit to Video (IE2V) pipeline.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
# 🎬 IE2V Fast Studio: Edit-Then-Animate Pipeline
|
|
@@ -57,4 +101,24 @@ Wan-AI Team for the foundational Wan 2.2 model weights.
|
|
| 57 |
|
| 58 |
FireRedTeam for the state-of-the-art Qwen-based editing backbone.
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
cbensimon for the pre-compiled WanTransformer3D AoT modules.
|
|
|
|
| 4 |
colorFrom: red
|
| 5 |
colorTo: purple
|
| 6 |
sdk: gradio
|
| 7 |
+
sdk_version: 6.18.0
|
| 8 |
+
app_file: app.py
|
| 9 |
+
pinned: false
|
| 10 |
+
license: apache-2.0
|
| 11 |
+
short_description: Fast Image Edit to Video (IE2V) pipeline.
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# 🎬 IE2V Fast Studio: Edit-Then-Animate Pipeline
|
| 15 |
+
|
| 16 |
+
Welcome to the **IE2V (Image Edit to Video) Fast Studio**. This space bridges the gap between static image manipulation and next-gen video synthesis by combining **FireRed-Image-Edit-1.1** and **Wan 2.2 14B (FP8 Dynamic Quantized & AoT Compiled)** into a unified, high-performance creative sequence.
|
| 17 |
+
|
| 18 |
+
Modify specific regions or styles of an image with precise natural language instructions, then immediately generate seamless, fluid cinematic motion from the modified canvas.
|
| 19 |
+
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
## 🚀 Pipeline Core Architecture
|
| 23 |
+
|
| 24 |
+
### 🛠️ Phase 1: Local Image Canvas Editing (FireRed 1.1)
|
| 25 |
+
- **Granular Control:** Powered by Qwen-Image-Edit-Rapid, enabling complex workflows like object replacement, style changes, or clothing modification (*e.g., "Convert it to a dotted cartoon style"*).
|
| 26 |
+
- **Flash Attention 3 (FA3):** Specialized hardware acceleration for instantaneous multi-reference canvas modifications.
|
| 27 |
+
|
| 28 |
+
### 🎥 Phase 2: Ultra-Fast Motion Generation (Wan 2.2 14B)
|
| 29 |
+
- **Lightning LoRA Integration:** Generates striking, stable video outputs in just **4 to 8 inference steps** instead of the traditional 30+.
|
| 30 |
+
- **FP8 Quantization & Graph Compilation:** Uses `torchao` and `spaces.aoti_load` to minimize VRAM latency, maximizing safety boundaries within ZeroGPU infrastructure.
|
| 31 |
+
|
| 32 |
+
---
|
| 33 |
+
|
| 34 |
+
## 🛠️ Infrastructure & Resource Management
|
| 35 |
+
|
| 36 |
+
To avoid memory fragmentation and `RuntimeError: CUDA out of memory` when switching between models, the application strictly isolates the execution contexts.
|
| 37 |
+
|
| 38 |
+
### VRAM Recycler Pattern (`app.py`)
|
| 39 |
+
Memory is forcefully claimed and flushed back to the OS before and after each inference step:
|
| 40 |
+
---
|
| 41 |
+
title: IE2V Fast Studio (FireRed 1.1 + Wan 2.2 14B)
|
| 42 |
+
emoji: 🎬
|
| 43 |
+
colorFrom: red
|
| 44 |
+
colorTo: purple
|
| 45 |
+
sdk: gradio
|
| 46 |
sdk_version: 6.17.3
|
| 47 |
app_file: app.py
|
| 48 |
pinned: false
|
| 49 |
license: apache-2.0
|
| 50 |
short_description: Fast Image Edit to Video (IE2V) pipeline.
|
| 51 |
+
tags:
|
| 52 |
+
- track:backyard
|
| 53 |
+
- achievement:offgrid
|
| 54 |
+
- achievement:offbrand
|
| 55 |
+
- achievement:fieldnotes
|
| 56 |
---
|
| 57 |
|
| 58 |
# 🎬 IE2V Fast Studio: Edit-Then-Animate Pipeline
|
|
|
|
| 101 |
|
| 102 |
FireRedTeam for the state-of-the-art Qwen-based editing backbone.
|
| 103 |
|
| 104 |
+
cbensimon for the pre-compiled WanTransformer3D AoT modules.
|
| 105 |
+
```python
|
| 106 |
+
def clear_vram():
|
| 107 |
+
import gc
|
| 108 |
+
import torch
|
| 109 |
+
gc.collect()
|
| 110 |
+
torch.cuda.empty_cache()
|
| 111 |
+
|
| 112 |
+
Installation Blueprint
|
| 113 |
+
Ensure your runtime environment satisfies the compiled requirements:
|
| 114 |
+
|
| 115 |
+
pre-requirements.txt: Enforces pip>=23.0.0 for safe wheel building.
|
| 116 |
+
|
| 117 |
+
packages.txt: Must include system-level ffmpeg for video multiplexing.
|
| 118 |
+
|
| 119 |
+
🤝 Credits & Acknowledgments
|
| 120 |
+
Wan-AI Team for the foundational Wan 2.2 model weights.
|
| 121 |
+
|
| 122 |
+
FireRedTeam for the state-of-the-art Qwen-based editing backbone.
|
| 123 |
+
|
| 124 |
cbensimon for the pre-compiled WanTransformer3D AoT modules.
|