Spaces:
Running on Zero
Running on Zero
enable triton+CUDA graphs via torch.compile(max-autotune)
Browse filesWarm ZeroGPU: triton kernels + CUDA graphs (1.5-2x). Cold: skip.
4K timing: mask 22.6s + load 5s + infer 28.9s + write 14.5s + stitch 9s = 80s
README.md
CHANGED
|
@@ -44,8 +44,10 @@ Based on [CorridorKey](https://github.com/nikopueringer/CorridorKey) by Corridor
|
|
| 44 |
|
| 45 |
- **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
|
| 46 |
- **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
|
| 47 |
-
- **
|
| 48 |
-
- **Inductor cache persistence**: `TORCHINDUCTOR_CACHE_DIR` persists compiled kernels across ZeroGPU sessions
|
|
|
|
|
|
|
| 49 |
|
| 50 |
## API
|
| 51 |
|
|
|
|
| 44 |
|
| 45 |
- **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
|
| 46 |
- **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
|
| 47 |
+
- **torch.compile(max-autotune)**: on warm ZeroGPU containers, cached triton kernels + CUDA graphs give 1.5-2x speedup — native CUDA kernels, fused operations, no CPU-GPU sync overhead. Cold starts skip compile (37s warmup exceeds 120s budget)
|
| 48 |
+
- **Inductor cache persistence**: `TORCHINDUCTOR_CACHE_DIR` persists compiled triton kernels across ZeroGPU sessions
|
| 49 |
+
|
| 50 |
+
**Pipeline timing** (89 frames, 3840x2160 4K, batch 32 @ 1024): CPU mask precompute 22.6s → GPU schedule+load 5s → inference 28.9s → write 14.5s → stitch+zip 9s = ~80s total (49.2s GPU)
|
| 51 |
|
| 52 |
## API
|
| 53 |
|
app.py
CHANGED
|
@@ -361,15 +361,15 @@ def _load_greenformer(img_size):
|
|
| 361 |
should_compile = True
|
| 362 |
elif os.path.isdir(_inductor_cache) and os.listdir(_inductor_cache):
|
| 363 |
should_compile = True
|
| 364 |
-
logger.info("Warm ZeroGPU: using cached inductor kernels")
|
| 365 |
if should_compile:
|
| 366 |
try:
|
| 367 |
-
compiled = torch.compile(model)
|
| 368 |
dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
|
| 369 |
with torch.inference_mode():
|
| 370 |
compiled(dummy)
|
| 371 |
model = compiled
|
| 372 |
-
logger.info("torch.compile() succeeded")
|
| 373 |
except Exception as e:
|
| 374 |
logger.warning("torch.compile() failed, using eager mode: %s", e)
|
| 375 |
torch.cuda.empty_cache()
|
|
|
|
| 361 |
should_compile = True
|
| 362 |
elif os.path.isdir(_inductor_cache) and os.listdir(_inductor_cache):
|
| 363 |
should_compile = True
|
| 364 |
+
logger.info("Warm ZeroGPU: using cached inductor+triton kernels")
|
| 365 |
if should_compile:
|
| 366 |
try:
|
| 367 |
+
compiled = torch.compile(model, mode="max-autotune")
|
| 368 |
dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
|
| 369 |
with torch.inference_mode():
|
| 370 |
compiled(dummy)
|
| 371 |
model = compiled
|
| 372 |
+
logger.info("torch.compile(max-autotune) succeeded — triton kernels + CUDA graphs active")
|
| 373 |
except Exception as e:
|
| 374 |
logger.warning("torch.compile() failed, using eager mode: %s", e)
|
| 375 |
torch.cuda.empty_cache()
|