Spaces:
Running on Zero
Running on Zero
disable torch.compile on ZeroGPU — net negative for GreenFormer
Browse filesTested on H200:
- max-autotune: 118s, 0 triton kernels (cublas wins for small matrices)
- reduce-overhead: 36s compile + 48s graph recording = 84s for 5% speedup
- eager: 0.32s/frame with zero overhead — best for 120s budget
torch.compile still enabled for local GPU (no time budget).
README.md
CHANGED
|
@@ -44,8 +44,7 @@ Based on [CorridorKey](https://github.com/nikopueringer/CorridorKey) by Corridor
|
|
| 44 |
|
| 45 |
- **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
|
| 46 |
- **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
|
| 47 |
-
- **torch.compile
|
| 48 |
-
- **Inductor cache persistence**: `TORCHINDUCTOR_CACHE_DIR` persists compiled triton kernels across ZeroGPU sessions
|
| 49 |
|
| 50 |
**Pipeline timing** (89 frames, batch 32 @ 1024px model res): CPU mask 22s → GPU load 5s → inference 29s → write 15s → stitch 9s ≈ 80s total, 49s GPU. Model always processes at 1024x1024 or 2048x2048 regardless of input resolution
|
| 51 |
|
|
|
|
| 44 |
|
| 45 |
- **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
|
| 46 |
- **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
|
| 47 |
+
- **torch.compile on local GPU only**: tested max-autotune (118s, 0 triton kernels) and reduce-overhead (36s compile + 48s CUDA graph recording = 84s overhead, 5% speedup). GreenFormer's small feature maps (112-896ch) are cublas-optimal, not triton-friendly. Disabled on ZeroGPU — eager mode at 0.32s/frame is better than 84s+ overhead
|
|
|
|
| 48 |
|
| 49 |
**Pipeline timing** (89 frames, batch 32 @ 1024px model res): CPU mask 22s → GPU load 5s → inference 29s → write 15s → stitch 9s ≈ 80s total, 49s GPU. Model always processes at 1024x1024 or 2048x2048 regardless of input resolution
|
| 50 |
|
app.py
CHANGED
|
@@ -365,22 +365,26 @@ def _load_greenformer(img_size):
|
|
| 365 |
torch.backends.cuda.mem_efficient_sdp_enabled(),
|
| 366 |
torch.backends.cuda.math_sdp_enabled())
|
| 367 |
|
| 368 |
-
# torch.compile
|
| 369 |
-
# max-autotune
|
| 370 |
-
|
|
|
|
|
|
|
|
|
|
| 371 |
t_compile = time.time()
|
| 372 |
try:
|
| 373 |
-
compiled = torch.compile(model
|
| 374 |
dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
|
| 375 |
with torch.inference_mode():
|
| 376 |
compiled(dummy)
|
| 377 |
model = compiled
|
| 378 |
-
logger.info("torch.compile(
|
| 379 |
time.time() - t_compile)
|
| 380 |
except Exception as e:
|
| 381 |
-
logger.warning("torch.compile() failed
|
| 382 |
-
time.time() - t_compile, e)
|
| 383 |
torch.cuda.empty_cache()
|
|
|
|
|
|
|
| 384 |
|
| 385 |
logger.info("GreenFormer loaded on CUDA (img_size=%d)", img_size)
|
| 386 |
return model
|
|
|
|
| 365 |
torch.backends.cuda.mem_efficient_sdp_enabled(),
|
| 366 |
torch.backends.cuda.math_sdp_enabled())
|
| 367 |
|
| 368 |
+
# torch.compile DISABLED on ZeroGPU: tested all modes on H200, all are net-negative.
|
| 369 |
+
# - max-autotune: 118s compile, 0 triton kernels generated (small matrices, cublas optimal)
|
| 370 |
+
# - reduce-overhead: 36s compile + 48s CUDA graph recording = 84s overhead for 5% speedup
|
| 371 |
+
# - GreenFormer's Hiera attention has small feature maps (112-896ch), not triton-friendly
|
| 372 |
+
# Eager mode: 0.32s/frame with no overhead — best for 120s ZeroGPU budget
|
| 373 |
+
if not HAS_SPACES and sys.platform in ("linux", "win32"):
|
| 374 |
t_compile = time.time()
|
| 375 |
try:
|
| 376 |
+
compiled = torch.compile(model)
|
| 377 |
dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
|
| 378 |
with torch.inference_mode():
|
| 379 |
compiled(dummy)
|
| 380 |
model = compiled
|
| 381 |
+
logger.info("torch.compile() in %.1fs (local GPU, no time budget)",
|
| 382 |
time.time() - t_compile)
|
| 383 |
except Exception as e:
|
| 384 |
+
logger.warning("torch.compile() failed: %s", e)
|
|
|
|
| 385 |
torch.cuda.empty_cache()
|
| 386 |
+
elif HAS_SPACES:
|
| 387 |
+
logger.info("Eager mode (torch.compile costs 84s+ on H200, not worth 120s budget)")
|
| 388 |
|
| 389 |
logger.info("GreenFormer loaded on CUDA (img_size=%d)", img_size)
|
| 390 |
return model
|