Nekochu commited on
Commit
f4a2965
·
1 Parent(s): c53eb28

disable torch.compile on ZeroGPU — net negative for GreenFormer

Browse files

Tested on H200:
- max-autotune: 118s, 0 triton kernels (cublas wins for small matrices)
- reduce-overhead: 36s compile + 48s graph recording = 84s for 5% speedup
- eager: 0.32s/frame with zero overhead — best for 120s budget
torch.compile still enabled for local GPU (no time budget).

Files changed (2) hide show
  1. README.md +1 -2
  2. app.py +11 -7
README.md CHANGED
@@ -44,8 +44,7 @@ Based on [CorridorKey](https://github.com/nikopueringer/CorridorKey) by Corridor
44
 
45
  - **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
46
  - **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
47
- - **torch.compile(max-autotune)**: on warm ZeroGPU containers, cached triton kernels + CUDA graphs give 1.5-2x speedup — native CUDA kernels, fused operations, no CPU-GPU sync overhead. Cold starts skip compile (37s warmup exceeds 120s budget)
48
- - **Inductor cache persistence**: `TORCHINDUCTOR_CACHE_DIR` persists compiled triton kernels across ZeroGPU sessions
49
 
50
  **Pipeline timing** (89 frames, batch 32 @ 1024px model res): CPU mask 22s → GPU load 5s → inference 29s → write 15s → stitch 9s ≈ 80s total, 49s GPU. Model always processes at 1024x1024 or 2048x2048 regardless of input resolution
51
 
 
44
 
45
  - **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
46
  - **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
47
+ - **torch.compile on local GPU only**: tested max-autotune (118s, 0 triton kernels) and reduce-overhead (36s compile + 48s CUDA graph recording = 84s overhead, 5% speedup). GreenFormer's small feature maps (112-896ch) are cublas-optimal, not triton-friendly. Disabled on ZeroGPU — eager mode at 0.32s/frame is better than 84s+ overhead
 
48
 
49
  **Pipeline timing** (89 frames, batch 32 @ 1024px model res): CPU mask 22s → GPU load 5s → inference 29s → write 15s → stitch 9s ≈ 80s total, 49s GPU. Model always processes at 1024x1024 or 2048x2048 regardless of input resolution
50
 
app.py CHANGED
@@ -365,22 +365,26 @@ def _load_greenformer(img_size):
365
  torch.backends.cuda.mem_efficient_sdp_enabled(),
366
  torch.backends.cuda.math_sdp_enabled())
367
 
368
- # torch.compile: reduce-overhead = CUDA graphs (fast compile, ~30s).
369
- # max-autotune was 118s on H200 and generated 0 triton kernels — not worth it.
370
- if sys.platform in ("linux", "win32"):
 
 
 
371
  t_compile = time.time()
372
  try:
373
- compiled = torch.compile(model, mode="reduce-overhead")
374
  dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
375
  with torch.inference_mode():
376
  compiled(dummy)
377
  model = compiled
378
- logger.info("torch.compile(reduce-overhead) in %.1fs — CUDA graphs active",
379
  time.time() - t_compile)
380
  except Exception as e:
381
- logger.warning("torch.compile() failed (%.1fs), eager mode: %s",
382
- time.time() - t_compile, e)
383
  torch.cuda.empty_cache()
 
 
384
 
385
  logger.info("GreenFormer loaded on CUDA (img_size=%d)", img_size)
386
  return model
 
365
  torch.backends.cuda.mem_efficient_sdp_enabled(),
366
  torch.backends.cuda.math_sdp_enabled())
367
 
368
+ # torch.compile DISABLED on ZeroGPU: tested all modes on H200, all are net-negative.
369
+ # - max-autotune: 118s compile, 0 triton kernels generated (small matrices, cublas optimal)
370
+ # - reduce-overhead: 36s compile + 48s CUDA graph recording = 84s overhead for 5% speedup
371
+ # - GreenFormer's Hiera attention has small feature maps (112-896ch), not triton-friendly
372
+ # Eager mode: 0.32s/frame with no overhead — best for 120s ZeroGPU budget
373
+ if not HAS_SPACES and sys.platform in ("linux", "win32"):
374
  t_compile = time.time()
375
  try:
376
+ compiled = torch.compile(model)
377
  dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
378
  with torch.inference_mode():
379
  compiled(dummy)
380
  model = compiled
381
+ logger.info("torch.compile() in %.1fs (local GPU, no time budget)",
382
  time.time() - t_compile)
383
  except Exception as e:
384
+ logger.warning("torch.compile() failed: %s", e)
 
385
  torch.cuda.empty_cache()
386
+ elif HAS_SPACES:
387
+ logger.info("Eager mode (torch.compile costs 84s+ on H200, not worth 120s budget)")
388
 
389
  logger.info("GreenFormer loaded on CUDA (img_size=%d)", img_size)
390
  return model