Nekochu commited on
Commit
f1d6e7f
·
1 Parent(s): cdef1d9

enable triton+CUDA graphs via torch.compile(max-autotune)

Browse files

Warm ZeroGPU: triton kernels + CUDA graphs (1.5-2x). Cold: skip.
4K timing: mask 22.6s + load 5s + infer 28.9s + write 14.5s + stitch 9s = 80s

Files changed (2) hide show
  1. README.md +4 -2
  2. app.py +3 -3
README.md CHANGED
@@ -44,8 +44,10 @@ Based on [CorridorKey](https://github.com/nikopueringer/CorridorKey) by Corridor
44
 
45
  - **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
46
  - **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
47
- - **Conditional torch.compile**: on warm ZeroGPU containers, cached torch.inductor kernels enable compiled inference (~0s warmup, 1.3-1.5x speedup). Cold starts skip compile (37s warmup exceeds 120s GPU budget). torch.inductor compiles to native CUDA kernels with fused operations and reduced kernel launches
48
- - **Inductor cache persistence**: `TORCHINDUCTOR_CACHE_DIR` persists compiled kernels across ZeroGPU sessions
 
 
49
 
50
  ## API
51
 
 
44
 
45
  - **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch
46
  - **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops
47
+ - **torch.compile(max-autotune)**: on warm ZeroGPU containers, cached triton kernels + CUDA graphs give 1.5-2x speedup — native CUDA kernels, fused operations, no CPU-GPU sync overhead. Cold starts skip compile (37s warmup exceeds 120s budget)
48
+ - **Inductor cache persistence**: `TORCHINDUCTOR_CACHE_DIR` persists compiled triton kernels across ZeroGPU sessions
49
+
50
+ **Pipeline timing** (89 frames, 3840x2160 4K, batch 32 @ 1024): CPU mask precompute 22.6s → GPU schedule+load 5s → inference 28.9s → write 14.5s → stitch+zip 9s = ~80s total (49.2s GPU)
51
 
52
  ## API
53
 
app.py CHANGED
@@ -361,15 +361,15 @@ def _load_greenformer(img_size):
361
  should_compile = True
362
  elif os.path.isdir(_inductor_cache) and os.listdir(_inductor_cache):
363
  should_compile = True
364
- logger.info("Warm ZeroGPU: using cached inductor kernels")
365
  if should_compile:
366
  try:
367
- compiled = torch.compile(model)
368
  dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
369
  with torch.inference_mode():
370
  compiled(dummy)
371
  model = compiled
372
- logger.info("torch.compile() succeeded")
373
  except Exception as e:
374
  logger.warning("torch.compile() failed, using eager mode: %s", e)
375
  torch.cuda.empty_cache()
 
361
  should_compile = True
362
  elif os.path.isdir(_inductor_cache) and os.listdir(_inductor_cache):
363
  should_compile = True
364
+ logger.info("Warm ZeroGPU: using cached inductor+triton kernels")
365
  if should_compile:
366
  try:
367
+ compiled = torch.compile(model, mode="max-autotune")
368
  dummy = torch.zeros(1, 4, img_size, img_size, dtype=torch.float16, device="cuda")
369
  with torch.inference_mode():
370
  compiled(dummy)
371
  model = compiled
372
+ logger.info("torch.compile(max-autotune) succeeded — triton kernels + CUDA graphs active")
373
  except Exception as e:
374
  logger.warning("torch.compile() failed, using eager mode: %s", e)
375
  torch.cuda.empty_cache()