Spaces:
Running on Zero
Running on Zero
Commit ·
f27c9e0
1
Parent(s): 4d8e705
Pad declared ZeroGPU duration with a cold-start buffer
Browse filesEvery @spaces.GPU call runs in a fresh worker, so the int8 text_encoder
reload (up to ~30s) and full pipe-to-cuda move (up to ~14s) are paid on
every request. When the UI's gpu_duration slider only budgeted for the
user's expected diffusion time, ZeroGPU would kill the call mid-run with
"GPU task aborted" once the declared duration was exceeded (issue #7).
Since ZeroGPU bills by actual usage rather than the declared duration,
padding it costs nothing but queue priority.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
app.py
CHANGED
|
@@ -613,7 +613,20 @@ def _log_infer_error(e, t0, timer):
|
|
| 613 |
timer.print_timings()
|
| 614 |
|
| 615 |
|
| 616 |
-
@spaces.GPU
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 617 |
def _infer_gpu(pil_images, prompt, seed, guidance_scale, steps, width, height, mode: Mode, gpu_duration=20):
|
| 618 |
_cuda_ok = torch.cuda.is_available()
|
| 619 |
timer = _InferTimer(_cuda_ok)
|
|
|
|
| 613 |
timer.print_timings()
|
| 614 |
|
| 615 |
|
| 616 |
+
# Every @spaces.GPU call lands on a fresh worker (see comment above _FAST_PATH_MIN_GB), so
|
| 617 |
+
# _ensure_int8_text_encoder's reload and _place_pipe_on_device's pipe.to(cuda) are paid on
|
| 618 |
+
# every single request, not just a one-time warmup. Observed worst case: ~30s for the int8
|
| 619 |
+
# text_encoder load (HF hub/disk cache miss) + ~14s to move the pipe onto the device — before
|
| 620 |
+
# any diffusion work even starts. The gpu_duration slider only reflects the user's expectation
|
| 621 |
+
# of diffusion+decode time, so pad the declared duration with this buffer — it doesn't cost
|
| 622 |
+
# extra quota (billing is by real usage, not the declared duration — see
|
| 623 |
+
# huggingface.co/docs/hub/spaces-zerogpu) but declaring too little makes ZeroGPU kill the
|
| 624 |
+
# call mid-run with "GPU task aborted".
|
| 625 |
+
_COLD_START_BUFFER_S = 45
|
| 626 |
+
_MAX_GPU_DURATION_S = 120 # matches the gpu_duration slider's max in the UI
|
| 627 |
+
|
| 628 |
+
|
| 629 |
+
@spaces.GPU(duration=lambda *a, **kw: min(int(a[8]) + _COLD_START_BUFFER_S, _MAX_GPU_DURATION_S) if len(a) > 8 else 60)
|
| 630 |
def _infer_gpu(pil_images, prompt, seed, guidance_scale, steps, width, height, mode: Mode, gpu_duration=20):
|
| 631 |
_cuda_ok = torch.cuda.is_available()
|
| 632 |
timer = _InferTimer(_cuda_ok)
|