Re-assert offline mode inside the ZeroGPU worker itself
Browse filesThe previous fix (1cb4beb) patched huggingface_hub.constants.HF_HUB_OFFLINE
in the main process at startup, verified correct in isolation — but a live
diagnostic (direct HTTP call to /gradio_api/queue/join for MMAudio's
_run_mmaudio, bypassing the browser entirely to reproduce the "Connection
errored out" report) showed open_clip's CLIP loader still re-downloading its
~4GB checkpoint live inside the actual GPU call, confirming the worker
process ZeroGPU dispatches @spaces.GPU calls to doesn't reliably inherit
that patch from the main process.
Fix: re-set both os.environ["HF_HUB_OFFLINE"] and the huggingface_hub
constant at the top of _catch_oom's wrapper, which already wraps all six
@spaces.GPU functions as the innermost decorator — so it executes inside
whatever process actually runs the GPU call, at call time, regardless of
ZeroGPU's fork/spawn timing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@@ -744,9 +744,22 @@ def _catch_oom(fn):
|
|
| 744 |
checkpoint re-downloaded, ~2 minutes) instead of just showing an error.
|
| 745 |
Applied as the innermost decorator (below @spaces.GPU) so spaces.GPU's
|
| 746 |
duration estimator still sees the original call signature via
|
| 747 |
-
functools.wraps.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 748 |
@functools.wraps(fn)
|
| 749 |
def _wrapped(*args, **kwargs):
|
|
|
|
|
|
|
| 750 |
try:
|
| 751 |
return fn(*args, **kwargs)
|
| 752 |
except torch.cuda.OutOfMemoryError as e:
|
|
|
|
| 744 |
checkpoint re-downloaded, ~2 minutes) instead of just showing an error.
|
| 745 |
Applied as the innermost decorator (below @spaces.GPU) so spaces.GPU's
|
| 746 |
duration estimator still sees the original call signature via
|
| 747 |
+
functools.wraps.
|
| 748 |
+
|
| 749 |
+
Also re-asserts offline mode here, at the top of the actual function body.
|
| 750 |
+
ZeroGPU runs @spaces.GPU functions in a separate worker process, and that
|
| 751 |
+
worker does not reliably inherit the main process's post-import os.environ
|
| 752 |
+
/ huggingface_hub.constants mutations made at startup (confirmed live: the
|
| 753 |
+
same os.environ + constants patch that fixed offline resolution when
|
| 754 |
+
tested in-process still didn't stop MMAudio's open_clip CLIP loader from
|
| 755 |
+
re-downloading its ~4GB checkpoint from inside an actual GPU call). Setting
|
| 756 |
+
it again here runs inside whatever process actually executes the GPU
|
| 757 |
+
call, so it can't miss the worker regardless of ZeroGPU's fork/spawn
|
| 758 |
+
timing."""
|
| 759 |
@functools.wraps(fn)
|
| 760 |
def _wrapped(*args, **kwargs):
|
| 761 |
+
os.environ["HF_HUB_OFFLINE"] = "1"
|
| 762 |
+
_hf_constants.HF_HUB_OFFLINE = True
|
| 763 |
try:
|
| 764 |
return fn(*args, **kwargs)
|
| 765 |
except torch.cuda.OutOfMemoryError as e:
|