Add explicit CUDA availability diagnostic inside GPU worker calls
Browse filesUser correctly pushed back on an unverified claim that the app already had
real GPU access via ZeroGPU regardless of the "Space Hardware" tile
selection. HF's own ZeroGPU docs explicitly say @spaces.GPU is a no-op
outside a Space with "ZeroGPU" hardware selected, and this Space currently
has "CPU upgrade" selected instead — which would perfectly explain the
12+ minute "hang" that was actually just very slow CPU-bound diffusion,
not a deadlock.
This contradicts the earlier-observed CUDA OOM errors and real GPU
allocation during prior testing this session, so rather than guess further,
print hard facts: torch.cuda.is_available(), device name, and device count
from inside an actual @spaces.GPU-decorated call. Next generation attempt
will show definitively whether real GPU access is happening right now.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@@ -258,6 +258,18 @@ def _ensure_offline_in_worker():
|
|
| 258 |
_hf_constants.HF_HUB_OFFLINE = True
|
| 259 |
print(f"[_ensure_offline_in_worker] pid={os.getpid()} "
|
| 260 |
f"HF_HUB_OFFLINE={getattr(_hf_constants, 'HF_HUB_OFFLINE', '?')}")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 261 |
|
| 262 |
# The one CONFIRMED, reproduced failure: MMAudio's open_clip CLIP loader
|
| 263 |
# (create_model_from_pretrained('hf-hub:apple/DFN5B-CLIP-ViT-H-14-384', ...))
|
|
|
|
| 258 |
_hf_constants.HF_HUB_OFFLINE = True
|
| 259 |
print(f"[_ensure_offline_in_worker] pid={os.getpid()} "
|
| 260 |
f"HF_HUB_OFFLINE={getattr(_hf_constants, 'HF_HUB_OFFLINE', '?')}")
|
| 261 |
+
# Settle, empirically, whether this call actually has real GPU access.
|
| 262 |
+
# ZeroGPU docs state @spaces.GPU is a no-op outside a ZeroGPU-hardware
|
| 263 |
+
# Space, silently falling back to CPU (or a CUDA-emulation shim) — which
|
| 264 |
+
# would perfectly explain a "hung" generation that's actually just very
|
| 265 |
+
# slow CPU-bound diffusion. Print hard facts instead of guessing.
|
| 266 |
+
try:
|
| 267 |
+
cuda_ok = torch.cuda.is_available()
|
| 268 |
+
print(f"[_ensure_offline_in_worker] torch.cuda.is_available()={cuda_ok} "
|
| 269 |
+
f"device_name={torch.cuda.get_device_name(0) if cuda_ok else 'N/A'} "
|
| 270 |
+
f"device_count={torch.cuda.device_count() if cuda_ok else 0}")
|
| 271 |
+
except Exception as e:
|
| 272 |
+
print(f"[_ensure_offline_in_worker] CUDA check raised: {e!r}")
|
| 273 |
|
| 274 |
# The one CONFIRMED, reproduced failure: MMAudio's open_clip CLIP loader
|
| 275 |
# (create_model_from_pretrained('hf-hub:apple/DFN5B-CLIP-ViT-H-14-384', ...))
|