Fix HF_HUB_OFFLINE never actually taking effect
Browse filesRoot-caused the "connection errored out" MMAudio failure via server logs:
_mmaudio_gpu_infer started fine, then open_clip's create_model_from_pretrained
('hf-hub:apple/DFN5B-CLIP-ViT-H-14-384', ...) re-downloaded the full ~4GB
checkpoint mid-GPU-call instead of using the encoder cache pre-populated at
startup, eating enough time/bandwidth inside the billed GPU call to drop the
connection.
Root cause: os.environ["HF_HUB_OFFLINE"] = "1" was set *after* huggingface_hub
was already imported (line ~32). huggingface_hub.constants.HF_HUB_OFFLINE is
computed once from os.environ at that import, so setting the env var later
never actually enables offline mode for the rest of the process — every
from_pretrained()/hf-hub: call was silently running online the whole time.
With the mirror-cache trick (_populate_hf_cache_from_mirror) writing a fake
'mirror-snapshot' commit hash rather than the real one, an online resolution
always treats it as a cache miss and re-downloads from the real upstream repo.
Verified against the real huggingface_hub package with an isolated two-repo
fixture mimicking _populate_hf_cache_from_mirror exactly: without patching the
already-imported constant, resolution ignores the mirrored cache and re-fetches
over the network every time; patching huggingface_hub.constants.HF_HUB_OFFLINE
directly (not just os.environ) makes it correctly resolve from the local
mirror with zero network calls. This is the single choke point shared by
transformers/diffusers/open_clip's hf-hub loading, so it should also protect
CLAP/AudioLDM2/SigLIP2 from the same failure mode, not just MMAudio's CLIP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@@ -234,7 +234,23 @@ print(f"[startup] All downloads done in {time.perf_counter() - _t_dl_start:.1f}s
|
|
| 234 |
# calls (CLAP, AudioLDM2, MMAudio's open_clip 'hf-hub:...') resolve from
|
| 235 |
# the populated cache instead of networking out to upstream — full
|
| 236 |
# upstream-protection mode.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 237 |
os.environ["HF_HUB_OFFLINE"] = "1"
|
|
|
|
|
|
|
| 238 |
|
| 239 |
# ================================================================== #
|
| 240 |
# SHARED CONSTANTS / HELPERS #
|
|
|
|
| 234 |
# calls (CLAP, AudioLDM2, MMAudio's open_clip 'hf-hub:...') resolve from
|
| 235 |
# the populated cache instead of networking out to upstream — full
|
| 236 |
# upstream-protection mode.
|
| 237 |
+
#
|
| 238 |
+
# Setting only the env var is NOT enough: huggingface_hub.constants.HF_HUB_OFFLINE
|
| 239 |
+
# is computed once from os.environ at *import time* (constants.py), and
|
| 240 |
+
# huggingface_hub was already imported above (line ~32) before this point, so
|
| 241 |
+
# the env var alone never takes effect for the rest of this process. Confirmed
|
| 242 |
+
# live: MMAudio's open_clip CLIP loader (create_model_from_pretrained('hf-hub:
|
| 243 |
+
# apple/DFN5B-CLIP-ViT-H-14-384', ...)) still resolved "main" over the network,
|
| 244 |
+
# didn't match our locally mirrored 'mirror-snapshot' cache entry's fake commit
|
| 245 |
+
# hash, and re-downloaded the full ~4GB checkpoint mid-GPU-call — verified via
|
| 246 |
+
# an isolated repro against the real huggingface_hub package (see scratchpad).
|
| 247 |
+
# Patching the already-imported module's constant directly is what actually
|
| 248 |
+
# gates the HTTP layer (huggingface_hub/utils/_http.py reads it live via
|
| 249 |
+
# constants.is_offline_mode()), which is shared by transformers/diffusers/
|
| 250 |
+
# open_clip alike since they all delegate the actual network call to it.
|
| 251 |
os.environ["HF_HUB_OFFLINE"] = "1"
|
| 252 |
+
import huggingface_hub.constants as _hf_constants
|
| 253 |
+
_hf_constants.HF_HUB_OFFLINE = True
|
| 254 |
|
| 255 |
# ================================================================== #
|
| 256 |
# SHARED CONSTANTS / HELPERS #
|