Spaces:
Running on Zero
docs: the dtype is load-bearing for ZeroGPU's snapshot cleanup
Browse filesZeroGPU frees the 72 GB HF snapshot after packing by reading
/proc/self/maps, mapping each tensor's data_ptr back to the blob it is
mmap'd from, and unlinking those. So the cleanup is conditional on the
tensors still BEING mmap'd — a dtype cast, a clone, an eager load or a
non-safetensors path materialises them into anonymous memory and there
is nothing left to capture. Then nothing is freed, 72 GB stays on disk
forever, and there is no error and no log line saying so.
torch_dtype has therefore stopped being a free parameter. The comment
sits at the risk site in get_pipeline rather than only in the plan,
because the next person to touch that line is the one who needs it and
they will not be reading §8.3 at the time.
Same class: the cache must stay under ~/.cache/huggingface/hub or the
autoprune pattern stops matching. That is a sharper reason not to move
HF_HOME than the preload one already recorded — it fails silently
rather than loudly.
TEST_PLAN corrections against the platform's own config:
xlarge_threshold is 42.00 GiB exactly, promoted from reported to
verified; duration_factor 1.5 is confirmed and gated on
gpu_size != 'xlarge', so it does not apply to us; and the 2× xlarge
billing multiplier is downgraded to UNVERIFIED because it appears
nowhere in the client. It sized the entire 7-run matrix. Budgeting the
pessimistic figure stays correct — being wrong costs unused quota,
while being wrong the other way costs a reservation failure
mid-matrix — and the first real run's billed seconds are the
measurement.
- docs/PLAN.md +35 -0
- docs/TEST_PLAN.md +20 -6
- ltx25_studio/pipelines.py +13 -0
- ltx25_studio/preflight.py +10 -6
|
@@ -1280,6 +1280,41 @@ normal GPU-box practice:
|
|
| 1280 |
|
| 1281 |
HF's guidance, quoted: **models must be placed on cuda at the root module level.**
|
| 1282 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1283 |
**Corollary: do NOT lazy-load models inside the decorated function.** HF documents that
|
| 1284 |
placements are optimised for startup; lazy loading defeats the packer entirely.
|
| 1285 |
|
|
|
|
| 1280 |
|
| 1281 |
HF's guidance, quoted: **models must be placed on cuda at the root module level.**
|
| 1282 |
|
| 1283 |
+
##### ⚠️ The snapshot is freed after packing — but only if the mmap survives
|
| 1284 |
+
|
| 1285 |
+
Read from `spaces` 0.51.1 (`zero/torch/patching.py:395`) rather than inferred:
|
| 1286 |
+
|
| 1287 |
+
```python
|
| 1288 |
+
with unmap_capture(mmap_addrs) as unmapped_paths:
|
| 1289 |
+
total_size = _pack(Config.zerogpu_offload_dir)
|
| 1290 |
+
for path in unmapped_paths:
|
| 1291 |
+
path.unlink(missing_ok=True)
|
| 1292 |
+
```
|
| 1293 |
+
|
| 1294 |
+
`unmap_capture` reads `/proc/self/maps`, maps each tensor's `data_ptr()` back to
|
| 1295 |
+
**the blob file it is mmap'd from**, and unlinks those matching
|
| 1296 |
+
`~/.cache/huggingface/hub/models--*/blobs/*`.
|
| 1297 |
+
|
| 1298 |
+
**So the 72 GB snapshot IS reclaimed — conditionally.** Two preconditions, both
|
| 1299 |
+
easy to break and both silent when broken:
|
| 1300 |
+
|
| 1301 |
+
1. **Tensors must still be mmap'd from the blobs when packing runs.** A dtype
|
| 1302 |
+
**cast**, a `.clone()`, an eager `low_cpu_mem_usage=False`, or a
|
| 1303 |
+
non-safetensors path materialises them into anonymous memory, leaving nothing
|
| 1304 |
+
in the map to capture. Nothing is freed, 72 GB stays forever, **no error and
|
| 1305 |
+
no log line**. This is why `torch_dtype` must equal the checkpoint's on-disk
|
| 1306 |
+
dtype rather than being a free parameter — recorded at the risk site in
|
| 1307 |
+
`pipelines.get_pipeline`.
|
| 1308 |
+
2. **The cache must live under `~/.cache/huggingface/hub`**, or the autoprune
|
| 1309 |
+
pattern stops matching. **This is the sharper reason not to move `HF_HOME`** —
|
| 1310 |
+
§6.2 gives the preload reason, and this one is worse because it fails
|
| 1311 |
+
silently rather than loudly.
|
| 1312 |
+
|
| 1313 |
+
**And the unlink runs strictly AFTER `_pack()` completes**, so both copies
|
| 1314 |
+
coexist for the whole packing window — which is the 144.37 GB peak in §7.4.
|
| 1315 |
+
Steady state afterwards is fine at ~72 GB. **It is the peak that evicts us, not
|
| 1316 |
+
the residency.**
|
| 1317 |
+
|
| 1318 |
**Corollary: do NOT lazy-load models inside the decorated function.** HF documents that
|
| 1319 |
placements are optimised for startup; lazy loading defeats the packer entirely.
|
| 1320 |
|
|
@@ -10,7 +10,7 @@ result from one does not answer the other.**
|
|
| 10 |
| RAM | 124 GB | ~68 GB (community observation) |
|
| 11 |
| disk | 136 GB free, persistent | 150 GB cap, **wiped on cold start** |
|
| 12 |
| tensor packing | none | O_DIRECT to `~/.zerogpu/tensors` |
|
| 13 |
-
| budget | time only | **2400 GPU-s/day**
|
| 14 |
| status | **critical path** | **parked** |
|
| 15 |
|
| 16 |
**Label every claim.** A mode that works on hinode is **HOST-PROVEN**; whether it
|
|
@@ -127,8 +127,8 @@ its economics do not apply to hinode and must not leak into Part A.
|
|
| 127 |
| | |
|
| 128 |
|---|---|
|
| 129 |
| PRO quota | **2400 GPU-seconds/day**, fixed 24 h window starting at first use |
|
| 130 |
-
| Our tier | **`xlarge`** — forced, because packed weights are 67.23 GiB and `xlarge_threshold` is 42.00 GiB (
|
| 131 |
-
| Multiplier | **2×**. A 75 s run
|
| 132 |
| Practical ceiling | **~16 runs/day** |
|
| 133 |
| **HARD STOP** | **1920 GPU-seconds (80%)** |
|
| 134 |
|
|
@@ -136,6 +136,19 @@ its economics do not apply to hinode and must not leak into Part A.
|
|
| 136 |
be fixed and retested on the same day. A test budget with no reserve is a test budget that
|
| 137 |
guarantees the last bug found is the one you cannot verify fixed.
|
| 138 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
### B.1.1 Two metering paths — they are not the same number
|
| 140 |
|
| 141 |
This distinction decides how we set `duration=`:
|
|
@@ -149,9 +162,10 @@ So: **keep `duration=` tight.** An over-generous request costs nothing in billin
|
|
| 149 |
in schedulability — late in the day, a `duration=300` reservation fails against 200 s remaining
|
| 150 |
while the actual 40 s run would have fit fine.
|
| 151 |
|
| 152 |
-
|
| 153 |
-
`duration=60` schedules as 90 s)
|
| 154 |
-
|
|
|
|
| 155 |
discover it, never hardcode it.
|
| 156 |
|
| 157 |
---
|
|
|
|
| 10 |
| RAM | 124 GB | ~68 GB (community observation) |
|
| 11 |
| disk | 136 GB free, persistent | 150 GB cap, **wiped on cold start** |
|
| 12 |
| tensor packing | none | O_DIRECT to `~/.zerogpu/tensors` |
|
| 13 |
+
| budget | time only | **2400 GPU-s/day**; an xlarge multiplier is *assumed*, see §B.1 |
|
| 14 |
| status | **critical path** | **parked** |
|
| 15 |
|
| 16 |
**Label every claim.** A mode that works on hinode is **HOST-PROVEN**; whether it
|
|
|
|
| 127 |
| | |
|
| 128 |
|---|---|
|
| 129 |
| PRO quota | **2400 GPU-seconds/day**, fixed 24 h window starting at first use |
|
| 130 |
+
| Our tier | **`xlarge`** — forced, because packed weights are 67.23 GiB and `xlarge_threshold` is **42.00 GiB exactly** (`45097156608` bytes, **verified in the platform's own `spaces/zero/configs.json`**) |
|
| 131 |
+
| Multiplier | **assumed 2×** — ⚠️ **UNVERIFIED.** A 75 s run is budgeted at 150 GPU-seconds |
|
| 132 |
| Practical ceiling | **~16 runs/day** |
|
| 133 |
| **HARD STOP** | **1920 GPU-seconds (80%)** |
|
| 134 |
|
|
|
|
| 136 |
be fixed and retested on the same day. A test budget with no reserve is a test budget that
|
| 137 |
guarantees the last bug found is the one you cannot verify fixed.
|
| 138 |
|
| 139 |
+
> ⚠️ **The 2× multiplier is unverified, and it sized this entire matrix.**
|
| 140 |
+
> Reading the `spaces` 0.51.1 client turned up `duration_factor = 1.5` applied at
|
| 141 |
+
> `client.py:112` as `if gpu_size != 'xlarge': duration_seconds *= duration_factor` — so the
|
| 142 |
+
> **1.5× inflation applies to `large` and NOT to `xlarge`, which is what we get** — but **no 2×
|
| 143 |
+
> xlarge billing multiplier appears anywhere in the client.** It may well be server-side, which
|
| 144 |
+
> is why this is downgraded rather than deleted.
|
| 145 |
+
>
|
| 146 |
+
> Consequence if it does not exist: the 7-run budget of 930 GPU-seconds is really ~465, and the
|
| 147 |
+
> matrix is half as expensive as planned. **Budgeting the pessimistic number is correct** — being
|
| 148 |
+
> wrong about it costs unused quota, while being wrong the other way costs a reservation failure
|
| 149 |
+
> mid-matrix. Do not re-plan around the cheaper figure until a real run's *billed* seconds
|
| 150 |
+
> confirm it. **The first run is the measurement**: compare billed against requested.
|
| 151 |
+
|
| 152 |
### B.1.1 Two metering paths — they are not the same number
|
| 153 |
|
| 154 |
This distinction decides how we set `duration=`:
|
|
|
|
| 162 |
in schedulability — late in the day, a `duration=300` reservation fails against 200 s remaining
|
| 163 |
while the actual 40 s run would have fit fine.
|
| 164 |
|
| 165 |
+
One trap from PLAN §5.8, now **confirmed at source**: `large` applies a hidden
|
| 166 |
+
**1.5× `duration_factor`** (so `duration=60` schedules as 90 s). `spaces/zero/client.py:112`
|
| 167 |
+
gates it on `gpu_size != 'xlarge'`, so it does **not** apply to us. The claim that `xlarge`
|
| 168 |
+
"doubles the request instead" is the unverified half — see the warning above. And the maximum duration is a **server-side, tier-dependent, unpublished cap** —
|
| 169 |
discover it, never hardcode it.
|
| 170 |
|
| 171 |
---
|
|
@@ -207,6 +207,19 @@ def _base_pipeline(*, dtype=None, device: str | None = None):
|
|
| 207 |
) from exc
|
| 208 |
|
| 209 |
try:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
built = LTX2ConditionPipeline.from_pretrained(
|
| 211 |
model_files.REPO_ID,
|
| 212 |
revision=model_files.REVISION,
|
|
|
|
| 207 |
) from exc
|
| 208 |
|
| 209 |
try:
|
| 210 |
+
# ⚠️ `torch_dtype` MUST match the checkpoint's on-disk dtype. It is not a
|
| 211 |
+
# free parameter, and the reason is invisible from here.
|
| 212 |
+
#
|
| 213 |
+
# ZeroGPU frees the 72 GB HF snapshot after packing, but only by reading
|
| 214 |
+
# /proc/self/maps, mapping each tensor's data_ptr() back to the blob it is
|
| 215 |
+
# mmap'd from, and unlinking those. Anything that materialises tensors
|
| 216 |
+
# into anonymous memory -- a dtype CAST, a .clone(), an eager
|
| 217 |
+
# low_cpu_mem_usage=False, a non-safetensors path -- leaves nothing in the
|
| 218 |
+
# map to capture. Then nothing is freed, 72 GB stays on disk forever, and
|
| 219 |
+
# there is no error and no log line saying so.
|
| 220 |
+
#
|
| 221 |
+
# The checkpoint is bf16 and WEIGHT_DTYPE is bfloat16, so this is a
|
| 222 |
+
# no-op load rather than a conversion. Do not "helpfully" add a cast.
|
| 223 |
built = LTX2ConditionPipeline.from_pretrained(
|
| 224 |
model_files.REPO_ID,
|
| 225 |
revision=model_files.REVISION,
|
|
@@ -15,10 +15,13 @@ byte is touched:
|
|
| 15 |
that failure cannot happen -- so we check the invariant instead of hoping for
|
| 16 |
it, and a miss names the file AND its URL.
|
| 17 |
|
| 18 |
-
Note we deliberately do NOT move ``HF_HOME``
|
| 19 |
-
``preload_from_hub`` writes to the default
|
| 20 |
-
``HF_HOME``, so relocating it makes the app
|
| 21 |
-
re-download 72 GB
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
3. **Resource preflight.** Measure free RAM and free disk against the
|
| 24 |
requirement computed from the resolved config, and raise an error that NAMES
|
|
@@ -87,8 +90,9 @@ def ensure_weights(
|
|
| 87 |
log = logging.getLogger("ltx25_studio")
|
| 88 |
paths = model_files.download_allow_list(components)
|
| 89 |
total_gb = model_files.total_size_gb(components)
|
| 90 |
-
log.info(
|
| 91 |
-
|
|
|
|
| 92 |
fetched: list[Path] = []
|
| 93 |
for index, path in enumerate(paths, 1):
|
| 94 |
# Progress is not decoration. This is a 72 GB cold start on every restart,
|
|
|
|
| 15 |
that failure cannot happen -- so we check the invariant instead of hoping for
|
| 16 |
it, and a miss names the file AND its URL.
|
| 17 |
|
| 18 |
+
Note we deliberately do NOT move ``HF_HOME``, and there are two reasons --
|
| 19 |
+
the second sharper than the first. ``preload_from_hub`` writes to the default
|
| 20 |
+
cache and does not follow a moved ``HF_HOME``, so relocating it makes the app
|
| 21 |
+
look somewhere empty and re-download 72 GB. **And ZeroGPU's post-pack cleanup
|
| 22 |
+
unlinks blobs matching** ``~/.cache/huggingface/hub/models--*/blobs/*``
|
| 23 |
+
**specifically** -- move the cache out from under that pattern and the 72 GB
|
| 24 |
+
snapshot is never freed, silently (PLAN §6.2, §8.3).
|
| 25 |
|
| 26 |
3. **Resource preflight.** Measure free RAM and free disk against the
|
| 27 |
requirement computed from the resolved config, and raise an error that NAMES
|
|
|
|
| 90 |
log = logging.getLogger("ltx25_studio")
|
| 91 |
paths = model_files.download_allow_list(components)
|
| 92 |
total_gb = model_files.total_size_gb(components)
|
| 93 |
+
log.info(
|
| 94 |
+
"fetching %d files (%.2f GB) at revision %s", len(paths), total_gb, model_files.REVISION
|
| 95 |
+
)
|
| 96 |
fetched: list[Path] = []
|
| 97 |
for index, path in enumerate(paths, 1):
|
| 98 |
# Progress is not decoration. This is a 72 GB cold start on every restart,
|