Spaces:
Running on Zero
Running on Zero
docs: every sentence stays true after the deploy: README and the release notes say this is deploy 2's code and point to issue #32 for what the Space shows, and no doc says the Space still runs deploy 1, is not deployed, or has not been redeployed since August (#32)
abc9150 unverified |
Download docs/SPACE_GENERATION_FINDING.md from techfreakworm/LTX2.5-Studio: direct link, hf CLI and curl.
- Browser
- Download file 39.3 kB
-
https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio/resolve/main/docs/SPACE_GENERATION_FINDING.md
- Command line
-
hf download hf://spaces/techfreakworm/LTX2.5-Studio/docs/SPACE_GENERATION_FINDING.md
-
curl -L -o SPACE_GENERATION_FINDING.md https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio/resolve/main/docs/SPACE_GENERATION_FINDING.md
39.3 kB
| # The Space: a killed worker, a falsified ceiling, and a working generation | |
| > ## Status on 2026-11-16 | |
| > | |
| > This is the August 2026 record of the first Space runs; its body is kept as written. | |
| > Superseded since: | |
| > | |
| > - **Β§2 annotation "`LTX25_TRANSFORMER_PRECISION=bf16` still PRESENT"**: the Space's variables, | |
| > read on 2026-11-16, no longer include it (nor `LTX25_TEXT_ENCODER_PRECISION`). | |
| > - **Β§2 "`spaces` 0.51.1"**: the Space reports `spaces` 0.51.3. | |
| > - **Β§5's startup report** (`zerogpu_size=auto duration=60s`, `dtype : bf16 (fixed; no | |
| > precision switch)`, `82.2 GB β¦ + image ~10`) is that day's format. The Space set `xlarge` | |
| > and 180 s through its variables, and the report was rewritten on 2026-11-16; the new one ran | |
| > on deploy 1 (`SPACE_MODE_MATRIX.md` Β§10.1). Since deploy 2's code the reservation follows | |
| > each run's size and length, and `LTX25_GPU_DURATION_SECONDS` is ignored. | |
| > - **Β§11 #4 "packed 51.1 GB β¦ `xlarge` bills at 2Γ"**: on the 2026-11-13 boot the platform's own | |
| > progress bar read `40.6G` packed, and the run in this file was on `large`, not `xlarge`. | |
| > - **Β§11 #1 "our error text tells users to read a traceback that does not exist"**: that sentence | |
| > is no longer in the UI. | |
| > | |
| > Still true: no steady-state RSS has been measured (Β§11 #6), and billed seconds cannot be read | |
| > back (Β§11 #4). Where the current truth is: [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) Β§10 | |
| > and [`../README.md`](../README.md). Deploy 1 (2026-11-16) ran all eleven modes on the Space | |
| > ([`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) Β§10.1); deploy 2's own check on the Space is | |
| > reported on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32). | |
| > ## β INVALIDATED (2026-08-25, evening) β "a real video" was COLOUR STATIC | |
| > | |
| > The 39,146-byte artifact this document resolves on, and every Space artifact | |
| > after it up to sha `ff142d9`, was produced by a transformer whose | |
| > **conditioning path had been quantised away**. The blanket int8 config below | |
| > named no `llm_int8_skip_modules`, so bitsandbytes replaced every `nn.Linear` | |
| > including the eight `LTX2AdaLayerNormSingle` modulation heads and the in/out | |
| > projections. "Content-verified" meant non-empty and non-frozen; nobody opened | |
| > a frame until the operator did, on 2026-08-25. | |
| > | |
| > β **Fixed at `eb1aa9f`** (316 of 1660 Linears spared, names read off the real | |
| > checkpoint), deployed as Space sha `e3b5333`, verified by eye. **Every | |
| > mechanical measurement in this document stands** β RSS, disk, packing, the | |
| > killed workers, the falsified ceiling, the fetch path, the fork crossing. | |
| > Only the output claim falls. Full record: the invalidation block at the top of | |
| > [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md). | |
| > | |
| > β **Β§9.3's "do not let any document claim the widening fixed generation" is | |
| > VINDICATED** in the strongest direction available: the widening is what broke | |
| > it. | |
| > | |
| > β **One hypothesis was investigated and REFUTED β do not re-chase it.** That | |
| > `LLM.int8()` computes matmuls in **fp16** while LTX is bf16-native, evidenced | |
| > by bitsandbytes' own bf16βfp16 cast warning in the Space log, **is not the | |
| > mechanism.** `llm_int8_threshold` sits at its **6.0 default**, which is exactly | |
| > the gate that routes fp16-overflowing columns through the bf16 outlier path, | |
| > and inside fp16's range a bf16βfp16 cast is exact. β The skip list alone fixed | |
| > it with `llm_int8_threshold` untouched β one variable, cleanly attributed. The | |
| > warning is real; it was a red herring. | |
| > ## β RESOLVED β the Space generates. **Read Β§9.3 before citing why.** | |
| > | |
| > A real **39,146-byte** video, **content-verified**, at sha `8fb292d6`. | |
| > | |
| > β οΈ **The fix is NOT attributed.** Two things changed between the failing and | |
| > succeeding runs, not one. **"The Space now serves" is established; "the | |
| > widening is why" is not.** Β§9.3. | |
| > | |
| > The investigation below is kept exactly as written. Its measurements stand, and | |
| > **two of its central claims were falsified** by the run that resolved it β | |
| > including one this build had been planning against since before any deploy. | |
| > **Original framing, first attempt β accurate as history:** | |
| > **First generation ever attempted on the live private Space. Startup succeeded; | |
| > the generation failed.** The forked ZeroGPU worker was **killed twice, with no | |
| > Python traceback**, and `spaces` returned `'GPU task aborted'`. | |
| > | |
| > This file is the only place these measurements exist. | |
| **Every load-bearing number is marked:** β *we measured it* Β· π *read off a | |
| pinned source or the platform* Β· β οΈ *inferred* Β· βΈοΈ *not established*. | |
| > ### β οΈ The rule this document is organised around | |
| > | |
| > **What we measured and what we inherited are kept apart.** The RAM ceiling has | |
| > two numbers attached to it and only one is ours. Blurring them is how a | |
| > borrowed figure becomes a fact, and this build has spent a day undoing exactly | |
| > that. | |
| --- | |
| ## 1. What happened | |
| | | | | |
| |---|---| | |
| | Space | `techfreakworm/LTX2.5-Studio`, **private**, stage RUNNING π | | |
| | Space SHA | `a1791c9e856f2b96ab10d89a05a84a0899c8dca5` π | | |
| | `spaces` | 0.51.1 π | | |
| | Hardware id | `zero-a10g` π β **a legacy id string, not an actual A10G** | | |
| | Account | PRO β (`whoami-v2` β `isPro: true`), 2400 GPU-s/day | | |
| | Client | `gradio_client` 2.6.0 β `/on_generate` | | |
| **Startup: healthy.** **Generation: failed.** Those are two different claims and | |
| the rest of this file exists to keep them apart. | |
| > β **Superseded on the retry at sha `8fb292d6`: generation now succeeds.** The | |
| > separation above is what made that legible β startup never broke, so the | |
| > failure was always in the second claim. Β§9.3. | |
| --- | |
| ## 2. The request | |
| Chosen to be the cheapest thing that exercises the path β 256 Γ 256 Γ 9 frames, | |
| no conditioning, no stage 2. | |
| ``` | |
| mode_id = text_to_video | |
| prompt = 'a red balloon rising over a calm lake at dawn' | |
| negative_prompt = '' | |
| image/video/audio = None | |
| conditions = (empty) | |
| width, height = 256, 256 | |
| num_frames = 9 | |
| auto_duration = False min_seconds = 2.0 max_seconds = 5.0 | |
| enable_stage_2 = False | |
| seed = 1234 | |
| strength = 0.8 noise_scale = 0.5 | |
| ``` | |
| β Pre-validated offline **before spending any quota**: | |
| `validate_dimensions(256, 256) -> (256, 256)`, `validate_num_frames(9) -> 9`. | |
| Space variables at the time of the run π: | |
| ``` | |
| LTX25_TRANSFORMER_PRECISION = bf16 LTX25_ZEROGPU_SIZE = auto | |
| LTX25_TEXT_ENCODER_PRECISION = bf16 LTX25_GPU_DURATION_SECONDS = 60 | |
| LTX25_QUANTIZATION = int8 LTX25_OUTPUT_RETENTION_GB = 20 | |
| HF_HUB_DISABLE_XET = 1 | |
| ``` | |
| β οΈ **The two `*_PRECISION` variables were inert** β not wired to anything, at | |
| this run or ever. They are recorded here because they were genuinely set on the | |
| Space at the time, not because they did anything. **Both have since been deleted | |
| from the Space.** See Β§9.1. | |
| > β **[On "Both have since been deleted"] β half-false when read back from the | |
| > platform (2026-08-25).** The live Space variables show | |
| > `LTX25_TRANSFORMER_PRECISION=bf16` still **PRESENT**; only | |
| > `LTX25_TEXT_ENCODER_PRECISION` is gone. The deletion was asserted, not read | |
| > back β this document's own organising rule, violated by this document. (Β§9.1's | |
| > "both precision variables were **removed**" remains true: that sentence is | |
| > about the code-side removal recorded in `PLAN.md` Β§2.6, which did happen.) | |
| > The survivor is doubly wrong-shaped: inert as ever, **and** it now advertises | |
| > bf16 on a Space whose transformer actually loads in bitsandbytes int8 β a live | |
| > instance of exactly the advertised-lever class Β§9.1 warns about. Its deletion | |
| > is being requested with the next deploy. | |
| --- | |
| ## 3. The failure | |
| β Returned value: | |
| ``` | |
| [null, null, "### β Generation failed β unexpected `Error` | |
| > 'GPU task aborted' | |
| This error is not one of ours, which usually means it came from the pipeline | |
| or the platform. The full traceback is in the Space logs."] | |
| ``` | |
| `'GPU task aborted'` is raised by the **`spaces` runtime**, not by our code π. | |
| β **No artifact was produced** β the download directory was empty afterwards. | |
| β `CONNECT_SECONDS 3.34` Β· `PREDICT_WALL_CLOCK_SECONDS 100.35` | |
| ### β There is no traceback. That is the finding, not a gap in the report. | |
| **There is no Python traceback anywhere in the Space run log, and no OOM or | |
| SIGKILL text either** β . | |
| > **The forked GPU worker was *killed*. It did not raise.** | |
| An exception unwinds and leaves a stack. A `SIGKILL` leaves nothing. **The | |
| absence is evidence**, and it is the single detail that most constrains the | |
| hypotheses in Β§8 β an application-level fault of any kind would have left a | |
| trace, and none did. | |
| ### β οΈ And our error text points the user at something that does not exist | |
| The message says *"The full traceback is in the Space logs."* **There isn't | |
| one.** Our own copy sends whoever reads it to look for a thing that is not | |
| there, in the one situation where the absence is the most important fact | |
| available. | |
| That is a defect in the error text, found by accident, and it is listed in Β§11. | |
| --- | |
| ## 4. The device-API trace β the container's entire history | |
| β **This run is the only GPU request this container has ever served.** Seven | |
| lines, in full: | |
| ``` | |
| startup-report 200 | |
| schedule taskId=140074752841728 200 | |
| allow tok=2bca839a pid=262 200 | |
| release tok=2bca839a fail=true 404 | |
| schedule taskId=140074752841728 200 <-- SAME taskId | |
| allow tok=1c3ef0c9 pid=286 200 | |
| release tok=1c3ef0c9 fail=true 404 | |
| ``` | |
| **Identical `taskId`, two different allow-tokens, two different worker pids.** | |
| So: **one logical task, two forked workers, both dead, one retry by `spaces`.** | |
| Both released with `fail=true` β the platform booked them as failures. | |
| The retry is the platform's, not ours. **It died the same way twice**, which | |
| rules out a transient. | |
| --- | |
| ## 5. Startup β healthy, and genuinely new evidence | |
| ``` | |
| fetched 37/37 files | |
| LTX-2.5 Studio starting | |
| config : zerogpu_size=auto duration=60s expected_weights=72.19GB outputs<=20GB | |
| dtype : bf16 (fixed; no precision switch) | |
| weights cached : 37/37 files | |
| disk : 82.2 GB of 150 GB cap [OK] = snapshot 72.2 + packed 0.0 + image ~10 | |
| zerogpu size : not yet known (reported after packing) | |
| model revision : 871165de037793df7d75c23a02dd7f45fd364c97 | |
| loaded transformer as int8 | peak RSS 0.84 -> 55.89 GiB | |
| The module 'LTX2VideoTransformer3DModel' has been loaded in `bitsandbytes` 8bit | |
| spaces.GPU kwargs: ['duration', 'size'] | |
| Cleaned 65.25GB of tensor files from ~/.cache/huggingface/hub/models--*/blobs/* after packing | |
| POST http://device-api.zero/startup-report "HTTP/1.1 200 OK" | |
| * Running on local URL: http://0.0.0.0:7860, with SSR | |
| ``` | |
| > **[On `loaded transformer as int8`]** β accurate as printed. At the time of this | |
| > run `components_to_quantize=["transformer"]`: **one** component was quantised, | |
| > and the single `bitsandbytes` line above corroborates it. The build changed twice | |
| > afterwards β `02fe943` added the text encoder, `c7617b1` made the log name every | |
| > component it quantises β so the same load today prints `quantised transformer, | |
| > text_encoder as int8`. Expect doc and code to differ **here and only here**; the | |
| > quote is left byte-identical because it is the record of what ran. Consequence | |
| > for Β§6: **the 55.89 GiB was measured with one component in int8, not two.** | |
| **Everything the deploy was built to do, it did** β β the 37-file runtime | |
| download at the pinned revision, the int8 transformer load, tensor packing, and | |
| the post-pack prune. **On a Space that had evicted twice and never run.** | |
| --- | |
| ## 6. The RAM ceiling β two measured bounds of ours, one inherited number now falsified | |
| > ### β Ours: the ceiling is **above 60.0 GB** | |
| > | |
| > Startup reached **55.89 GiB = 60.0 GB** peak RSS **and was not killed.** | |
| > Therefore the cgroup ceiling is measurably higher than that. **This is our own | |
| > datum, from this platform, from this run** β the first real information anyone | |
| > on this build has had about that number. | |
| > ### β Ours, superseding: the ceiling is **above 93.56 GB** | |
| > | |
| > The route-B run at sha `8fb292d6` β transformer **and** text encoder in int8 β | |
| > reached **87.13 GiB = 93.56 GB** peak RSS and **was not killed either.** | |
| > Startup completed, packing ran, the Space came up RUNNING. Same platform, same | |
| > kind of datum, and it replaces the bound above as our best lower bound. | |
| > | |
| > The 60.0 GB figure stays exactly as recorded: it is a true measurement of a | |
| > **different configuration** (Β§5), and the note at that log line already says so. | |
| > ### β Inherited: the **~68 GB** figure is **FALSIFIED** | |
| > | |
| > A community cgroup observation. **We never measured it, and now we have | |
| > measured past it.** A process on this platform survived **93.56 GB β 25.56 GB | |
| > above it.** It is no longer "unconfirmed but consistent"; for this Space it is | |
| > **wrong**, and every headroom figure computed against it was wrong by at least | |
| > that margin. | |
| > | |
| > **What it cost while it stood:** it made the 60.0 GB run look like ~8 GB from | |
| > the edge. The true headroom at that moment was **β₯ 33.56 GB**. The whole | |
| > cgroup-OOM hypothesis in Β§8 was built inside a gap that did not exist. | |
| **The axis was right, and the inherited half is the half that broke.** Keeping | |
| "ours" apart from "inherited" is what made this correctable in one step: nothing | |
| we measured had to be revisited, and the falsification landed on exactly the | |
| claims already marked β οΈ. Had the two been collapsed into a single "β68 GB | |
| ceiling", surviving 93.56 GB would have read as our own data contradicting | |
| itself, rather than as a refutation of someone else's number. | |
| --- | |
| ## 7. `ZEROGPU_DISK.md` Β§4.2 predicted this, and it lands | |
| Written **before** the deploy, when both statements were βΈοΈ: | |
| > *"int8's RSS should be expected to approach the **materialised** weight size β | |
| > order 38 GB for A, 53 GB for B β¦ **Not 20 GiB.** That turns the ~68 GB cgroup | |
| > figure from a number someone mentioned into **the binding constraint, and the | |
| > first thing to measure.**"* | |
| | prediction | measured | | | |
| |---|---|---| | |
| | RSS β 53 GB on route B, **not** ~20 GiB | **60.0 GB** β | lands, with ~1.13Γ for the transient bf16 source held during conversion | | |
| | the prune **may free nothing** | **fired: 65.25 GB freed** β | the favourable branch, which Β§4.2 declined to assume | | |
| | RAM, not disk, is the binding constraint | β οΈ leading hypothesis, Β§8 | the disk line read 82.2 / 150 GB **[OK]** and the workload still died | | |
| **Both βΈοΈ items are now β .** The 20.05 GiB bf16 figure was correctly rejected as | |
| a floor for a quantising load β the mechanism given at the time (safetensors | |
| mmaps; quantisation materialises anonymous memory) is the same mechanism that | |
| governs the prune, and both halves behaved as that predicted. | |
| β **One correction inside that quote.** It called the ~68 GB figure *"the | |
| binding constraint, and the first thing to measure."* **The instruction was | |
| right and the number was wrong** β we measured it, and Β§6 now records it as | |
| falsified. The quote is left byte-identical because it is the record of what was | |
| predicted; what it got right was *what to measure*, not *what the value was*. | |
| βΈοΈ **`ZEROGPU_DISK.md` Β§4.2 should be upgraded βΈοΈ β β from this run.** Not done | |
| here β that file is gated behind the operator's docs-last order. See Β§11. | |
| --- | |
| ### β The same mechanism, third application β and this time it ran against us | |
| Β§4.2 gave **one** mechanism: a quantising load materialises int8 tensors as | |
| anonymous memory, losing the laziness `safetensors` mmap provides. It used that | |
| one mechanism to settle two questions β why int8 RSS approaches the materialised | |
| weight size, and why the post-pack prune might free nothing. **It has now | |
| settled a third, in the direction opposite to the one we expected:** | |
| > **Quantising the text encoder materialised what `safetensors` had been keeping | |
| > lazy. Peak RSS went UP by 31.24 GiB (33.54 GB)** β from 55.89 GiB to | |
| > **87.13 GiB** β in a change made to bring it *down*. | |
| That is a stronger result than a new explanation would have been. It is not a | |
| mechanism we lacked; it is **the mechanism we already had, predicting an outcome | |
| we still got wrong.** | |
| β **And Β§9 said so, in prose, two paragraphs below the table that got it wrong:** | |
| *"Quantising more raises **transient** load-time RSS even as it lowers steady | |
| state ... If the kill is at load rather than at fork, B makes things worse, not | |
| better."* The transient half of that landed exactly. | |
| --- | |
| ### The instrument lesson β why the arithmetic missed what the prose had | |
| Projected **~46.5 GB**; measured **93.56 GB**. Wrong by 47.6 GB, in the opposite | |
| direction. The cause is visible in Β§9's table, inside a single column: | |
| | row | `expected RSS` | what that number actually is | | |
| |---|---:|---| | |
| | now β transformer int8 only | **60.0 GB β measured** | a **peak**, from `peak_rss_gib()` | | |
| | + text encoder int8 | ~46.5 GB β οΈ | a projected **steady state** | | |
| **Those are different quantities, and the table subtracted one from the other.** | |
| A weight-footprint saving β ~12 GB of bf16 text encoder β was deducted from a | |
| measured *peak*, yielding a number that is neither one nor the other. The | |
| transient cost was named in the prose underneath and never entered the | |
| arithmetic. | |
| `peak_rss_gib()` reports the maximum across the load. **That is the right number | |
| for "will loading survive" and the wrong number for "how much room does a forked | |
| worker have."** It answered the question it was built to answer; the question we | |
| needed β steady state, once the load settles β was invisible to it, and still is. | |
| βΈοΈ **We have no steady-state measurement, and steady state is what the RAM | |
| hypothesis was always about.** This is Β§9.1.6's ladder one rung further out: the | |
| argument reached the function, with the right arguments, of the right type β and | |
| measured **a different quantity than the one the decision needed.** | |
| --- | |
| ## 8. Hypotheses | |
| ### 1. β WAS LEADING β host RAM: the forked worker is cgroup-OOM-killed | |
| > β **The arithmetic this hypothesis stood on is falsified.** It needed a | |
| > ceiling near β68 GB with the load sitting ~8 GB beneath it. The ceiling is | |
| > **above 93.56 GB** (Β§6) and the load at that moment sat **β₯ 33.56 GB** beneath | |
| > it. **There was no narrow margin for a copy-on-write fork to exhaust.** | |
| β οΈ **Two errors are corrected here, not one.** This paragraph previously opened | |
| *"Steady state **60.0 GB β **"*. **60.0 GB was a peak, not a steady state** β | |
| `peak_rss_gib()` reports the maximum across the load (Β§7). Marked β , a | |
| mislabelled quantity was reading as a confirmed one, and it was the quantity the | |
| hypothesis turned on. | |
| **The signature still fits, and still has nothing to attach to:** instant death, | |
| **twice**, **no traceback**, no exception, both `fail=true`. That is `SIGKILL` β | |
| which is what a cgroup OOM does, and also what several other things do. | |
| **Precisely what died and what did not:** | |
| | claim | status | | |
| |---|---| | |
| | the ceiling is β68 GB and we sat ~8 GB below it | β **falsified** (Β§6) | | |
| | the kill was a cgroup OOM of the forked worker | β οΈ **severely weakened, not disproven** | | |
| | RAM is involved at all | βΈοΈ **open** | | |
| RAM-at-fork is **not strictly excluded**: we know neither the steady state (Β§7) | |
| nor what the forked worker materialises on top of it. What is gone is the | |
| *reason to believe it* β and that reason was arithmetic, not evidence. | |
| ### 2. WEAKER β the 60-second duration reservation | |
| Both attempts died at **~50 s** β (100.35 s wall-clock across two) β **under** | |
| the reservation. **There was no timeout to hit.** | |
| Not excluded: a slow fork-and-restore could be culled near the boundary. But it | |
| is a poor fit for a death that leaves no trace, and it does not explain why the | |
| retry failed identically. | |
| ### βΈοΈ Not established | |
| Whether the kill happens **at fork** or **at load inside the worker**. That | |
| distinction decides whether Β§9's recommendation helps or hurts, and nothing in | |
| this run separates them. | |
| --- | |
| ## 9. Recommendation β route B, and its first signal is free | |
| **Widen `components_to_quantize` to include the text encoder, redeploy, and read | |
| the peak-RSS line.** | |
| > β **An earlier version of this section said "flip | |
| > `LTX25_TEXT_ENCODER_PRECISION` to int8". That instruction was false** β the | |
| > variable is wired to nothing. See **Β§9.1**, which is the more important half | |
| > of this section. | |
| ### β The first signal costs zero GPU quota | |
| Packing and loading happen **at module scope, before any `@spaces.GPU` call** β | |
| and the device-api trace confirms it: `startup-report` precedes the first | |
| `schedule` β . So: | |
| - **RSS drops to ~46 GB** β retry the same 256 Γ 256 Γ 9 request, and that is a | |
| real test of the RAM hypothesis. | |
| - **Startup dies instead** β revert, and **we have learned the kill is at load | |
| rather than at fork.** Also free, and it answers the βΈοΈ in Β§8. | |
| Either outcome is informative and neither spends quota to get there. | |
| > β **Neither branch fired. The signal was free, and it was a third outcome the | |
| > section did not enumerate:** RSS **rose** to 93.56 GB and startup **survived** | |
| > anyway. Both branches assumed the ceiling was close; the run falsified the | |
| > ceiling instead of choosing between them. β The claim that the signal costs no | |
| > quota held β packing and loading precede the first `@spaces.GPU` call, exactly | |
| > as the device-api trace predicted. | |
| ### The corrected saving | |
| β οΈ **A previously-circulated figure of 15.13 GB was wrong.** It came from | |
| `ZEROGPU_DISK.md`'s bound **A**, which quantises the text encoder **and** | |
| connectors. The change actually available β adding `"text_encoder"` to | |
| `components_to_quantize` in `_quantization_config` β covers the **text encoder | |
| alone**: 23.92 GB bf16 π, so **β 12 GB saved.** | |
| | | packed | expected RSS | headroom if ceiling β 68 GB β *(falsified, Β§6)* | | |
| |---|---:|---:|---:| | |
| | **now** β transformer int8 only | 53.2 GB | **60.0 GB β measured** | ~8 GB β οΈ | | |
| | **+ text encoder int8** | ~41.2 GB β οΈ | ~46.5 GB β οΈ | ~21.5 GB β οΈ | | |
| β οΈ The derived column scales by the **1.13 transient factor** the measured run | |
| implies β **one data point**, so it is an extrapolation from a single | |
| observation, not a rate. | |
| > β **Superseded by measurement β kept because it is the reasoning the change | |
| > was made on.** Two independent faults: the ceiling is not β68 GB (Β§6), and the | |
| > `expected RSS` column mixes a measured **peak** with a projected **steady | |
| > state** (Β§7). Measured outcome of the `+ text encoder int8` row: | |
| > **87.13 GiB = 93.56 GB peak**, against ~46.5 GB projected β **wrong by 47.6 GB | |
| > and in the opposite direction.** The `packed ~41.2 GB` estimate is not | |
| > contradicted by this run; only the RSS column is. | |
| ### β οΈ Two costs, and the first is serious | |
| **Route B puts at risk the one thing on this Space that demonstrably works.** | |
| **Quantising more raises *transient* load-time RSS even as it lowers steady | |
| state**, because each module briefly holds both its bf16 source and its int8 | |
| output. **If the kill is at load rather than at fork, B makes things worse, not | |
| better.** The peak-RSS line in `_base_pipeline` is what tells those apart on the | |
| next run β which is why the recommendation is *"flip it and read the line"* | |
| rather than *"flip it and retry the generation"*. | |
| **Attributability.** Β§4.2 wanted int8 sequenced as a same-seed pair against a | |
| signed-off bf16 baseline, so *"what does int8 cost in quality?"* could be | |
| answered in one look. **Running it as a fix spends that**, and it cannot be | |
| recovered later without re-running both halves. | |
| --- | |
| ## 9.1 β The instruction in Β§9 was false, and this document nearly proved the RAM hypothesis wrong | |
| **`LTX25_TEXT_ENCODER_PRECISION` is not a lever. It is wired to nothing.** | |
| β Verified by mutation rather than by reading, twice and independently: | |
| ``` | |
| grep -rn TEXT_ENCODER_PRECISION ltx25_studio/ app.py -> no match, any file | |
| config, variable UNSET -> RuntimeConfig(..., quantization='bf16') | |
| config, variable = 'int8' -> RuntimeConfig(..., quantization='bf16') | |
| IDENTICAL under mutation True | |
| illegal value 'fp4-nonsense' accepted silently True | |
| CONTRAST β a lever that IS wired: | |
| LTX25_QUANTIZATION='fp4-nonsense' -> ConfigError: not a recognised value. | |
| ``` | |
| It is not half-wired. **It is not a `RuntimeConfig` field at all**, so it is | |
| never resolved, never validated and never logged. π `PLAN.md` records why: both | |
| precision variables were **removed** because *"an advertised lever attached to | |
| nothing is worse than no lever at all"*. The Space Variables were leftovers from | |
| before that removal. | |
| **And the running Space said so, in a line already quoted in Β§5:** | |
| ``` | |
| dtype : bf16 (fixed; no precision switch) | |
| ``` | |
| **Nobody read it.** The evidence was in the log we were all working from. | |
| ### What it would have cost | |
| Flipping the variable and restarting would have produced **no change in peak | |
| RSS** β because nothing would have changed. **We would have read an unmoved | |
| number as evidence against the RAM hypothesis** (Β§8.1), abandoned the leading | |
| explanation, and gone looking elsewhere. | |
| > **A wrong conclusion from a real experiment, indistinguishable from a | |
| > legitimate negative result.** | |
| That is the same failure shape this project has now hit four times, and it is | |
| the one `997b080`'s commit message describes for the transformer config: *if the | |
| config never reaches `from_pretrained`, the Space deploys bf16 while the variable | |
| says int8 β¦ and the conclusion drawn is "int8 does not fit", from a run in which | |
| int8 was never applied.* | |
| **This document was written to warn about that class, and contained a live | |
| instance of it.** Writing the warning does not confer immunity to it. | |
| ### How it happened β the sequence, which is the useful part | |
| 1. The **orchestrator** set the Space Variable believing it was a live lever. In | |
| his words: *"it is not wired, so it did nothing except trigger a pointless | |
| restart β¦ A false instruction sitting in the only record of this failure is | |
| the worst possible place for one, and it is my instruction that is false."* | |
| 2. The **lead's dispatch repeated it** without checking whether it was wired. | |
| 3. The **implementer wrote it into this file** β the only record of the failure | |
| β likewise unchecked. | |
| 4. The **operator imposed a wiring check** before any reading from a restart | |
| would be trusted. | |
| 5. The **mutation test settled it** before a restart could be misinterpreted. | |
| **Three people passed the same unverified claim along, and the check that caught | |
| it cost about thirty seconds.** The variable had been sitting in the Space doing | |
| nothing, and the startup log had been announcing that fact the whole time. | |
| π `PLAN.md` Β§9.1.6 lists ladder rung 5 as *"hand-built, one call site at a | |
| time"*. **This is rung 5 applied to an environment variable rather than a call | |
| site β and it caught a live one.** The ladder is not only about function | |
| arguments; anything that looks like an input has a rung 5. | |
| --- | |
| ### The routes not recommended | |
| - β οΈ **Route A β duration 60 β 180.** No free signal, and it targets the branch | |
| the evidence already argues against: both workers died **under** the existing | |
| reservation. | |
| - β **Route C β instrument the worker. NOW THE LEADING ROUTE**, no longer the | |
| fallback. B's free signal has been spent: it returned a real measurement, that | |
| measurement **falsified the ceiling** rather than confirming the margin, and | |
| the kill is unexplained again. Route C is the only remaining path that returns | |
| **evidence** instead of another inference β and every inference we have made | |
| about this kill has now been wrong at least once. | |
| --- | |
| ## 9.2 What route B bought, what it cost, and the risk it introduced | |
| **Bought:** β the ceiling, as a real number of ours β *above 93.56 GB* β which | |
| falsified an inherited figure that had been load-bearing in planning since before | |
| any deploy. β Confirmation that the widening reaches the checkpoint: **two** | |
| `bitsandbytes` module lines where the measured run had one β | |
| ``` | |
| The module 'Gemma4UnifiedForConditionalGeneration' has been loaded in `bitsandbytes` 8bit | |
| The module 'LTX2VideoTransformer3DModel' has been loaded in `bitsandbytes` 8bit | |
| ``` | |
| **Cost:** β οΈ the attributability Β§9 warned about is spent β int8 was run as a fix | |
| rather than as a same-seed pair against a signed-off bf16 baseline, and it cannot | |
| be recovered without re-running both halves. | |
| ### β οΈ The new risk, stated plainly | |
| **The load now peaks 33.54 GB higher than the configuration that was working, | |
| and we know the ceiling only as "above 93.56 GB" β not by how much.** | |
| We may be loading far closer to the real limit than before, and we are paying | |
| that for a **steady-state saving we have never measured** (Β§7). The motivating | |
| hypothesis for the change β a tight RAM margin at fork β is itself the thing the | |
| change disproved. That is an uncomfortable position: the justification did not | |
| survive the experiment it justified. | |
| > β **Operator's standing decision: if the retry fails, `02fe943` is reverted.** | |
| > Keeping a change whose motivating hypothesis is dead, at **31.24 GiB** of | |
| > additional peak RSS, for an unmeasured benefit, is not defensible. | |
| ### β Retry outcome β it succeeded | |
| **It did not fail, so `02fe943` STAYS.** The standing revert decision above is | |
| discharged, not overridden. Detail and the attribution limit: Β§9.3. | |
| --- | |
| ## 9.3 β The Space generated β and what that does **not** establish | |
| ``` | |
| PREDICT_WALL_CLOCK_SECONDS 21.44 | |
| RESULT: ["output.mp4", null, "β Text to video complete. seed 1234 Β· run directory outputs/run-fc6d3a762eaa"] | |
| ``` | |
| ### The artifact is real β structure **and** content | |
| Structure alone was not accepted, because **H9 taught this build that a green | |
| stream table and a silent file are indistinguishable.** So the video got the | |
| analogue of the RMS check that caught H9: | |
| ``` | |
| 39,146 bytes Β· h264 Β· 256x256 Β· 24 fps Β· 9 frames Β· duration 0.375 s | |
| verify_streams text_to_video -> MATCHES, exit 0 (video: True, audio: False -- correct for this mode) | |
| global pixel std 72.657 (~0 would be a flat/blank frame) | |
| mean |frame-to-frame| 18.894 per-pair 57.17 .. 8.35, none near zero | |
| distinct frames 9 of 9 | |
| dynamic range 0.0 .. 255.0 on every frame | |
| ``` | |
| β **Varied, moving, all frames distinct.** Not a blank file that passes a | |
| schema. | |
| ### The device-API contrast β the cleanest evidence in this file | |
| ``` | |
| FAILED run: schedule -> allow tok=2bca839a pid=262 -> release fail=true 404 | |
| schedule -> allow tok=1c3ef0c9 pid=286 -> release fail=true 404 (same taskId, spaces retried) | |
| SUCCESS run: schedule -> allow tok=34ddf8b8 pid=258 -> release fail=FALSE 200 (one worker, no retry) | |
| ``` | |
| **`fail=true` twice against `fail=false` once**, from the platform's own | |
| accounting rather than from ours. | |
| --- | |
| ### β The caveat, stated as prominently as the success: **the fix is not attributed** | |
| **The failing and succeeding runs differ by TWO things:** | |
| 1. `02fe943` β the text encoder moved to int8. | |
| 2. **A full container rebuild** β new image, new download, plausibly a different | |
| worker node. | |
| **This is not the clean A/B we had for the RSS number, and the difference is | |
| worth being exact about.** For RSS, an AST diff of the shipped package showed the | |
| quantised set was the *sole* runtime variable between the two measurements β so | |
| the 31.24 GiB rise is attributable. Here a rebuild sits in the middle, and a | |
| rebuild changes things we do not observe. | |
| > **n=1 success against n=1 failure, with two variables between them, supports | |
| > "the Space now serves" and does not support "the widening is why."** | |
| **And the reason to resist the causal reading is stronger than the usual one:** | |
| the hypothesis that motivated `02fe943` β cgroup OOM against a ~68 GB ceiling β | |
| is **dead**, falsified by our own 87.13 GiB survival (Β§6). So we changed | |
| something for a reason that turned out to be wrong, and the thing then worked. | |
| **Those are two facts, not one causal chain.** βΈοΈ Readings still open: | |
| - RAM at fork was real, with different arithmetic than ours, and the lower steady | |
| state genuinely helped. | |
| - The first failure was a platform-side transient; **the rebuild alone fixed it.** | |
| - Something else about the new container. | |
| β **Route C β instrument the worker β remains the leading path.** This success | |
| does not retire it; it makes it *cheaper to justify*, because there is now a | |
| **working baseline to instrument against** instead of a dead Space. | |
| ### Costs | |
| ``` | |
| failed run 100.35 s wall-clock, TWO worker attempts | |
| success run 21.44 s wall-clock, ONE worker attempt | |
| session total 121.79 s wall-clock; at xlarge 2x that is <= ~243.58 GPU-seconds | |
| budget 2400 GPU-seconds/day (PRO) -> ~10.2% consumed | |
| ``` | |
| βΈοΈ Exact billed seconds remain unmeasurable β no readable ZeroGPU quota endpoint | |
| (404 on all three probed). The figure above is a **bound**, not a measurement. | |
| --- | |
| ## 10. β One thing worked, and it was the acceptance criterion | |
| On failure the app returned **`[null, null, error]`** and **did not invent a | |
| file path** β . | |
| **The no-silent-empty-output invariant held under a real platform failure.** | |
| That is the 2.3 outage class β a Space reporting success while producing | |
| nothing β and it was the acceptance criterion in the original brief. It was | |
| tested by accident, on the first real failure, and it passed. | |
| The error was surfaced as an error, named as **not ours**, and no artifact was | |
| fabricated to accompany it. | |
| > β **And now the other half.** On the successful run the app returned a path | |
| > **and the file behind it was real** β 39,146 bytes, 9 distinct frames, pixel | |
| > std 72.657 (Β§9.3). The invariant has now been exercised in both directions: | |
| > **no file claimed on failure, and a genuine file behind the claim on success.** | |
| > A build that only ever failed could not have shown the second half. | |
| --- | |
| ## 11. Follow-ups this run created | |
| | | item | status | | |
| |---|---|---| | |
| | 1 | **Our error text tells users to read a traceback that does not exist** (Β§3). In a kill, there is nothing to read, and that is the most informative fact β the copy should say so instead. | βΈοΈ open | | |
| | 2 | **`ZEROGPU_DISK.md` Β§4.2**: upgrade the RSS prediction and the prune branch βΈοΈ β β from this run. **β Applied in the 2026-08-25 docs pass** β Β§4.2 now carries the dated upgrade annotation. | β resolved (2026-08-25) | | |
| | 3 | **`OPERATOR_HANDOFF.md` Β§3 and Β§5 are stale in *both* directions** β Β§3 says the Space *"has never run"*, Β§5 says a deploy is *"not currently possible"*. **Startup demonstrably works and generation demonstrably does not**, and the document currently cannot express that distinction. **β Doubly stale itself by 2026-08-25: generation now demonstrably works (item 10 β 11 of 11), and the handoff was reworked to "one mode, once", which the mode matrix then superseded in turn. The handoff now carries dated annotations for both; this row is kept as the record of the gap as it stood after the first run.** | β resolved (2026-08-25) | | |
| | 4 | βΈοΈ **No readable ZeroGPU quota endpoint** β all three candidate URLs 404, so billed seconds cannot be measured, only bounded. Worst case ~200 GPU-s of 2400/day for this run (`xlarge` bills at 2Γ; packed 51.1 GB is over the 42.00 GiB threshold π). | βΈοΈ open | | |
| | 5 | **Bus defect**: `lead-gate.mjs:1709` computes `isLead = (session_id === SOMA_LEAD_SESSION_UUID)`. A `/clear` rotates the session id while the env var stays baked in from launch, so every soma bus tool is denied and the lead cannot file a report. **Side effect: it also disarms the lead-only `git push` / `git merge` guard.** | βΈοΈ open, infrastructure | | |
| | 6 | βΈοΈ **No steady-state RSS measurement exists.** `peak_rss_gib()` reports the load peak; the RAM question is about what remains resident when the load settles. Every RSS number in this document is a peak. **This is the measurement that would have prevented the wrong prediction in Β§7** and it is still not taken. | βΈοΈ open | | |
| | 7 | β **The ~68 GB ceiling was inherited, never measured, and is now falsified** (Β§6). Audit any other planning number with the same provenance β carried in from outside, never verified here, and load-bearing. It survived this long because it was *consistent* with our data, which is not the same as supported by it. | βΈοΈ open | | |
| | 8 | β οΈ **`02fe943` STAYS** β the retry succeeded, so the revert condition never triggered (Β§9.2). But it stands on a hypothesis this run killed and costs 31.24 GiB of peak RSS for a steady-state saving still unmeasured. **Kept because it did not fail, not because it was shown to help.** | β resolved, β οΈ unjustified | | |
| | 9 | β **The fix is unattributed** β a container rebuild changed alongside `02fe943`, so n=1 vs n=1 with two variables (Β§9.3). Distinguishing them needs either a revert-and-retry on the same image, or route C. **Until then no document should say the widening fixed generation.** | βΈοΈ open | | |
| | 10 | β **SUPERSEDED β all 11 modes have now run and generate**, at the same 256x256x9. The mel front-end and the 16 kHz/48 kHz asymmetry have both been exercised on this platform and both passed. The matrix found 9 of 11 and one real defect β both video-INPUT modes, from a filepath `str` reaching `LTX2VideoCondition(frames=β¦)` β fixed at `d2eb6f2` and β **re-verified on the Space at sha `287ba9c`**. Full evidence: **`docs/SPACE_MODE_MATRIX.md`**. βΈοΈ Still one configuration only: 256x256x9, stage 2 off, no LoRA. | β resolved | | |
| --- | |
| ## 12. Repo state at the time of writing | |
| - **`origin/soma/ltx25-studio` is at `1802bfd`** β the six earlier commits were | |
| pushed by the orchestrator; the findings commits after it are unpushed. Pushing | |
| is the orchestrator's β this team does not push by design. Verified with | |
| `git rev-list --count origin/soma/ltx25-studio..HEAD`, which is the only | |
| authority on this; an earlier count taken from *"what I authored this round"* | |
| said four and silently omitted `51e1985`, this document. | |
| - PR **#1** is OPEN and draft: | |
| <https://github.com/techfreakworm/ltx2.5-studio/pull/1> | |
| - Offline gate at that tip: **2053 passed, 13 skipped, 0 failed.** | |
| > **Superseded as repo state (annotated 2026-08-25).** `origin/soma/ltx25-studio` | |
| > advanced to `17b589c` β everything through the 11-of-11 findings is pushed | |
| > (`git rev-list --count` = 0, verified in the 2026-08-25 audit). Commits since | |
| > are local and queued for the next push: the fork-boundary pair `6bb211a` | |
| > (SPACE_MODE_MATRIX Β§4), five hardening fixes through `f0daa75` (LoRA | |
| > lifecycle, duration estimator, upload bound, desync guard `59cfa6e`, mmap | |
| > startup report), and this docs pass. PR #1 remains OPEN and draft. The lines | |
| > above stand as the record of the state when this file was written. | |