LTX2.5-Studio / docs /SPACE_GENERATION_FINDING.md
techfreakworm's picture
docs: every sentence stays true after the deploy: README and the release notes say this is deploy 2's code and point to issue #32 for what the Space shows, and no doc says the Space still runs deploy 1, is not deployed, or has not been redeployed since August (#32)
abc9150 unverified
|
Raw History Blame Contribute Delete
39.3 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

The Space: a killed worker, a falsified ceiling, and a working generation

Status on 2026-11-16

This is the August 2026 record of the first Space runs; its body is kept as written. Superseded since:

  • Β§2 annotation "LTX25_TRANSFORMER_PRECISION=bf16 still PRESENT": the Space's variables, read on 2026-11-16, no longer include it (nor LTX25_TEXT_ENCODER_PRECISION).
  • Β§2 "spaces 0.51.1": the Space reports spaces 0.51.3.
  • Β§5's startup report (zerogpu_size=auto duration=60s, dtype : bf16 (fixed; no precision switch), 82.2 GB … + image ~10) is that day's format. The Space set xlarge and 180 s through its variables, and the report was rewritten on 2026-11-16; the new one ran on deploy 1 (SPACE_MODE_MATRIX.md Β§10.1). Since deploy 2's code the reservation follows each run's size and length, and LTX25_GPU_DURATION_SECONDS is ignored.
  • Β§11 #4 "packed 51.1 GB … xlarge bills at 2Γ—": on the 2026-11-13 boot the platform's own progress bar read 40.6G packed, and the run in this file was on large, not xlarge.
  • Β§11 #1 "our error text tells users to read a traceback that does not exist": that sentence is no longer in the UI.

Still true: no steady-state RSS has been measured (Β§11 #6), and billed seconds cannot be read back (Β§11 #4). Where the current truth is: SPACE_MODE_MATRIX.md Β§10 and ../README.md. Deploy 1 (2026-11-16) ran all eleven modes on the Space (SPACE_MODE_MATRIX.md Β§10.1); deploy 2's own check on the Space is reported on issue #32.

β›” INVALIDATED (2026-08-25, evening) β€” "a real video" was COLOUR STATIC

The 39,146-byte artifact this document resolves on, and every Space artifact after it up to sha ff142d9, was produced by a transformer whose conditioning path had been quantised away. The blanket int8 config below named no llm_int8_skip_modules, so bitsandbytes replaced every nn.Linear including the eight LTX2AdaLayerNormSingle modulation heads and the in/out projections. "Content-verified" meant non-empty and non-frozen; nobody opened a frame until the operator did, on 2026-08-25.

βœ… Fixed at eb1aa9f (316 of 1660 Linears spared, names read off the real checkpoint), deployed as Space sha e3b5333, verified by eye. Every mechanical measurement in this document stands β€” RSS, disk, packing, the killed workers, the falsified ceiling, the fetch path, the fork crossing. Only the output claim falls. Full record: the invalidation block at the top of SPACE_MODE_MATRIX.md.

β›” Β§9.3's "do not let any document claim the widening fixed generation" is VINDICATED in the strongest direction available: the widening is what broke it.

β›” One hypothesis was investigated and REFUTED β€” do not re-chase it. That LLM.int8() computes matmuls in fp16 while LTX is bf16-native, evidenced by bitsandbytes' own bf16β†’fp16 cast warning in the Space log, is not the mechanism. llm_int8_threshold sits at its 6.0 default, which is exactly the gate that routes fp16-overflowing columns through the bf16 outlier path, and inside fp16's range a bf16β†’fp16 cast is exact. βœ… The skip list alone fixed it with llm_int8_threshold untouched β€” one variable, cleanly attributed. The warning is real; it was a red herring.

βœ… RESOLVED β€” the Space generates. Read Β§9.3 before citing why.

A real 39,146-byte video, content-verified, at sha 8fb292d6.

⚠️ The fix is NOT attributed. Two things changed between the failing and succeeding runs, not one. "The Space now serves" is established; "the widening is why" is not. §9.3.

The investigation below is kept exactly as written. Its measurements stand, and two of its central claims were falsified by the run that resolved it β€” including one this build had been planning against since before any deploy.

Original framing, first attempt β€” accurate as history: First generation ever attempted on the live private Space. Startup succeeded; the generation failed. The forked ZeroGPU worker was killed twice, with no Python traceback, and spaces returned 'GPU task aborted'.

This file is the only place these measurements exist.

Every load-bearing number is marked: βœ… we measured it Β· πŸ“„ read off a pinned source or the platform Β· ⚠️ inferred Β· ⏸️ not established.

⚠️ The rule this document is organised around

What we measured and what we inherited are kept apart. The RAM ceiling has two numbers attached to it and only one is ours. Blurring them is how a borrowed figure becomes a fact, and this build has spent a day undoing exactly that.


1. What happened

Space techfreakworm/LTX2.5-Studio, private, stage RUNNING πŸ“„
Space SHA a1791c9e856f2b96ab10d89a05a84a0899c8dca5 πŸ“„
spaces 0.51.1 πŸ“„
Hardware id zero-a10g πŸ“„ β€” a legacy id string, not an actual A10G
Account PRO βœ… (whoami-v2 β†’ isPro: true), 2400 GPU-s/day
Client gradio_client 2.6.0 β†’ /on_generate

Startup: healthy. Generation: failed. Those are two different claims and the rest of this file exists to keep them apart.

βœ… Superseded on the retry at sha 8fb292d6: generation now succeeds. The separation above is what made that legible β€” startup never broke, so the failure was always in the second claim. Β§9.3.


2. The request

Chosen to be the cheapest thing that exercises the path β€” 256 Γ— 256 Γ— 9 frames, no conditioning, no stage 2.

mode_id           = text_to_video
prompt            = 'a red balloon rising over a calm lake at dawn'
negative_prompt   = ''
image/video/audio = None
conditions        = (empty)
width, height     = 256, 256
num_frames        = 9
auto_duration     = False        min_seconds = 2.0   max_seconds = 5.0
enable_stage_2    = False
seed              = 1234
strength          = 0.8          noise_scale = 0.5

βœ… Pre-validated offline before spending any quota: validate_dimensions(256, 256) -> (256, 256), validate_num_frames(9) -> 9.

Space variables at the time of the run πŸ“„:

LTX25_TRANSFORMER_PRECISION  = bf16      LTX25_ZEROGPU_SIZE          = auto
LTX25_TEXT_ENCODER_PRECISION = bf16      LTX25_GPU_DURATION_SECONDS  = 60
LTX25_QUANTIZATION           = int8      LTX25_OUTPUT_RETENTION_GB   = 20
HF_HUB_DISABLE_XET           = 1

⚠️ The two *_PRECISION variables were inert β€” not wired to anything, at this run or ever. They are recorded here because they were genuinely set on the Space at the time, not because they did anything. Both have since been deleted from the Space. See Β§9.1.

β›” [On "Both have since been deleted"] β€” half-false when read back from the platform (2026-08-25). The live Space variables show LTX25_TRANSFORMER_PRECISION=bf16 still PRESENT; only LTX25_TEXT_ENCODER_PRECISION is gone. The deletion was asserted, not read back β€” this document's own organising rule, violated by this document. (Β§9.1's "both precision variables were removed" remains true: that sentence is about the code-side removal recorded in PLAN.md Β§2.6, which did happen.) The survivor is doubly wrong-shaped: inert as ever, and it now advertises bf16 on a Space whose transformer actually loads in bitsandbytes int8 β€” a live instance of exactly the advertised-lever class Β§9.1 warns about. Its deletion is being requested with the next deploy.


3. The failure

βœ… Returned value:

[null, null, "### ❌ Generation failed β€” unexpected `Error`

> 'GPU task aborted'

This error is not one of ours, which usually means it came from the pipeline
or the platform. The full traceback is in the Space logs."]

'GPU task aborted' is raised by the spaces runtime, not by our code πŸ“„.

βœ… No artifact was produced β€” the download directory was empty afterwards. βœ… CONNECT_SECONDS 3.34 Β· PREDICT_WALL_CLOCK_SECONDS 100.35

β›” There is no traceback. That is the finding, not a gap in the report.

There is no Python traceback anywhere in the Space run log, and no OOM or SIGKILL text either βœ….

The forked GPU worker was killed. It did not raise.

An exception unwinds and leaves a stack. A SIGKILL leaves nothing. The absence is evidence, and it is the single detail that most constrains the hypotheses in Β§8 β€” an application-level fault of any kind would have left a trace, and none did.

⚠️ And our error text points the user at something that does not exist

The message says "The full traceback is in the Space logs." There isn't one. Our own copy sends whoever reads it to look for a thing that is not there, in the one situation where the absence is the most important fact available.

That is a defect in the error text, found by accident, and it is listed in Β§11.


4. The device-API trace β€” the container's entire history

βœ… This run is the only GPU request this container has ever served. Seven lines, in full:

startup-report                                   200
schedule  taskId=140074752841728                 200
allow     tok=2bca839a  pid=262                  200
release   tok=2bca839a  fail=true                404
schedule  taskId=140074752841728                 200   <-- SAME taskId
allow     tok=1c3ef0c9  pid=286                  200
release   tok=1c3ef0c9  fail=true                404

Identical taskId, two different allow-tokens, two different worker pids. So: one logical task, two forked workers, both dead, one retry by spaces. Both released with fail=true β€” the platform booked them as failures.

The retry is the platform's, not ours. It died the same way twice, which rules out a transient.


5. Startup β€” healthy, and genuinely new evidence

fetched 37/37 files
LTX-2.5 Studio starting
  config          : zerogpu_size=auto duration=60s expected_weights=72.19GB outputs<=20GB
  dtype           : bf16 (fixed; no precision switch)
  weights cached  : 37/37 files
  disk            : 82.2 GB of 150 GB cap [OK] = snapshot 72.2 + packed 0.0 + image ~10
  zerogpu size    : not yet known (reported after packing)
  model revision  : 871165de037793df7d75c23a02dd7f45fd364c97
loaded transformer as int8 | peak RSS 0.84 -> 55.89 GiB
The module 'LTX2VideoTransformer3DModel' has been loaded in `bitsandbytes` 8bit
spaces.GPU kwargs: ['duration', 'size']
Cleaned 65.25GB of tensor files from ~/.cache/huggingface/hub/models--*/blobs/* after packing
POST http://device-api.zero/startup-report "HTTP/1.1 200 OK"
* Running on local URL: http://0.0.0.0:7860, with SSR

[On loaded transformer as int8] β€” accurate as printed. At the time of this run components_to_quantize=["transformer"]: one component was quantised, and the single bitsandbytes line above corroborates it. The build changed twice afterwards β€” 02fe943 added the text encoder, c7617b1 made the log name every component it quantises β€” so the same load today prints quantised transformer, text_encoder as int8. Expect doc and code to differ here and only here; the quote is left byte-identical because it is the record of what ran. Consequence for Β§6: the 55.89 GiB was measured with one component in int8, not two.

Everything the deploy was built to do, it did βœ… β€” the 37-file runtime download at the pinned revision, the int8 transformer load, tensor packing, and the post-pack prune. On a Space that had evicted twice and never run.


6. The RAM ceiling β€” two measured bounds of ours, one inherited number now falsified

βœ… Ours: the ceiling is above 60.0 GB

Startup reached 55.89 GiB = 60.0 GB peak RSS and was not killed. Therefore the cgroup ceiling is measurably higher than that. This is our own datum, from this platform, from this run β€” the first real information anyone on this build has had about that number.

βœ… Ours, superseding: the ceiling is above 93.56 GB

The route-B run at sha 8fb292d6 β€” transformer and text encoder in int8 β€” reached 87.13 GiB = 93.56 GB peak RSS and was not killed either. Startup completed, packing ran, the Space came up RUNNING. Same platform, same kind of datum, and it replaces the bound above as our best lower bound.

The 60.0 GB figure stays exactly as recorded: it is a true measurement of a different configuration (Β§5), and the note at that log line already says so.

β›” Inherited: the ~68 GB figure is FALSIFIED

A community cgroup observation. We never measured it, and now we have measured past it. A process on this platform survived 93.56 GB β€” 25.56 GB above it. It is no longer "unconfirmed but consistent"; for this Space it is wrong, and every headroom figure computed against it was wrong by at least that margin.

What it cost while it stood: it made the 60.0 GB run look like ~8 GB from the edge. The true headroom at that moment was β‰₯ 33.56 GB. The whole cgroup-OOM hypothesis in Β§8 was built inside a gap that did not exist.

The axis was right, and the inherited half is the half that broke. Keeping "ours" apart from "inherited" is what made this correctable in one step: nothing we measured had to be revisited, and the falsification landed on exactly the claims already marked ⚠️. Had the two been collapsed into a single "β‰ˆ68 GB ceiling", surviving 93.56 GB would have read as our own data contradicting itself, rather than as a refutation of someone else's number.


7. ZEROGPU_DISK.md Β§4.2 predicted this, and it lands

Written before the deploy, when both statements were ⏸️:

*"int8's RSS should be expected to approach the materialised weight size β€” order 38 GB for A, 53 GB for B … Not 20 GiB. That turns the ~68 GB cgroup figure from a number someone mentioned into the binding constraint, and the first thing to measure."*

prediction measured
RSS β‰ˆ 53 GB on route B, not ~20 GiB 60.0 GB βœ… lands, with ~1.13Γ— for the transient bf16 source held during conversion
the prune may free nothing fired: 65.25 GB freed βœ… the favourable branch, which Β§4.2 declined to assume
RAM, not disk, is the binding constraint ⚠️ leading hypothesis, §8 the disk line read 82.2 / 150 GB [OK] and the workload still died

Both ⏸️ items are now βœ…. The 20.05 GiB bf16 figure was correctly rejected as a floor for a quantising load β€” the mechanism given at the time (safetensors mmaps; quantisation materialises anonymous memory) is the same mechanism that governs the prune, and both halves behaved as that predicted.

β›” One correction inside that quote. It called the ~68 GB figure "the binding constraint, and the first thing to measure." The instruction was right and the number was wrong β€” we measured it, and Β§6 now records it as falsified. The quote is left byte-identical because it is the record of what was predicted; what it got right was what to measure, not what the value was.

⏸️ ZEROGPU_DISK.md Β§4.2 should be upgraded ⏸️ β†’ βœ… from this run. Not done here β€” that file is gated behind the operator's docs-last order. See Β§11.


β›” The same mechanism, third application β€” and this time it ran against us

Β§4.2 gave one mechanism: a quantising load materialises int8 tensors as anonymous memory, losing the laziness safetensors mmap provides. It used that one mechanism to settle two questions β€” why int8 RSS approaches the materialised weight size, and why the post-pack prune might free nothing. It has now settled a third, in the direction opposite to the one we expected:

Quantising the text encoder materialised what safetensors had been keeping lazy. Peak RSS went UP by 31.24 GiB (33.54 GB) β€” from 55.89 GiB to 87.13 GiB β€” in a change made to bring it down.

That is a stronger result than a new explanation would have been. It is not a mechanism we lacked; it is the mechanism we already had, predicting an outcome we still got wrong.

βœ… And Β§9 said so, in prose, two paragraphs below the table that got it wrong: "Quantising more raises transient load-time RSS even as it lowers steady state ... If the kill is at load rather than at fork, B makes things worse, not better." The transient half of that landed exactly.


The instrument lesson β€” why the arithmetic missed what the prose had

Projected ~46.5 GB; measured 93.56 GB. Wrong by 47.6 GB, in the opposite direction. The cause is visible in Β§9's table, inside a single column:

row expected RSS what that number actually is
now β€” transformer int8 only 60.0 GB βœ… measured a peak, from peak_rss_gib()
+ text encoder int8 ~46.5 GB ⚠️ a projected steady state

Those are different quantities, and the table subtracted one from the other. A weight-footprint saving β€” ~12 GB of bf16 text encoder β€” was deducted from a measured peak, yielding a number that is neither one nor the other. The transient cost was named in the prose underneath and never entered the arithmetic.

peak_rss_gib() reports the maximum across the load. That is the right number for "will loading survive" and the wrong number for "how much room does a forked worker have." It answered the question it was built to answer; the question we needed β€” steady state, once the load settles β€” was invisible to it, and still is.

⏸️ We have no steady-state measurement, and steady state is what the RAM hypothesis was always about. This is Β§9.1.6's ladder one rung further out: the argument reached the function, with the right arguments, of the right type β€” and measured a different quantity than the one the decision needed.


8. Hypotheses

1. β›” WAS LEADING β€” host RAM: the forked worker is cgroup-OOM-killed

β›” The arithmetic this hypothesis stood on is falsified. It needed a ceiling near β‰ˆ68 GB with the load sitting ~8 GB beneath it. The ceiling is above 93.56 GB (Β§6) and the load at that moment sat β‰₯ 33.56 GB beneath it. There was no narrow margin for a copy-on-write fork to exhaust.

⚠️ Two errors are corrected here, not one. This paragraph previously opened "Steady state 60.0 GB βœ…". 60.0 GB was a peak, not a steady state β€” peak_rss_gib() reports the maximum across the load (Β§7). Marked βœ…, a mislabelled quantity was reading as a confirmed one, and it was the quantity the hypothesis turned on.

The signature still fits, and still has nothing to attach to: instant death, twice, no traceback, no exception, both fail=true. That is SIGKILL β€” which is what a cgroup OOM does, and also what several other things do.

Precisely what died and what did not:

claim status
the ceiling is β‰ˆ68 GB and we sat ~8 GB below it β›” falsified (Β§6)
the kill was a cgroup OOM of the forked worker ⚠️ severely weakened, not disproven
RAM is involved at all ⏸️ open

RAM-at-fork is not strictly excluded: we know neither the steady state (Β§7) nor what the forked worker materialises on top of it. What is gone is the reason to believe it β€” and that reason was arithmetic, not evidence.

2. WEAKER β€” the 60-second duration reservation

Both attempts died at ~50 s βœ… (100.35 s wall-clock across two) β€” under the reservation. There was no timeout to hit.

Not excluded: a slow fork-and-restore could be culled near the boundary. But it is a poor fit for a death that leaves no trace, and it does not explain why the retry failed identically.

⏸️ Not established

Whether the kill happens at fork or at load inside the worker. That distinction decides whether Β§9's recommendation helps or hurts, and nothing in this run separates them.


9. Recommendation β€” route B, and its first signal is free

Widen components_to_quantize to include the text encoder, redeploy, and read the peak-RSS line.

β›” An earlier version of this section said "flip LTX25_TEXT_ENCODER_PRECISION to int8". That instruction was false β€” the variable is wired to nothing. See Β§9.1, which is the more important half of this section.

βœ… The first signal costs zero GPU quota

Packing and loading happen at module scope, before any @spaces.GPU call β€” and the device-api trace confirms it: startup-report precedes the first schedule βœ…. So:

  • RSS drops to ~46 GB β†’ retry the same 256 Γ— 256 Γ— 9 request, and that is a real test of the RAM hypothesis.
  • Startup dies instead β†’ revert, and we have learned the kill is at load rather than at fork. Also free, and it answers the ⏸️ in Β§8.

Either outcome is informative and neither spends quota to get there.

β›” Neither branch fired. The signal was free, and it was a third outcome the section did not enumerate: RSS rose to 93.56 GB and startup survived anyway. Both branches assumed the ceiling was close; the run falsified the ceiling instead of choosing between them. βœ… The claim that the signal costs no quota held β€” packing and loading precede the first @spaces.GPU call, exactly as the device-api trace predicted.

The corrected saving

⚠️ A previously-circulated figure of 15.13 GB was wrong. It came from ZEROGPU_DISK.md's bound A, which quantises the text encoder and connectors. The change actually available β€” adding "text_encoder" to components_to_quantize in _quantization_config β€” covers the text encoder alone: 23.92 GB bf16 πŸ“„, so β‰ˆ 12 GB saved.

packed expected RSS headroom if ceiling β‰ˆ 68 GB β›” (falsified, Β§6)
now β€” transformer int8 only 53.2 GB 60.0 GB βœ… measured ~8 GB ⚠️
+ text encoder int8 ~41.2 GB ⚠️ ~46.5 GB ⚠️ ~21.5 GB ⚠️

⚠️ The derived column scales by the 1.13 transient factor the measured run implies β€” one data point, so it is an extrapolation from a single observation, not a rate.

β›” Superseded by measurement β€” kept because it is the reasoning the change was made on. Two independent faults: the ceiling is not β‰ˆ68 GB (Β§6), and the expected RSS column mixes a measured peak with a projected steady state (Β§7). Measured outcome of the + text encoder int8 row: 87.13 GiB = 93.56 GB peak, against ~46.5 GB projected β€” wrong by 47.6 GB and in the opposite direction. The packed ~41.2 GB estimate is not contradicted by this run; only the RSS column is.

⚠️ Two costs, and the first is serious

Route B puts at risk the one thing on this Space that demonstrably works.

Quantising more raises transient load-time RSS even as it lowers steady state, because each module briefly holds both its bf16 source and its int8 output. If the kill is at load rather than at fork, B makes things worse, not better. The peak-RSS line in _base_pipeline is what tells those apart on the next run β€” which is why the recommendation is "flip it and read the line" rather than "flip it and retry the generation".

Attributability. Β§4.2 wanted int8 sequenced as a same-seed pair against a signed-off bf16 baseline, so "what does int8 cost in quality?" could be answered in one look. Running it as a fix spends that, and it cannot be recovered later without re-running both halves.


9.1 β›” The instruction in Β§9 was false, and this document nearly proved the RAM hypothesis wrong

LTX25_TEXT_ENCODER_PRECISION is not a lever. It is wired to nothing.

βœ… Verified by mutation rather than by reading, twice and independently:

grep -rn TEXT_ENCODER_PRECISION ltx25_studio/ app.py   ->  no match, any file

config, variable UNSET     -> RuntimeConfig(..., quantization='bf16')
config, variable = 'int8'  -> RuntimeConfig(..., quantization='bf16')
IDENTICAL under mutation                                  True
illegal value 'fp4-nonsense' accepted silently            True

CONTRAST β€” a lever that IS wired:
LTX25_QUANTIZATION='fp4-nonsense'  ->  ConfigError: not a recognised value.

It is not half-wired. It is not a RuntimeConfig field at all, so it is never resolved, never validated and never logged. πŸ“„ PLAN.md records why: both precision variables were removed because "an advertised lever attached to nothing is worse than no lever at all". The Space Variables were leftovers from before that removal.

And the running Space said so, in a line already quoted in Β§5:

dtype : bf16 (fixed; no precision switch)

Nobody read it. The evidence was in the log we were all working from.

What it would have cost

Flipping the variable and restarting would have produced no change in peak RSS β€” because nothing would have changed. We would have read an unmoved number as evidence against the RAM hypothesis (Β§8.1), abandoned the leading explanation, and gone looking elsewhere.

A wrong conclusion from a real experiment, indistinguishable from a legitimate negative result.

That is the same failure shape this project has now hit four times, and it is the one 997b080's commit message describes for the transformer config: if the config never reaches from_pretrained, the Space deploys bf16 while the variable says int8 … and the conclusion drawn is "int8 does not fit", from a run in which int8 was never applied.

This document was written to warn about that class, and contained a live instance of it. Writing the warning does not confer immunity to it.

How it happened β€” the sequence, which is the useful part

  1. The orchestrator set the Space Variable believing it was a live lever. In his words: "it is not wired, so it did nothing except trigger a pointless restart … A false instruction sitting in the only record of this failure is the worst possible place for one, and it is my instruction that is false."
  2. The lead's dispatch repeated it without checking whether it was wired.
  3. The implementer wrote it into this file β€” the only record of the failure β€” likewise unchecked.
  4. The operator imposed a wiring check before any reading from a restart would be trusted.
  5. The mutation test settled it before a restart could be misinterpreted.

Three people passed the same unverified claim along, and the check that caught it cost about thirty seconds. The variable had been sitting in the Space doing nothing, and the startup log had been announcing that fact the whole time.

πŸ“„ PLAN.md Β§9.1.6 lists ladder rung 5 as "hand-built, one call site at a time". This is rung 5 applied to an environment variable rather than a call site β€” and it caught a live one. The ladder is not only about function arguments; anything that looks like an input has a rung 5.


The routes not recommended

  • ⚠️ Route A β€” duration 60 β†’ 180. No free signal, and it targets the branch the evidence already argues against: both workers died under the existing reservation.
  • βœ… Route C β€” instrument the worker. NOW THE LEADING ROUTE, no longer the fallback. B's free signal has been spent: it returned a real measurement, that measurement falsified the ceiling rather than confirming the margin, and the kill is unexplained again. Route C is the only remaining path that returns evidence instead of another inference β€” and every inference we have made about this kill has now been wrong at least once.

9.2 What route B bought, what it cost, and the risk it introduced

Bought: βœ… the ceiling, as a real number of ours β€” above 93.56 GB β€” which falsified an inherited figure that had been load-bearing in planning since before any deploy. βœ… Confirmation that the widening reaches the checkpoint: two bitsandbytes module lines where the measured run had one β€”

The module 'Gemma4UnifiedForConditionalGeneration' has been loaded in `bitsandbytes` 8bit
The module 'LTX2VideoTransformer3DModel'          has been loaded in `bitsandbytes` 8bit

Cost: ⚠️ the attributability Β§9 warned about is spent β€” int8 was run as a fix rather than as a same-seed pair against a signed-off bf16 baseline, and it cannot be recovered without re-running both halves.

⚠️ The new risk, stated plainly

The load now peaks 33.54 GB higher than the configuration that was working, and we know the ceiling only as "above 93.56 GB" β€” not by how much.

We may be loading far closer to the real limit than before, and we are paying that for a steady-state saving we have never measured (Β§7). The motivating hypothesis for the change β€” a tight RAM margin at fork β€” is itself the thing the change disproved. That is an uncomfortable position: the justification did not survive the experiment it justified.

β›” Operator's standing decision: if the retry fails, 02fe943 is reverted. Keeping a change whose motivating hypothesis is dead, at 31.24 GiB of additional peak RSS, for an unmeasured benefit, is not defensible.

βœ… Retry outcome β€” it succeeded

It did not fail, so 02fe943 STAYS. The standing revert decision above is discharged, not overridden. Detail and the attribution limit: Β§9.3.


9.3 βœ… The Space generated β€” and what that does not establish

PREDICT_WALL_CLOCK_SECONDS 21.44
RESULT: ["output.mp4", null, "βœ… Text to video complete. seed 1234 Β· run directory outputs/run-fc6d3a762eaa"]

The artifact is real β€” structure and content

Structure alone was not accepted, because H9 taught this build that a green stream table and a silent file are indistinguishable. So the video got the analogue of the RMS check that caught H9:

39,146 bytes Β· h264 Β· 256x256 Β· 24 fps Β· 9 frames Β· duration 0.375 s
verify_streams text_to_video -> MATCHES, exit 0   (video: True, audio: False -- correct for this mode)

global pixel std        72.657      (~0 would be a flat/blank frame)
mean |frame-to-frame|   18.894      per-pair 57.17 .. 8.35, none near zero
distinct frames         9 of 9
dynamic range           0.0 .. 255.0 on every frame

βœ… Varied, moving, all frames distinct. Not a blank file that passes a schema.

The device-API contrast β€” the cleanest evidence in this file

FAILED run:  schedule -> allow tok=2bca839a pid=262 -> release fail=true  404
             schedule -> allow tok=1c3ef0c9 pid=286 -> release fail=true  404   (same taskId, spaces retried)
SUCCESS run: schedule -> allow tok=34ddf8b8 pid=258 -> release fail=FALSE 200   (one worker, no retry)

fail=true twice against fail=false once, from the platform's own accounting rather than from ours.


β›” The caveat, stated as prominently as the success: the fix is not attributed

The failing and succeeding runs differ by TWO things:

  1. 02fe943 β€” the text encoder moved to int8.
  2. A full container rebuild β€” new image, new download, plausibly a different worker node.

This is not the clean A/B we had for the RSS number, and the difference is worth being exact about. For RSS, an AST diff of the shipped package showed the quantised set was the sole runtime variable between the two measurements β€” so the 31.24 GiB rise is attributable. Here a rebuild sits in the middle, and a rebuild changes things we do not observe.

n=1 success against n=1 failure, with two variables between them, supports "the Space now serves" and does not support "the widening is why."

And the reason to resist the causal reading is stronger than the usual one: the hypothesis that motivated 02fe943 β€” cgroup OOM against a ~68 GB ceiling β€” is dead, falsified by our own 87.13 GiB survival (Β§6). So we changed something for a reason that turned out to be wrong, and the thing then worked. Those are two facts, not one causal chain. ⏸️ Readings still open:

  • RAM at fork was real, with different arithmetic than ours, and the lower steady state genuinely helped.
  • The first failure was a platform-side transient; the rebuild alone fixed it.
  • Something else about the new container.

βœ… Route C β€” instrument the worker β€” remains the leading path. This success does not retire it; it makes it cheaper to justify, because there is now a working baseline to instrument against instead of a dead Space.

Costs

failed run    100.35 s wall-clock, TWO worker attempts
success run    21.44 s wall-clock, ONE worker attempt
session total 121.79 s wall-clock; at xlarge 2x that is <= ~243.58 GPU-seconds
budget        2400 GPU-seconds/day (PRO) -> ~10.2% consumed

⏸️ Exact billed seconds remain unmeasurable β€” no readable ZeroGPU quota endpoint (404 on all three probed). The figure above is a bound, not a measurement.


10. βœ… One thing worked, and it was the acceptance criterion

On failure the app returned [null, null, error] and did not invent a file path βœ….

The no-silent-empty-output invariant held under a real platform failure. That is the 2.3 outage class β€” a Space reporting success while producing nothing β€” and it was the acceptance criterion in the original brief. It was tested by accident, on the first real failure, and it passed.

The error was surfaced as an error, named as not ours, and no artifact was fabricated to accompany it.

βœ… And now the other half. On the successful run the app returned a path and the file behind it was real β€” 39,146 bytes, 9 distinct frames, pixel std 72.657 (Β§9.3). The invariant has now been exercised in both directions: no file claimed on failure, and a genuine file behind the claim on success. A build that only ever failed could not have shown the second half.


11. Follow-ups this run created

item status
1 Our error text tells users to read a traceback that does not exist (Β§3). In a kill, there is nothing to read, and that is the most informative fact β€” the copy should say so instead. ⏸️ open
2 ZEROGPU_DISK.md Β§4.2: upgrade the RSS prediction and the prune branch ⏸️ β†’ βœ… from this run. β†’ Applied in the 2026-08-25 docs pass β€” Β§4.2 now carries the dated upgrade annotation. βœ… resolved (2026-08-25)
3 OPERATOR_HANDOFF.md Β§3 and Β§5 are stale in both directions β€” Β§3 says the Space "has never run", Β§5 says a deploy is "not currently possible". Startup demonstrably works and generation demonstrably does not, and the document currently cannot express that distinction. β†’ Doubly stale itself by 2026-08-25: generation now demonstrably works (item 10 β€” 11 of 11), and the handoff was reworked to "one mode, once", which the mode matrix then superseded in turn. The handoff now carries dated annotations for both; this row is kept as the record of the gap as it stood after the first run. βœ… resolved (2026-08-25)
4 ⏸️ No readable ZeroGPU quota endpoint β€” all three candidate URLs 404, so billed seconds cannot be measured, only bounded. Worst case ~200 GPU-s of 2400/day for this run (xlarge bills at 2Γ—; packed 51.1 GB is over the 42.00 GiB threshold πŸ“„). ⏸️ open
5 Bus defect: lead-gate.mjs:1709 computes isLead = (session_id === SOMA_LEAD_SESSION_UUID). A /clear rotates the session id while the env var stays baked in from launch, so every soma bus tool is denied and the lead cannot file a report. Side effect: it also disarms the lead-only git push / git merge guard. ⏸️ open, infrastructure
6 ⏸️ No steady-state RSS measurement exists. peak_rss_gib() reports the load peak; the RAM question is about what remains resident when the load settles. Every RSS number in this document is a peak. This is the measurement that would have prevented the wrong prediction in §7 and it is still not taken. ⏸️ open
7 β›” The ~68 GB ceiling was inherited, never measured, and is now falsified (Β§6). Audit any other planning number with the same provenance β€” carried in from outside, never verified here, and load-bearing. It survived this long because it was consistent with our data, which is not the same as supported by it. ⏸️ open
8 ⚠️ 02fe943 STAYS β€” the retry succeeded, so the revert condition never triggered (Β§9.2). But it stands on a hypothesis this run killed and costs 31.24 GiB of peak RSS for a steady-state saving still unmeasured. Kept because it did not fail, not because it was shown to help. βœ… resolved, ⚠️ unjustified
9 β›” The fix is unattributed β€” a container rebuild changed alongside 02fe943, so n=1 vs n=1 with two variables (Β§9.3). Distinguishing them needs either a revert-and-retry on the same image, or route C. Until then no document should say the widening fixed generation. ⏸️ open
10 βœ… SUPERSEDED β€” all 11 modes have now run and generate, at the same 256x256x9. The mel front-end and the 16 kHz/48 kHz asymmetry have both been exercised on this platform and both passed. The matrix found 9 of 11 and one real defect β€” both video-INPUT modes, from a filepath str reaching LTX2VideoCondition(frames=…) β€” fixed at d2eb6f2 and βœ… re-verified on the Space at sha 287ba9c. Full evidence: docs/SPACE_MODE_MATRIX.md. ⏸️ Still one configuration only: 256x256x9, stage 2 off, no LoRA. βœ… resolved

12. Repo state at the time of writing

  • origin/soma/ltx25-studio is at 1802bfd β€” the six earlier commits were pushed by the orchestrator; the findings commits after it are unpushed. Pushing is the orchestrator's β€” this team does not push by design. Verified with git rev-list --count origin/soma/ltx25-studio..HEAD, which is the only authority on this; an earlier count taken from "what I authored this round" said four and silently omitted 51e1985, this document.
  • PR #1 is OPEN and draft: https://github.com/techfreakworm/ltx2.5-studio/pull/1
  • Offline gate at that tip: 2053 passed, 13 skipped, 0 failed.

Superseded as repo state (annotated 2026-08-25). origin/soma/ltx25-studio advanced to 17b589c β€” everything through the 11-of-11 findings is pushed (git rev-list --count = 0, verified in the 2026-08-25 audit). Commits since are local and queued for the next push: the fork-boundary pair 6bb211a (SPACE_MODE_MATRIX Β§4), five hardening fixes through f0daa75 (LoRA lifecycle, duration estimator, upload bound, desync guard 59cfa6e, mmap startup report), and this docs pass. PR #1 remains OPEN and draft. The lines above stand as the record of the state when this file was written.