LTX2.5-Studio / docs /OPERATOR_HANDOFF.md
techfreakworm's picture
docs: every sentence stays true after the deploy: README and the release notes say this is deploy 2's code and point to issue #32 for what the Space shows, and no doc says the Space still runs deploy 1, is not deployed, or has not been redeployed since August (#32)
abc9150 unverified
|
Raw History Blame Contribute Delete
68.9 kB
# Operator handoff β€” LTX-2.5 Studio > ## Status on 2026-11-16 > > This handoff is the August 2026 record. Its body is kept as written; it no longer says where > the project stands. Superseded since: > > - **Stage 2's "ceiling between 25 and 49 frames"** (the two blocks below): the 49-frame failure > was the 47.375 GiB `large` partition. 704Γ—480Γ—49 passes on `xlarge` (55.906 GiB reserved, > [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md) Β§2). The ceiling on `xlarge` and > stage 2's picture quality are not measured. > - **"Total day ≀ ~675.5 GPU-s"**: the quota ledger for UTC 2026-08-25 reads ≀ 1528.672 GPU-s > over 20 runner calls. Adding the ≀ ~93.3 GPU-s of manual calls bounded by hand in > [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md) Β§8 puts the day's bound at > ~1622 GPU-s (wall Γ— 2 upper bounds, not billed seconds), about 84 % of the 1920 stop line. > The evening's manual calls and all of 2026-08-26/27 were never recorded. > - **"The post-fix eleven-mode matrix goes in `SPACE_MODE_MATRIX.md` Β§10"**: it ran that > evening but was never written up, and its video and audio files are gone. > - **Β§3 "82.2 GB of 150 [OK]"**: read before ZeroGPU packs (the packed term was 0.0). On the > 2026-11-13 boot the platform's own progress bar read `40.6G` packed; the disk peak has never been > measured. > - **Β§5 "On a Space"**: "deploying is a push to that remote's `main`" is wrong as stated. The > Space repo has its own root commit, so pushing a work commit is rejected as non-fast-forward; > every deploy so far was a merge commit on the live Space head > (`git commit-tree <work>^{tree} -p <work> -p <Space head>`) pushed as a fast-forward. Never > `--force`. The 72.19 GB download happened at startup, not at build time: the 2026-11-13 cold > start took 13 min 59 s, about 11.5 min of it the download. `tests/test_deploy_drift.py` > does not assert `startup_duration_timeout`. > - **Where the weights come from** (Β§5; Β§3's "private-Space fetch path"; the deploy table's > `startup_duration_timeout` and token rows): the Space is public now, and the code has the > image bring the weights. `preload_from_hub` in the README's front matter fetches the 35 > model files the int8 Space loads (71.92 GB; the temporal upsampler is no longer fetched) and > the prompt enhancer's 8 files (2.49 GB) before the app starts. `preflight.ensure_weights()` > then checks the cache, downloads only what is missing, and refuses before writing a byte a > download the 150 GB disk could not hold. How long a cold start takes this way: deploy 2's > check on the Space, on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32). > - **Β§5 "On hinode"**: hinode is retired. `scripts/sync_to_hinode.sh`, `scripts/hinode_run.sh` > and every `scripts/host_*.py` were removed on 2026-11-16 (`git show 2f356b5:scripts/<name>` > restores one). > - **`auto` as the GPU size**: `LTX25_ZEROGPU_SIZE` now defaults to `xlarge` in code. > - **"local MPS bf16" sign-off**: those artifacts came from tooling outside this repo, which had > no Apple-silicon path. One was added on 2026-11-16 and verified on real weights (see > `README.md`). > > Where the project stands: [`../README.md`](../README.md) (what the app does, how to run it, > what it costs), [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) Β§10 (what has been > verified, and how) and [`RELEASE_NOTES.md`](RELEASE_NOTES.md) (what each deploy contains). > Deploy 1 (Space commit `0cd54f2`, the tree of `1e9fd20`, 2026-11-16) ran all eleven modes on > the Space, each looked at (`SPACE_MODE_MATRIX.md` Β§10.1); deploy 2 (2026-11-17) replaced it, > and its own check on the Space is reported on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32). > **What you have, what you do not have, and what to do next.** Written for > someone who was not here. [`PLAN.md`](PLAN.md) records how things were > decided; [`TEST_PLAN.md`](TEST_PLAN.md) records what was run; this tells you > where the project actually stands. **Every load-bearing number below is marked with where it came from:** βœ… *measured on hinode* Β· πŸ“„ *read off a pinned source or config* Β· ⚠️ *inferred* Β· ⏸️ *not yet established*. If a number carries no mark, treat that as a defect in this document. > ## β›” READ FIRST β€” 2026-08-25, evening: the deployed Space has NEVER produced a real image > > **Every generation result this document attributes to the Space was produced by > a broken transformer.** Every Space sha up to and including `ff142d9` ran > blanket bitsandbytes int8 with no `llm_int8_skip_modules`, so bnb quantised the > LTX-2.5 **conditioning path** β€” the eight top-level `LTX2AdaLayerNormSingle` > modulation heads and the in/out projections β€” along with everything else. The > denoiser stopped being steered by the timestep and the prompt, and every Space > artifact ever banked is **colour static or a flat field**: correctly shaped, > high variance, fully deterministic, and not a picture. > > βœ… **Fixed at `eb1aa9f`** (316 of 1660 Linears spared, names read off the real > checkpoint), **deployed as Space sha `e3b5333`**, and proven there β€” the same > seed, prompt and geometry that produced colour static now produce a > photographic frame. Before/after: > `~/.soma/leads/ltx25-studio/frames/COMPARE_before_after_skipfix.png` πŸ“„. > > **βœ… MECHANICAL claims survive, all of them**: all eleven modes execute on the > Space; both fork-boundary directions carry their cargo; determinism and > byte-reproducibility hold; tiled decode enables 1216Γ—704Γ—121; stage 2 runs and > doubles geometry with a ceiling between 25 and 49 frames; LoRA applies, > detaches, and both routes agree; the disk, RAM and quota arithmetic. > > **β›” OUTPUT claims do not survive, none of them.** Every "content-verified" > Space artifact in this file was verified **non-empty and non-frozen, never as > an image**. Specifically: the conditioning-fidelity conclusion (H4b), the LoRA > inertness check (H12b) and the stage-2 pass were all measured on garbage β€” they > show the code paths ran, not that the output was correct. > > **β›” `b78803cc` is byte-reproducible NOISE.** The determinism claim is intact > and its artifact is worthless. > > **β›” Β§2 is UNAFFECTED, and this is the load-bearing distinction in this > document.** hinode ran **bf16, unquantised** (Β§2.1's `base.dtype` row records > it), and the operator's only visual sign-off was on **local MPS bf16** > artifacts served over tailscale. Every bf16 path anyone has eyeballed looked > right; the only path ever eyeballed on the Space was broken. Β§3's discipline > was correct all along, and the annotations that merged the two evidence bases > were wrong: **they merged across a dtype boundary no run ever crossed.** > > **β›” The method failure, as policy.** Statistical gates cannot distinguish a > render from mush. They are a **pre-filter, never a verdict β€” they demote > artifacts, they never promote one.** No artifact is a PASS until a human or > model eye has seen a frame at native resolution, and every report claiming an > output is good carries the frame. This binds the structure detector added at > `e54477c` too: it FAILS the flat grey (SR 0.5298) and **PASSES colour static** > (SR 0.9183, against 0.9961–0.9968 for the real photograph), so even it cannot > promote. > > **β›” The cheap-config lesson.** The 256Γ—256Γ—9 config used for most of the > campaign produced output too small and too blobby for anyone to judge by eye. > That is how three weeks of confident nonsense got manufactured. **A config too > small to look at is a config that cannot be verified.** Β§2.2's own "Structure > is not content" and Β§4.1's `verify_streams`-exit-0-on-a-silent-file had both > already written down why β€” and "content-verified" was allowed to mean > non-empty bytes anyway. > > Full record: the invalidation blocks at the top of > [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) and > [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md). The post-fix > eleven-mode matrix goes in `SPACE_MODE_MATRIX.md` Β§10. > ## βœ… State as of 2026-08-25 β€” read [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md) > > A full GPU campaign ran on the Space that day and settled most of what this > document lists as open: **byte-level reproducibility sha256-VERIFIED** (three > containers, three code shas), **1216Γ—704Γ—121 generates** (tiled decode, > `cdb3211`), **stage 2 ran on the Space** (PASS at 704Γ—480Γ—25 input β†’ > 1408Γ—960; FAIL at Γ—49 β€” its ceiling is between those), **both LoRA > directions proven with a passing inertness check** (H12b's claim, > delivered), **conditioning shown to drive the output** (H4b's question, > answered by clip-swap), and the fork-boundary error fix **proven live in > both classes**. Total day ≀ ~675.5 GPU-s of 2400. > > β›” ***Corrected the same evening β€” see the block above.** Every run in that > campaign used the broken int8 build. The mechanical results stand exactly as > written (reproducibility, geometry, ceilings, error paths, cost). The two > OUTPUT claims in this paragraph do not: **"both LoRA directions proven with a > passing inertness check (H12b's claim, delivered)"** β€” the adapter changed one > garbage render into a different one, and H12b is NOT delivered β€” and > **"conditioning shown to drive the output (H4b's question, answered by > clip-swap)"** β€” both clip-swap outputs were colour static, and H4b is OPEN > again. H10b is not the last discriminator standing; all three are.* > > **What genuinely remains open after that session:** the stage-2 ceiling > (diffusion-decoder chunking β€” its own investigation), the `video_to_audio` > βˆ’19.5 dBFS observation (now consistent-with the conditioning finding, still > not directly probed β€” β›” *that "consistent-with" is withdrawn: it derived from > the conditioning finding, which is withdrawn; and βˆ’19.5 dBFS was audio > generated by a broken transformer*), the 2026-08-20 SIGKILL history > (documented-unexplained, by operator ruling) β€” and, still never exercised on > the platform: `auto_duration` and the **H10b** audio-input discriminator. --- ## 1. What this is A Gradio studio for **LTX-2.5** (Lightricks' 22B video+audio model) covering **all eleven generation modes**, with **user-uploadable LoRA validated at inference time**. One model load serves every mode. Python 3.12, `diffusers` pinned to an exact SHA, weights pinned to an exact revision. Destination is a private HuggingFace ZeroGPU Space. βœ… **It runs and generates there** β€” startup, one clean GPU worker, and a content-verified video, at Space sha `8fb292d6`. > β›” ***Corrected (2026-08-25, evening): "a content-verified video" was a > non-empty, non-frozen file.** It was colour static β€” `8fb292d6` ran the > blanket int8 that quantised the conditioning path. What stands: the Space > **runs**, startup completes, a GPU worker forks cleanly, and a well-formed > artifact comes back. What does not: that anything it returned was a video in > any sense a user means. Fixed at `eb1aa9f`, deployed `e3b5333`. Banner at the > top of this file.* ⏸️ **That is one mode, once.** Ten of the eleven have never executed on that platform; Β§2's eleven-of-eleven is a **hinode** result and stays one. Β§3 is the section that keeps those two evidence bases apart β€” read it before treating either as the other. > β›” ***The annotation below is itself corrected (2026-08-25, evening):** > "content-verified" meant non-empty and non-frozen, and sha `287ba9c` ran the > broken int8. All eleven modes **execute** on the Space β€” that half stands and > is the useful half. Nothing about their output does. Banner at the top.* > > βœ… **SUPERSEDED (run 2026-08-22, annotated 2026-08-25): all eleven modes now > generate on the live Space**, content-verified, at sha `287ba9c` β€” > [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) is the record. "One mode, once" > was true when written and is no longer the state: eleven-of-eleven is now > **also** a Space claim, ⏸️ at one configuration only (256Γ—256Γ—9, stage 2 off, > no LoRA, no `auto_duration`). Stage 2, LoRA, and every shippable resolution > remain hinode-only. Β§3's separation of the two evidence bases is what made > the merge legible β€” its annotation carries the detail. > > *Narrowed further on 2026-08-25: stage 2 (to its 25↔49-frame ceiling), LoRA > (both directions) and 1216Γ—704Γ—121 have now run on the Space too β€” see the > top-of-file note. Still hinode-only: the duration head.* --- ## 2. HOST-PROVEN Everything in this section ran on **hinode**: an **unpartitioned RTX PRO 6000 with 124 GB RAM** βœ… β€” through `get_pipeline` exactly as `app.py` calls it, not through a test harness. ### 2.1 Startup gates | Gate | Result | | |---|---|---| | Weights present | **37 / 37** files, 72.185 GB | βœ… | | Full repo, for contrast | 56 files, **164,192,096,189 B (164.19 GB)** | πŸ“„ manifest | | Components shared between the two pipelines | **9 / 9 identical by `is`** | βœ… | | `base.dtype` after deriving | **`torch.bfloat16`**, unchanged | βœ… | | CUDA cost of the second pipeline | **βˆ’1024 bytes** | βœ… | | Total for both pipelines | **65.370 GiB** | βœ… | The **βˆ’1024 bytes** is the number to re-check after any change to pipeline construction: it is what proves the second workflow shares the first's weights rather than loading its own copy. A positive figure in the tens of GB means the sharing has broken. ### 2.2 Modes Baseline is **704 Γ— 480 Γ— 121 frames**, seed pinned to **1234**. "Streams" is what `tests/verify_streams.py` observed in the container, not a human reading. | # | Mode | Result | Streams observed | Size | Time | |---|---|---|---|---|---| | H1 | `text_to_video` | βœ… | h264 704Γ—480@24, f=121 | 791,782 B | 8.4 s | | H2 | `image_to_video` | βœ… | h264 704Γ—480@24, f=121 | 307,583 B | 7.8 s | | H3 | `image_text_to_video` | βœ… | h264 704Γ—480@24, f=121 | 775,213 B | 8.0 s | | H4 | `video_to_video` | βœ… | h264 704Γ—480@24, f=121 | 708,885 B | 8.6 s | | H5 | `text_to_audio_video` | βœ… | h264 + **aac 48000 Hz 2 ch** | 792,803 B | 7.8 s | | H6 | `image_to_audio_video` | βœ… | h264 + aac 48000 Hz 2 ch | 374,184 B | 7.9 s | | H7 | `image_text_to_audio_video` | βœ… | h264 + aac 48000 Hz 2 ch | 995,843 B | 8.2 s | | H8 | `text_to_audio` | βœ… | **pcm_s16le 48000 Hz 2 ch**, no video | 961,998 B | 1.8 s | | H9 | `video_to_audio` | βœ… | pcm_s16le 48000 Hz 2 ch, no video | 961,998 B | 2.0 s | | H10 | `audio_to_audio` | βœ… | pcm_s16le 48000 Hz 2 ch, **βˆ’10.4 dBFS** | 961,998 B | 2.8 s | | H11 | `audio_to_video` | βœ… | h264 704Γ—480@24, f=121, no audio | 193,825 B | 8.0 s | **Eleven of eleven.** All βœ… measured on the real `get_pipeline` path, no bypass and no harness accommodation; `verify_streams` returned 0 on every one. | beyond the baseline | | | |---|---|---| | **H14** stage 2 | 1408 Γ— 960, f=121 | 1,807,646 B Β· 35.2 s | | **H15** `duration_head` | 704 Γ— 480, **f=97** not 121 | 593,527 B Β· 5.9 s | | **H16** high resolution | **1216 Γ— 704** + aac stereo | 1,775,743 B Β· 21.0 s | | **H17** mel round-trip | 4.000 s in β†’ 4.010 s out, **4/4 tones** | strict gate | | **H12 / H13a / H13b** | LoRA accepted / rejected / rejected | stage 6.5 | > **Wall-clock excludes the model load**, which is **14.0 s once** per batch βœ…. > ### ⚠️ Structure is not content > > Every run here passed `verify_streams`. **The one defect that mattered most > was invisible to it**: `video_to_audio` came back 50 dB below its siblings > with correct codec, channels, rate, duration and a non-empty file. **Silence > and a healthy signal have identical stream layout.** > > It was found because someone measured RMS on an artifact that had already > been signed off, and it led to a defect affecting **all twelve modes** (gap > 7). A green table says these are well-formed files. **It is a weaker claim > than "correct output", and the gap between the two is where this build spent > its afternoon.** > > H8 and H9 are byte-identical in size because uncompressed PCM of equal > duration always is. That is why size proves nothing about content, and why it > took an RMS measurement to notice. > > **On audio levels generally.** Measured levels sort by *condition type*, not > by mode βœ… β€” text-only βˆ’5.2 to βˆ’6.2 dBFS, image-conditioned βˆ’8.2 to βˆ’15.2, > video-conditioned at strength 0.8 βˆ’54.5 to βˆ’55.0. The text-only group peaks at > exactly **1.0000 with no samples exceeding**, so the export is faithful and > **the saturation is upstream, in the model, not in our writer**. Those files > are consistent with each other and sit at the rail; ⏸️ whether that is correct > for this model has not been established. ### 2.5 What the recipe fix bought β€” before / after Same seed, same geometry on both sides of every pair; **only the distilled recipe (gap 7) differs.** | # | mode | before | after | wall-clock | |---|---:|---:|---:|---| | H1 | `text_to_video` | 886,265 | 791,782 | 25.7 β†’ **8.4 s** | | H2 | `image_to_video` | 432,486 | 307,583 | 26.3 β†’ **7.8 s** | | H3 | `image_text_to_video` | 1,130,178 | 775,213 | 26.6 β†’ **8.0 s** | | H4 | `video_to_video` | 727,490 | 708,885 | 27.0 β†’ **8.6 s** | | H5 | `text_to_audio_video` | 1,091,468 | 792,803 | 25.0 β†’ **7.8 s** | | H6 | `image_to_audio_video` | 620,537 | 374,184 | 26.6 β†’ **7.9 s** | | H7 | `image_text_to_audio_video` | 784,282 | **995,843** | 26.7 β†’ **8.2 s** | | H8 | `text_to_audio` | 961,998 | 961,998 | 5.6 β†’ **1.8 s** | | H9 | `video_to_audio` | 961,998 | 961,998 | 6.0 β†’ **2.0 s** | | H14 | stage 2 | 2,254,799 | 1,807,646 | 84.3 β†’ **35.2 s** | | H15 | `duration_head` | 684,374 | 593,527 | 19.4 β†’ **5.9 s** | | H16 | 1216 Γ— 704 | 2,331,528 | 1,775,743 | 68.7 β†’ **21.0 s** | **Geometry is identical on both sides of every pair** β€” H14 stays 1408 Γ— 960, H15 stays 97 frames, the rest 704 Γ— 480 Γ— 121. *Sizes changed; geometry did not* is the shape of the result. Three things a reader needs: - **H7 is the only artifact that grew.** h264 size tracks content complexity, so that is neither good nor bad on its own. Flagged, not interpreted. - **H8 and H9 are byte-identical** because uncompressed PCM at one duration, rate and channel count always is. **Their change is entirely in level**, not in bytes β€” see below β€” so reading this table alone would call them unchanged when they are two of the three that moved most. - **~3Γ— faster everywhere.** At `guidance_scale=1.0` with STG and modality zeroed there are no classifier-free branches, so it is one forward pass per step instead of two or three. **This is the one column here that seeding does not confound** β€” wall-clock is not drawn from a distribution β€” and it follows from the mechanism rather than merely accompanying it. > ### ⚠️ These pairs are not yet controlled β€” the audio evidence is > > **The seed never reached the model** until `4ebb8b9` (gap 8), so both halves > of every pair above were unconditionally random. **The pixel differences > cannot evidence that the recipe fix caused them** β€” two runs of one mode > differed anyway. βœ… Seeding is fixed, and verified byte-identical **for > `audio_to_audio`**. > > β›” **The seeded gallery that was to supersede this table is DROPPED.** It lived > on a host that no longer exists, and nothing replaced it. **So this table is > permanently unattributable, not temporarily so** β€” these figures remain as the > record of what the defect cost, and that is now all they will ever be. > > **The audio levels do stand on their own**, because they are systematic and > orders of magnitude outside run-to-run variation: > > | | before | after | > |---|---:|---:| > | H9 `video_to_audio` | βˆ’55.0 dBFS | **βˆ’17.8 dBFS** | > | H8 `text_to_audio` | βˆ’5.2 dBFS, **peak 1.0000** (railed) | **βˆ’15.3 dBFS**, peak 0.6953 | > | highest peak, any mode | **1.7811** | **1.0050** | > > A 50 dB gap and a clipped peak going away are not a seed. > > β›” **And the replacement will not arrive.** Bitwise reproducibility is verified > for `audio_to_audio`. The **video** path has since been *evidenced* on the > Space β€” a run returned bytes identical to one banked the previous day on a > different container at the same seed β€” but ⏸️ **evidenced is not proven**: no > two-run frame-hash comparison exists. See `docs/SPACE_MODE_MATRIX.md` Β§5. The > gallery that was to close this is dropped with the host it lived on. > > β›” *Correction (2026-08-25) on "returned bytes identical": that overstates > what was measured. The cross-day match was **file size** (39,146 B) plus > **every decoded statistic to three decimals** β€” no hash of the day-1 artifact > was ever recorded, and that file is gone, so byte identity against it is > permanently unverifiable. The "evidenced is not proven" hedge above was > right, and even understated: no byte comparison existed either. The planned > close is a same-seed re-run sha256-compared against the surviving 2026-08-22 > artifact β€” `SPACE_MODE_MATRIX.md` Β§5's correction carries the hash.* > ### βœ… Sizes differed between the two code paths β€” almost certainly just the seed > > Three modes were run through a **bypass** (`_base_pipeline` plus an injected > `StageRunner`) and through the **real** `get_pipeline` path, **all pre-fix**: > > | | bypass | real path | > |---|---:|---:| > | H1 | 865,447 B | 886,265 B | > | H2 | **577,051 B** | **432,486 B** β€” 25% smaller | > | H3 | 1,040,007 B | 1,130,178 B | > > This was recorded as an open question β€” *"GPU non-determinism, or a genuine > difference between the routes?"* β€” and described as *"at the same seed."* > > **It was not at the same seed. The seed reached nothing** (gap 8). These are > six unconditionally random runs, and two runs of one mode differed by this > much anyway. **There was never an anomaly to explain**, and the phrase *"at > the same seed"* was doing all the work: it made an ordinary difference look > like a finding. > > ⏸️ Cheap to settle properly now β€” run one mode both ways at a fixed seed. > Nobody has, so it is *probably* the seed rather than certainly. > > The advice it produced stands on its own and is worth more than the question: > **assert stream layout, never an exact size.** ### 2.3 Beyond the baseline | Run | What it proves | Result | |---|---|---| | **H16** high resolution | usability, not just correctness | βœ… `text_to_audio_video` at **1216 Γ— 704** @24, f=121 + aac 48000 Hz 2 ch β€” 1,775,743 B, **21.0 s** | | **H15** `duration_head` | the model chose the length, rather than a default leaking through | βœ… **f=97**, not 121 β€” 704 Γ— 480, 593,527 B, 5.9 s | | **H14** stage 2 | upsample β†’ 3-step schedule β†’ diffusion decode | βœ… **1408 Γ— 960**, f=121 β€” 1,807,646 B, 35.2 s | *Post-fix figures, matching Β§2.2. Β§2.5 carries the pre-fix column.* H15 is worth understanding: `num_frames` is *omitted* and the duration head picks. Had it returned 121 β€” the default β€” the run would have looked identical to a pass while proving the opposite. **97 is the whole result.** **H14's proof is its resolution, not a comparison.** βœ… 1408 Γ— 960 against H1's 704 Γ— 480 is **exactly 2Γ— in both dimensions** β€” a size only the spatial upsampler can produce, and **the one claim here that no vintage or seed can affect**, since geometry is identical across every before/after pair. That is stronger evidence than the same-seed diff the matrix planned, because a same-seed comparison shows *a* difference while this shows *the specific* difference stage 2 exists to make. ### 2.4 The LoRA validator, both directions | Run | Adapter | Result | |---|---|---| | **H12** | `joyfox/LTX-2.3-Transition-LORA` | βœ… **ACCEPTED**, **576 modules** across transformer + connectors | | **H13a** | `charlied98/…` `lora_weights_step_03000` | βœ… **REJECTED at stage 6.5**, `LoRADimensionError` | | **H13b** | same repo, `comfy_lora_weights_step_03000` | βœ… **REJECTED at stage 6.5**, same mechanism | H12 carries **mixed dimensions** β€” 4096 at attention, 2048 at `audio_ff`, 16384 at `ff.net.2` βœ… β€” and none of that is a mismatch. An earlier "expect hidden dim 4096" rule would have **rejected a perfectly good adapter**; the false-positive guard is what stops that. H13a and H13b differ only in key prefix (`transformer.` vs `diffusion_model.`) and reach the same verdict by the same mechanism, which is what makes the rejection trustworthy rather than lucky. --- ## 3. SPACE-PROVEN vs HOST-PROVEN β€” read this before believing Β§2 > ### βœ… The Space runs and generates. ⏸️ That is **one mode, once.** > > `text_to_video` at 256Γ—256Γ—9 produced a real, content-verified video on the > live private Space βœ…. **Ten of the eleven modes have never executed there.** > > β›” ***Corrected (2026-08-25, evening): it was not a real video. It was colour > static** β€” the origin claim of the entire Space evidence base, and it was > wrong from the first run. "Content-verified" meant non-empty and non-frozen. > What survives: the Space runs, and one mode executed there. Banner at the > top.* > > A green table on a dedicated host and a working deployment are different > claims. **Both have now been made β€” about different things.** That is exactly > why this section still exists: the two evidence bases have diverged rather > than merged, and Β§2's eleven-of-eleven remains a **hinode** claim. > βœ… **SUPERSEDED (run 2026-08-22, annotated 2026-08-25).** The mode matrix ran > on the live Space and **all eleven modes generate there, content-verified**, > at sha `287ba9c` β€” [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md). The two > evidence bases have **merged at exactly one configuration**: 256Γ—256Γ—9, > stage 2 off, no LoRA, no `auto_duration`. > > β›” ***THE MERGE NEVER HAPPENED (corrected 2026-08-25, evening).** hinode ran > **bf16, unquantised** β€” Β§2.1's `base.dtype` row records exactly that β€” and the > Space ran blanket int8. The two evidence bases were separated by a dtype > boundary no run ever crossed, and that difference was the whole defect. **This > section's original discipline was right and this annotation was the error:** > it merged on "the same mode name at the same geometry" when the thing that > differed was the quantisation of the conditioning path. Keep Β§3. Banner at the > top.* What is still hinode-only: stage 2 > (1408Γ—960), the duration head, LoRA in both directions, the mel round-trip at > full length, and every shippable resolution. This section's discipline is > unchanged β€” it now keeps *configurations* apart instead of *platforms*. > > *That hinode-only list shrank again on 2026-08-25: stage 2 reached 1408Γ—960 > on the Space (from ≀25-frame input; its decoder ceiling is between 25 and 49 > frames), LoRA ran in both directions, and 1216Γ—704Γ—121 passed via the tiled > decode. Remaining hinode-only: the duration head, and full-length audio > work. `SPACE_SESSION_2026-08-25.md`.* **Detailed record: [`SPACE_GENERATION_FINDING.md`](SPACE_GENERATION_FINDING.md)** β€” the first deploy, the killed workers, the falsified ceiling, and the run that generated. Not restated here on purpose. hinode is an **unpartitioned** card with **124 GB RAM** and an ordinary filesystem. ZeroGPU is none of those things. **The first deploy answered several of these questions and changed the answer to one of them:** - βœ… **Tensor packing** β€” exercised. Packing ran and the post-pack prune fired, freeing 65.25 GB. - βœ… **The 150 GB ephemeral-disk ceiling** β€” never the binding constraint. The disk line read **82.2 GB of 150 [OK]** while the workload died of something else. Arithmetic in [`ZEROGPU_DISK.md`](ZEROGPU_DISK.md). - β›” **RAM β€” the ~68 GB figure is FALSIFIED.** See the finding below; this is the single most important correction in this document. - βœ… **The private-Space fetch path** β€” works. 37/37 files at the pinned revision, downloaded at runtime because `preload_from_hub` 401s on a private repo. - βœ… **The fork boundary** β€” crossed. A worker forked, ran, and released cleanly (`fail=false` β†’ 200). - βœ… **The `size` kwarg's *plumbing*** β€” the name survived `gpu_decorator_kwargs`'s signature filter and raised **no import-time `TypeError`**. That is the risk PLAN Β§5.8 actually named β€” the platform pins its own `spaces` and lags PyPI β€” so this is a real βœ… **on its own terms**. - β›” **An explicit `size` value β€” never passed, and THAT WAS THE BUG.** **β†’ RESOLVED (2026-08-27), and the inference above was wrong.** `app.py` sent `size=None` whenever `LTX25_ZEROGPU_SIZE` was `auto`, which was its default and what the Space ran. The reasoning recorded here β€” *"ZeroGPU chose `xlarge` by inference, packed 51.1 GB exceeds the 42.00 GiB threshold"* β€” was the fourth document in this repo to assert it and none of the four had followed the promoted value past the function that sets it. It never reaches the scheduler: `default_gpu_size()` is consumed only by `get_duration_seconds` (`client.py:109`) and feeds only the duration multiplier, while `gpuSize` is sent to the platform only when the *decorator's* size is not None (`api.py:112-113`). βœ… **`auto` therefore ran the Space on `large` β€” 47.375 GiB β€” for its entire life, instead of `xlarge`'s 94.971 GiB.** Measured from inside the worker, both before and after. `LTX25_ZEROGPU_SIZE` is now `xlarge`. Full trace: [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md) Β§1. ⚠️ Note the exact shape of the error, because it is the interesting part: the *reservation* really was computed as `xlarge` (no 1.5Γ— `duration_factor`). The hardware was `large`. We were priced as one and allocated the other. - ⏸️ **Quota metering** β€” still unreadable. All three candidate endpoints 404, so billed seconds can only be **bounded**, never measured. - ⏸️ **Ten of eleven modes** β€” no conditioning, no stage 2, no LoRA, and **no audio path** has run on ZeroGPU. The audio modes exercise the mel front-end, the `av` loader and the 16 kHz/48 kHz asymmetry; none of that has executed there once. β€” *Annotation (2026-08-25): mostly falsified by the 2026-08-22 mode matrix. Image, audio and video conditioning **have** run on ZeroGPU; both audio-input modes passed; the mel front-end and the 16 kHz/48 kHz asymmetry were exercised there and passed (`SPACE_GENERATION_FINDING.md` Β§11 item 10, `SPACE_MODE_MATRIX.md`). Still true, and only these: **stage 2, LoRA and `auto_duration` have never run on ZeroGPU.*** β€” *β›” And that annotation is corrected in turn (2026-08-25, evening): "passed" meant a well-formed, non-empty, non-frozen file from a transformer whose conditioning path had been quantised away. Those paths **executed** on ZeroGPU β€” the mel front-end, the `av` loader, the 16 kHz/48 kHz asymmetry all really ran, and that half stands. The ⏸️ this replaced was cautioning about something real, and it should not have been retired on the strength of "passed". Banner at the top.* --- ### The three findings from the first deploy > #### 1. β›” The **~68 GB** RAM ceiling is **FALSIFIED** > > It was a community cgroup observation β€” **inherited, never ours**, and > load-bearing in planning since before any deploy. Startup reached > **87.13 GiB = 93.56 GB** peak RSS and **was not killed**. > > βœ… **Replace it with our own measured lower bound: the ceiling is above > 93.56 GB.** Every headroom figure computed against ~68 GB was wrong by at > least 25.56 GB. > #### 2. ⚠️ Quantising **more** raised peak RSS by **31.24 GiB** > > Adding the text encoder to `components_to_quantize` moved RSS **up**, from > 55.89 to 87.13 GiB, in a change made to bring it down. `safetensors` mmaps > lazily; **quantisation materialises those weights as anonymous memory**, so > quantising a component destroys the laziness that was keeping it cheap. > > This is the **same single mechanism** `ZEROGPU_DISK.md` Β§4.2 already used to > settle two other questions, now deciding a third β€” in the opposite direction > from what we predicted. > > β›” ***Annotation (2026-08-25, evening): this is the change that broke > generation, and this document records it as a RAM result and nothing else.** > Widening `components_to_quantize` is the blanket int8 that quantised the > conditioning path; from here until `eb1aa9f` the Space produced colour static > in every mode. The RSS measurement above is correct and unaffected. What is > missing β€” and was missing everywhere, `README.md`'s int8 tier note included β€” > is any sentence connecting quantisation to output fidelity. Nobody wrote one, > so nobody looked.* > #### 3. ⏸️ The `SIGKILL` is **still unexplained** > > cgroup-OOM is **dead** as an explanation: with the ceiling above 93.56 GB > there was never the narrow margin the hypothesis required. βœ… **Route C β€” > instrument the worker β€” is now the LEADING path, not the fallback.** It is the > only route that returns evidence rather than another inference, and every > inference made about this kill has been wrong at least once. > #### β›” And the fix is **UNATTRIBUTED** > > Two things changed between the failing and succeeding runs: the quantisation > widening **and a full container rebuild**. n=1 against n=1 with two variables. > **"The Space now serves" is established; "the widening is why" is not** β€” and > the hypothesis that motivated the widening is itself dead. Do not let any > document claim the widening fixed generation. > > βœ… ***VINDICATED, in the strongest available direction (2026-08-25, evening).** > That last sentence was written as a caution against over-crediting the > widening. The widening **broke** generation: blanket int8 quantised the > conditioning path, and every Space artifact from here to `eb1aa9f` is colour > static. "The Space now serves" was established and was exactly as narrow as > this paragraph insisted.* **Cost so far: ≀ 243.58 GPU-seconds** of 2400/day ⚠️ *a bound, not a measurement* β€” quota endpoint 404s. > **Superseded (annotated 2026-08-25).** That figure covered the > first-generation session only. The mode-matrix day β€” UTC quota day > **2026-08-21** β€” closed at **≀ ~413 GPU-s** of 2400 (`SPACE_MODE_MATRIX.md` > Β§7, matrix plus re-run; still a bound, never a measurement). Quota days are > per-UTC-date, so as of 2026-08-25 the day is fresh and neither number > describes current consumption. **One hinode number with a direct ZeroGPU consequence.** Stage 2 peaked at **83.897 GiB** βœ…, against 70.497 GiB for a single-stage run. So **stage 2 cannot run on a `large` 48 GB MIG slice at all**, and on `xlarge` (96 GB) it leaves roughly **11 GiB** before any packing overhead. That is measured on an unpartitioned card; the packing overhead is ⏸️ unknown and comes off that margin. One live risk that Part A *has* surfaced: the stage-2 diffusion decoder resolves a **NATTEN kernel from the Hub at generation time** πŸ“„, which is the only network dependency that fires *after* the user presses Generate. Whether it caches per image or per cold worker is ⏸️ unknown and decides its severity. --- ## 4. Known gaps, stated plainly 0. **~~The `size` kwarg has never carried a value.~~ β›” CLOSED, AND IT WAS THE MOST EXPENSIVE ITEM IN THIS LIST (2026-08-27).** The gap was written as *"if anyone sets `LTX25_ZEROGPU_SIZE` to force a size, they are exercising an untested path"* β€” with the unstated assumption that leaving it at `auto` was the safe, tested default. It was the opposite. `auto` sends **no** size, and ZeroGPU's own promotion never reaches the scheduler, so `auto` **was** the `large` branch: 47.375 GiB. βœ… The note above about `large` being a slice "that Β§3's own stage-2 measurement says cannot hold stage 2 at all" was therefore describing the partition the Space was already running on β€” and it predicted R4b's failure precisely, a day before anyone connected the two. `LTX25_ZEROGPU_SIZE=xlarge` is now set and βœ… verified from inside the worker at 94.971 GiB. Stage 2 at 704Γ—480Γ—49 passes on it: 53.08 s, 1408Γ—960, peak 55.906 GiB reserved. See [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md). 1. **H10 / H11 β€” the audio-input modes β€” have not passed.** ⏸️ The path they need was connected to nothing until very recently: the UI collected a file, the encoder existed and was exported, and **nothing called it**. It is built and tested now, and it has not yet produced an artifact on the card. **The acceptance test is H10b**, not H10: two `noise_scale` values at the same pinned seed. *Identical audio means the input is still not reaching the model*, whatever the stream check says β€” because `noise_scale=1.0` discards the input by design, so a bug that drops it entirely is indistinguishable from correct operation. β€” *Annotation (2026-08-25): the first sentence was overtaken twice and this paragraph was never updated. Both modes then passed on the card β€” Β§2.2's own rows record H10 βœ… (961,998 B at βˆ’10.4 dBFS) and H11 βœ… (193,825 B) β€” and both passed again on the live Space (2026-08-22, `SPACE_MODE_MATRIX.md`). What stands is the acceptance-test framing: **H10b has still never run**, so "the user's audio reaches the model" remains ⏸️ unestablished; everything that has passed is consistent with the input being silently discarded.* 2. **Stage 2 works; its NATTEN fetch is characterised.** βœ… **Resolved once per process, not per generation** β€” two stage-2 generations in one process, one kernel fetch. So a Space needs outbound network at a worker's **first** stage-2 use rather than on every press of Generate. Still a real dependency, because **each cold worker pays it once**. Retained below for the mechanism. βœ… H14 ran and produced 1408 Γ— 960. But the diffusion decoder resolves its NATTEN kernel through a **version β†’ ref lookup that goes over the network even when the kernel files are cached** πŸ“„, and **whether that fires once per process or once per generation is ⏸️ unknown**. It cannot be grepped after the fact β€” the run log contains **zero** natten mentions β€” so answering it needs deliberate instrumentation. On a private Space this is the 401 class again, except firing *after* the user presses Generate. 3. **`connectors: 0` on every accepted adapter.** βœ… observed. Stage 7 asserts *total* modules attached > 0, which passes on the transformer alone β€” so an adapter that *ought* to touch `connectors` and silently does not would still be reported as attached. Probably correct for the three adapters tested (2.3/2B-era), so **not currently a defect** β€” but the assertion as written cannot distinguish "legitimately unused" from "silently missed". A per-component expectation would close it. See PLAN Β§4.4. 4. **`temporal_latent_upsampler` has no known role.** πŸ“„ It is the same class as the spatial upsampler, differing only by axis, and upstream's component table and both documented recipes mention only the spatial one. We download it (0.2619 GB) and do **not** wire it in. Not a defect; an open question. 5. ### βœ… `video_to_audio` near-silence β€” **fixed, and the first diagnosis was wrong** H9 came back at **βˆ’55.0 dBFS** against `text_to_audio`'s βˆ’5.2 βœ…. **The cause was the distilled recipe reaching only the stage-2 path (item 7).** With that corrected, H9 runs at **βˆ’17.8 dBFS** β€” 2.5 dB from its siblings β€” at the **unchanged** condition strength of 0.8. | | before | after | |---|---:|---:| | H9 `video_to_audio` | βˆ’55.0 dBFS, peak 0.0146 | **βˆ’17.8 dBFS**, peak 0.5078 | | H8 `text_to_audio` | βˆ’5.2 dBFS, **peak 1.0000** (railed) | **βˆ’15.3 dBFS**, peak 0.6953 | | gap | **~50 dB** | **2.5 dB** | **One mode 50 dB down and another driven into the rail were the same misconfiguration seen from opposite ends.** ⚠️ **The first diagnosis blamed condition strength and was wrong.** A single-lever sweep at fixed seed gave 0.8 β†’ βˆ’55.0 and 0.5 β†’ βˆ’5.1, and that was read as the cause. It was a real interaction **between condition strength and the broken guidance** β€” a genuine effect inside a pipeline that no longer exists. A per-mode strength cap was written, committed and reverted. > **A single-lever same-seed sweep is strong evidence that something moves, > and no evidence about why.** The source citation said throughout that a video condition *cannot* attenuate audio in the denoise loop β€” the conditioning mask is applied to `noise_pred_video` and never touches `noise_pred_audio`. It was correct, and a clean correlation was believed over it. 6. **The audio input rate is 16000, the vocoder output is 48000.** πŸ“„ They are not symmetric. Anything that assumes one rate for both directions will pitch-shift audio without erroring. 7. ### βœ… The distilled recipe reached only the stage-2 path β€” **fixed** Eleven of twelve runs sent **no sigmas at all**, so upstream built its own schedule and five guidance parameters took their **SFT** defaults β€” `audio_guidance_scale` at **7.0** against an intended 1.0. ``` sent: [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125] intended: [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875] ``` βœ… All twelve artifacts differed before and after, with magnitudes ordering by conditioning strength (text-only 37–59 mean |diff|, image-conditioned 15–16, video-conditioned **3.71**). ⏸️ **Neither of those establishes the fix caused them, and the ordering is not the corroboration it looks like.** Both halves were unseeded (gap 8), so two runs of one mode differed anyway β€” and a **more strongly conditioned mode is more constrained, so its run-to-run variance is smaller too.** The ordering is exactly what random variation would produce. It cannot separate the two explanations, and an earlier draft of this section claimed it could. **What does stand, unconfounded:** - **the timings.** 25.7 β†’ 8.4 s is not run-to-run variation, and it follows directly from the mechanism: unguided means one forward pass per step instead of two or three. **The defect cost 3Γ— the compute**, and that is measured rather than inferred. - **the audio levels and peaks** β€” βˆ’55.0 β†’ βˆ’17.8 dBFS, highest peak 1.7811 β†’ 1.0050. Systematic and orders of magnitude outside variation. - πŸ“„ **that the schedule and guidance were wrong at all**, which is a source fact and needs no run. **It cost 3Γ— the compute as well as the quality.** At `guidance_scale=1.0` with STG and modality zeroed there are no classifier-free branches, so it is one forward pass per step rather than two or three: | | before | after | |---|---:|---:| | H1 `text_to_video` | 25.7 s | **8.4 s** | | H8 `text_to_audio` | 5.6 s | **1.8 s** | | H14 stage 2 | 84.3 s | **35.2 s** | | H16 **1216Γ—704 + stereo** | 68.7 s | **21.0 s** | Clipping went with it β€” highest peak **1.7811 β†’ 1.0050** βœ… β€” so no limiter is needed and `outputs._write_audio`'s *"what comes back is what the vocoder produced"* principle stands. **Our path never added gain; the material was arriving at Β±1.0.** Stage 2's brightness falloff was this too, not stage 2: mean `99.48 β†’ 34.44` collapsing before, `82.52 β†’ 110.47` stable after. 8. ### βœ… The seed reached only the sidecar β€” **fixed, and verified byte-identical** `_resolve_seed` ran and the number went into the encoder metadata. **There was no generator anywhere in the package**, so every run was unconditionally random and every *"same seed"* comparison was two independent runs. βœ… **Fixed** in `4ebb8b9`: a generator seeded from the request, built on the pipeline's own device, threaded to the pipeline **and** to `audio_vae.encode`'s `.sample()`. βœ… **Verified on the card, for `audio_to_audio`.** Two runs at identical seed and configuration: ``` run-a1d35aaaab06/output.wav md5 2988f01159a5322afb28abb6e9a73d26 run-b9dde07ed8bb/output.wav md5 2988f01159a5322afb28abb6e9a73d26 ``` βœ… And the fix was complete when it landed, not repaired afterwards: `git log -S 'generator' 4804da1..e4e8df1 -- ltx25_studio/` returns `4ebb8b9` alone. > **On this path, `seed` determines the output byte-for-byte, on GPU, with no > deterministic-algorithm flags set.** Stronger than expected β€” the weaker > *"seed-consistent but not bit-reproducible"* was drafted and never needed. ⏸️ **Scope, and it matters for Β§2.5:** that measurement is **one mode**. The **video path is not shown to be bitwise reproducible** β€” it is *evidenced* on the Space (byte-identical output across two days and two containers at one seed, plus `text_to_audio_video` matching `text_to_video`'s video statistics to three decimals) and no two-run frame-hash comparison has ever been made. `docs/SPACE_MODE_MATRIX.md` Β§5 states that limit explicitly. β€” *Correction (2026-08-25): "byte-identical output across two days" is too strong β€” the cross-day evidence is **size-identical plus statistics-identical**; no day-1 hash exists and the day-1 file is gone, so byte identity was never checked and now cannot be. The conclusion of this passage β€” not shown to be bitwise reproducible β€” is unchanged and was the correct one. `SPACE_MODE_MATRIX.md` Β§5's correction records the planned sha256-compared re-run against the surviving 2026-08-22 artifact.* β›” **The twelve-mode pixel-diff gallery that depended on it is DROPPED** β€” the host it ran on is gone. **So the consequence is permanent rather than pending**: the Β§2.5 comparison table was measured *before* seeding worked, its pixel diffs cannot attribute the change to the recipe fix, and nothing is coming to replace it. Everything structural was never affected β€” H14's exact 2Γ— resolution, H15's 97 frames, H17's frequency recovery, and every audio level. > The lever that should have caught it was ours: `decode_kwargs` took a > `generator` argument, **with a docstring explaining why the diffusion > decoder must be seeded**, and read a key nothing ever set. See PLAN Β§9 > rule 12. 9. ### ⏸️ `noise_scale` and `sigmas[0]` disagree β€” open, deliberately unfixed πŸ“„ Upstream infers one from the other β€” *"first (largest) sigma value if using custom sigmas"* β€” and our `stage_2_kwargs` honours it. The audio-input modes do not: they send `noise_scale` while `sigmas[0]` stays 1.0. **It was diagnosed as the cause of a silent `audio_to_audio` and it was not** (Β§4.1). A fix was written, committed and **withdrawn**: truncating the schedule at `noise_scale` takes an eight-step run to **two** steps at 0.5 and **one** at 0.1 β€” values that measured healthy on eight. **Agreeing with a contract is not evidence that a change improves anything.** A test pins the current behaviour and explains why, so the next attempt arrives with a measurement rather than with agreement. --- ## 4.1 ⚠️ Known limitation β€” unusual audio input produces a silent file **Not a defect we can properly fix, and the most likely way a user meets a disappointing result.** `audio_vae.encode` is **never called by the pipeline**. Our `encode_waveform_to_latents` is the only producer of those latents, and **nothing anywhere validates their distribution** β€” not upstream, not us. So audio that is far from what the model was trained on encodes to latents the model cannot use, and the result is **a structurally perfect, inaudible file with no error and nothing to act on.** βœ… A real instance, and it cost a day of misdiagnosis: a two-tone synthetic fixture at **βˆ’14.8 dBFS in** produced **βˆ’87.9 dBFS out**, with correct codec, channels, rate and duration, and `verify_streams` exit 0. The same mode fed a real recording produces **βˆ’10.4 dBFS**. **Likely triggers:** pure tones, sine sweeps, test signals, near-silence, heavily synthesised material. **Ordinary recorded audio is fine** β€” every realistic input tried has worked. β›” **Do not "fix" this with an out-of-distribution detector.** Reliable OOD detection at that boundary is not something this project can do, and a detector that is wrong in either direction is worse than the limitation. ⏸️ **A narrower thing is worth doing and is deliberately not done yet.** We measure the input level and can measure the output level, so **a file that comes back essentially silent could say so** in `GenerationResult.warnings`. The decision that makes it honest is that it must be a **measurement, not a diagnosis**: *"this output is βˆ’87.9 dBFS, effectively silent"* is a fact about the artifact, checkable and cause-free. *"Your audio was out of distribution"* is a claim we cannot support. It is not built because it needs a threshold, a threshold needs evidence, and the PR opens with **one** named defect. **Adding an unvalidated heuristic to the export path hours beforehand is the same move as shipping a schedule change on a contract reading** β€” which was reverted for that reason. It belongs in the next pass, with a measured floor. --- ## 5. How to run it ### On hinode ```bash ./scripts/sync_to_hinode.sh # ships git archive HEAD, not the working tree ``` `rsync` is deliberately not used: it is not installed on the host and there is no sudo to add it. Needs an HF token with access to the gated model repo. Then `scripts/hinode_run.sh`, or `scripts/host_matrix.py` for the full matrix. **Do not set `HF_HUB_OFFLINE=1`.** Construction loads from the local snapshot directory, so nothing is fetched at load time β€” and stage 2 needs outbound network for the NATTEN kernel. Forcing offline breaks the second without helping the first. ### Locally, with no GPU The whole test suite runs with **no GPU, no weights and no network**. That is a deliberate property, not a coincidence, and it is worth preserving: it is what makes it possible to catch a defect before it costs a run. ```bash python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]' .venv/bin/python -m pytest -q ``` `torch`, `diffusers` and `torchaudio` are **not** needed for the suite. `av` and `numpy` are, because they are how real containers get written and read. ### On a Space βœ… **This works.** The repo *is* the Space β€” `README.md`'s YAML front matter is the Space config, and the `space` remote points at it: ```bash git remote -v | grep space # space https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio ``` Deploying is a push to that remote's `main`. β›” **This team does not push** β€” not to `origin`, not to `space`. The orchestrator does it; that is a deliberate boundary, not a missing capability. **What the deploy depends on, all of it already load-bearing:** | | | |---|---| | `app_file: app.py`, `sdk: gradio 6.24.0`, `python_version: "3.12"` | πŸ“„ front matter | | `suggested_hardware: zero-a10g` | πŸ“„ **a legacy id string β€” not an A10G.** Changing it moves the Space onto paid dedicated hardware | | `startup_duration_timeout: 1h` | πŸ“„ **required.** The default is 30 min and this Space downloads 72.19 GB at build time. When it blows, the failure looks like something else entirely | | an HF token secret with access to the gated model repo | πŸ“„ weights fetch at **runtime**, because `preload_from_hub` 401s on a private Space | `tests/test_deploy_drift.py` asserts these rather than trusting them β€” run the suite before any deploy and the drift shows up locally instead of as a 40-minute build that dies at the end. **Reading the startup log is the cheapest diagnostic available**, and it costs zero GPU quota because packing and loading happen at module scope, before the first `@spaces.GPU` call. The load line names its own configuration: ``` quantised transformer, text_encoder as int8 | peak RSS 0.91 -> 87.13 GiB ``` ⚠️ **That peak is the number to watch.** It is a *peak across the load*, not a steady state β€” we have never measured steady state, and the RAM questions are about steady state. See [`SPACE_GENERATION_FINDING.md`](SPACE_GENERATION_FINDING.md) Β§7. --- ## 6. The lessons worth carrying Not a diary. These are the ones that would change how the next person works. ### A fake derived from the mapping can never catch a wrong mapping The audio VAE's six parameters were read from `audio_vae.config` **by names that do not exist there** πŸ“„ β€” four live under different keys, two are instance attributes absent from `config.json` entirely. Every runtime path through that module raised on real weights, and every offline test passed, because the fake used *our* names. It agreed with the code and with nothing else. The instinctive repair β€” build the fake from the same mapping the code uses β€” is a trap for exactly the same reason. The fix is a **recorded fact about reality**: the real config's key set, checked into the tests with its provenance. The fake checks semantics; the recorded keys check the world. ### The fixture that makes a test possible is itself untested by that test Three instances in one day, all in tests that were otherwise excellent: 1. `test_encoder_contract.py` drove the **real** encoder through **real** containers to a real file β€” and built its audio as `np.float32`. The model produces `bfloat16`. **Six of eleven modes were broken at the last step and 1926 tests passed** βœ… *both counts measured β€” the modes on hinode, the tests in this repo at the commit before the fix*. 2. The replacement fake's `.float()` returned an `ndarray` where torch returns a *tensor*. Written the convenient way, it would have passed a broken fix. 3. The audio-VAE fake used our own key names β€” the case above. The mitigation that worked: **check the fake against the real thing.** In (2) that check failed within a minute of existing. ### A search that cannot see the claim reports that the claim is gone Three defects in this build share one shape: **the instrument returned a confident negative it was structurally incapable of turning positive.** - **A wrapped line.** Β§7 said *"The Space has evicted twice and has never run"* as one line. Β§1 said the same thing **split across a line break**, so a grep for the phrase found the occurrences that fit on one line and pronounced the audit clean. - **A shell modifier.** `"$c:ltx25_studio/..."` is parsed by zsh as the `:l` lowercase modifier, so `git show` got a mangled path and **every** commit answered *"constant absent"* β€” including the one that demonstrably contains it. - **A stale marker.** A `⚠️ (inherited)` pointing at a section that had been rewritten to say **FALSIFIED** still scanned as *merely unverified*. **None of the three raised an error.** Each returned a clean, plausible, wrong answer, and two of them nearly closed an audit. The defence is not a better search string. It is **running the instrument against a case whose answer you already know**: if the check cannot go positive on a known positive, its negatives carry no information. The zsh case was caught only because *"absent at HEAD too"* was impossible on its face. ### A claim moving known β†’ unknown is nearly always a regression Documents accrete facts and rarely lose them. When an edit is about to record something as uncertain, search first for it being *known* β€” the likeliest explanation is that someone established it and the current author has not read it. This has since caught its own inverse: a correction issued against this document's history that the history did not actually contain. ### Checks over comments Twice, a constraint written down did not survive the next function written by its own author β€” a `dtype` requirement violated two commits after the ⚠️ comment demanding it, and a 44.33 GB refetch reintroduced *while reading the docstring that predicts it*. A comment is a message to a future stranger; it has no purchase on the author five minutes later. **Make the wrong thing unsayable instead:** derive the value once and share it, so the two callers cannot drift. ### The instruments encode the failure you imagined, not the failures available `host-runner`'s generalisation, in their words: > *all three instruments encoded the failure I imagined rather than the failures > available* β€” a `pkill -f` pattern that matched the killer, a wait loop with no failure exit, and a fixture guard that handled a fixture being *absent* but not one that *raises*. **The three had different authors, including the lead.** That is the point of recording it: it is not one person's blind spot, it is what happens whenever the person who imagines the failure is the person who writes the check for it. ### We can prove a function ran. We cannot prove it ran with the right arguments. **The gap this build ends on**, and the one worth closing next. `DISTILLED_SIGMA_VALUES` was defined, documented as mandatory, quoted in its module's docstring, and **referenced** β€” so every check we own passed. It was referenced from `stage_1_kwargs()`, which `StageRunner` applies **only when stage 2 is enabled**. Eleven of twelve host runs were generated with upstream's linear fallback and its **SFT** guidance defaults instead: ``` sent: [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125] intended: [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875] ``` A reference check passes. A reachability check at module level passes. Even the function-level sweep passes, unless someone thought to name that particular function. **What nothing checked was the arguments the pipeline was actually called with.** The pattern that does work, used three times here, is to **record the vendor's specification independently and assert against that** β€” `MODEL_CARD_DISTILLED`, `REAL_AUDIO_VAE_CONFIG_KEYS`, `REAL_VOCODER_CONFIG_KEYS`. Each is a fact copied from the source of truth rather than derived from our own constants, so changing ours cannot quietly redefine what correct means. Without that independence the test agrees with the code and with nothing else. > **The escalation ladder, in the order this build climbed it:** is the module > imported β†’ is it *reached by the path that does the work* β†’ did this specific > *function* run β†’ **were the arguments the ones the vendor specifies.** Only > the last rung is still hand-built, one call site at a time. ### A module can be reached and still contain a dead function The reachability sweep that exists to catch importer-count-zero missed one, for two independent reasons: the tracer recorded which **file** ran and discarded which **function** β€” the name was in the same frame object β€” and the probe drove **one mode**, so a path reachable from no other mode was invisible by construction. **Either fix alone would still have missed it.** "Sharpen the instrument" was the obvious response and would not have worked. ### The one time source-reading and measurement confirmed each other Every other pairing today went the other way: one corrected the other, and the lesson was about which to trust when they disagreed. Once, they agreed. A reproducibility control failed β€” two runs at one seed, 8.7% apart β€” and the obvious reading was an unseeded draw somewhere. Instead of hunting, **every noise source in the pinned pipeline was enumerated**: `_create_noised_state`, the conditioning mask, `prepare_*_latents`, the decode timestep, the condition VAE encode. All threaded the generator; the encode used `sample_mode="argmax"` and was deterministic regardless. **Source said it should already be reproducible.** Rather than write either sentence β€” *"the seed is attached to nothing"* or *"we are seed-consistent but not bit-reproducible"* β€” a scoping run was requested. Before it ran, the control was re-checked: **it had been measured against a tree that predated the seeding fix.** At the tip, two runs are byte-identical. > **The true claim was stronger than either draft**: on `audio_to_audio`, `seed` > determines the output byte-for-byte, on GPU, with no deterministic-algorithm > flags. Both drafted sentences were weaker *and* wrong, and both would have > shipped if the reluctance to write an unmeasured claim had not held for one > more exchange. **And then the corrected sentence was written without its scope, twice more in this document** β€” as a property of the build rather than of one mode. ⏸️ The video path is still unverified. Recording that here because the lesson is not *"be careful"*: it is that **a claim narrowed in one place does not narrow itself in the others**, and a document is exactly where that goes wrong. Β§4 gap 8 carries the scope; Β§2.5 and this paragraph each had to be found and fixed separately. **Refusing to write a sentence you cannot yet support is the same discipline as refusing to change a threshold to make a test green** β€” and it is the cheaper of the two, because nobody has to notice you did it. ### Staffing β€” two makers and a lead who verifies **Not five, and not one.** This build ran briefly with five teammates and settled at two: one implementer owning the code, one runner owning the hardware. The three stood down were not underused β€” they were **a coordination cost with no matching throughput**, and the collision they produced (two agents assigned `outputs.py`; a cross-boundary change landing under a commit message about other files) cost more than their output was worth. Two makers with **disjoint, stated file ownership** is the shape that worked, and the line held all day under pressure: the runner found four product defects and edited none of them. **The scarce resource was execution on real hardware, not review capacity.** At the point the suite reported **1926 passing** βœ…, six of eleven modes were broken, the audio-input path was connected to nothing, stage 2 was unreachable, and every runtime path through `audio.py` raised on real weights. None of that was found by reviewing harder β€” it was found by an afternoon on a real GPU. **Another reviewer would have added agreement, not information.** Budget for one more agent is better spent on a second executor with *different* hardware than on a second reviewer. **A lead who cannot edit is forced to verify by observation rather than by reading their own diff**, and that constraint turned out to be a feature. Every claim the lead checked was checked by running something β€” `git merge-base --is-ancestor` rather than trusting a commit message, `av` on the container rather than trusting a verifier's exit code, `pgrep` on the box rather than trusting a status report. That caught the box silently trailing the tip **three times**, a stale bypass premise, and a wait loop that had frozen a teammate. A lead who could edit would have been tempted to fix those, and would have stopped looking. **The default-REJECT verifier role was filled by reality rather than by an agent.** The plan called for an independent verifier holding a default-REJECT stance; in practice the hardware held it. It rejected the `from_pipe` cast, the bf16 encode, the missing audio parameters and the unreachable stage 2 without being asked, and without any possibility of being talked round. **Where a real gate exists, prefer it to a reviewer agent β€” an agent can be persuaded and a `RuntimeError` cannot.** **What it cost.** Message-crossing: three times a blocking item was buried inside a long message and lost a cycle behind work already in flight. The fix is mechanical and worth adopting from the start β€” **one blocker, one message, nothing else in it.** Long messages are for context; short ones are for priority. --- ## 7. If you do one thing **Re-run the comparisons seeded.** Everything works; the open item is that some of the *evidence* was gathered before the seed reached the model. ### The runs that need repeating, and what each would then establish | run | what it would establish | why it does not yet | |---|---|---| | **Β§2.5 before/after pairs** | that the recipe fix caused the pixel differences | both halves were unconditionally random | | **H10b** β€” `noise_scale` 0.1 vs 0.9 | that the user's audio reaches the model | same, plus the audio encode drew unseeded | | **H12b** β€” LoRA effect | that attaching the adapter changed the output | same | | **H4b** β€” `strength` 0.3 vs 0.9 | that the conditioning clip reaches the model | same | Add a **same-value control** to each β€” two runs at *identical* settings. Under a working seed its spread should be **zero**, and that is what turns "the halves differ" into evidence rather than a coin toss. An earlier control measured 1.8% and was read as a bound on the variance; **an unseeded control cannot bound anything**, which is the mistake worth not repeating. > β›” **The first row of that table is dead (annotated 2026-08-25), and this > file already said so elsewhere.** The Β§2.5 before/after pairs cannot be > re-run: the seeded gallery is dropped with its host, and Β§2.5/Β§4 gap 8 record > that as *permanent* β€” the table is unattributable forever, not pending. The > live content of this instruction is the other three rows: **H10b, H12b and > H4b are still runnable** β€” on the Space (none needs stage 2; each is one > same-seed pair at 256Γ—256Γ—9-class cost) or on a future host β€” and each still > establishes exactly what the table says. > > *Two of the three were delivered on the Space later on 2026-08-25: H12b's > claim (the adapter changed the output β€” `20a3a84b…` β‰  `b78803cc…` at the > same seed) and H4b's question (the conditioning clip reaches the model β€” > answered by clip-swap: outputs 17.277 vs 0.864, mirroring their inputs, > against a byte-identical control). **H10b is the last discriminator > standing.** `SPACE_SESSION_2026-08-25.md` Β§6–§7.* > > β›” ***BOTH DELIVERIES ARE WITHDRAWN (2026-08-25, evening).** Every run cited > there used the broken int8 build. H12b's two hashes differ because one garbage > render differs from another; H4b's 17.277-vs-0.864 compared the motion energy > of two colour statics. The mechanical readings stand β€” the adapter injection > path ran, the conditioning path carries a scalar, the noise floor at a fixed > seed is exactly zero β€” and **all three discriminators are open again: H10b, > H12b and H4b.** They are also worth spending on now, because there is finally > a build whose output is a picture (`e3b5333`).* ### What is already earned, and does not need repeating **Eleven of eleven modes produce artifacts with the expected stream layout on the real path.** βœ… H14's exact 2Γ— resolution, H15's 97 frames, H17's round-trip recovering 4/4 tones under a strict gate, both LoRA directions, and every audio level β€” all structural or systematic, and none of them depends on seeding. **And read Β§3 before treating any of this as a deployment.** βœ… The Space now runs and generates β€” ⏸️ **for one of the eleven modes.** Everything in this section is a **hinode** result; the Space has proven `text_to_video` at 256Γ—256Γ—9 and nothing else. Those are different claims about different machines, and Β§3 is where they are kept apart. β€” *β›” Corrected (2026-08-25, evening): the Space had proven that `text_to_video` **executes**; its output was colour static. **This paragraph's instinct β€” keep Β§2's hinode results apart from anything the Space says β€” was the correct one**, and every later annotation that softened it was wrong. Everything in this section is hinode, bf16, unquantised, and unaffected.* > β›” ***And that supersession is itself corrected (2026-08-25, evening):** > "content-verified" meant non-empty and non-frozen, on the broken int8 build. > All eleven modes **execute** on the Space; none of their output was ever > verified as an image. Banner at the top of this file.* > > βœ… **Superseded (annotated 2026-08-25): the Space has since proven all eleven > modes at 256Γ—256Γ—9** β€” content-verified, sha `287ba9c`, 2026-08-22 > (`SPACE_MODE_MATRIX.md`). What it has still proven nothing about: stage 2, > LoRA, `auto_duration`, and any resolution anyone would ship. Β§3's annotation > carries the merged state. > > *And later on 2026-08-25 that residue shrank to `auto_duration` alone (plus > the stage-2 ceiling's exact position): stage 2, LoRA and 1216Γ—704Γ—121 all > ran on the Space that day. Top-of-file note; `SPACE_SESSION_2026-08-25.md`.*
# Operator handoff β€” LTX-2.5 Studio
> ## Status on 2026-11-16
>
> This handoff is the August 2026 record. Its body is kept as written; it no longer says where
> the project stands. Superseded since:
>
> - **Stage 2's "ceiling between 25 and 49 frames"** (the two blocks below): the 49-frame failure
> was the 47.375 GiB `large` partition. 704Γ—480Γ—49 passes on `xlarge` (55.906 GiB reserved,
> [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md) Β§2). The ceiling on `xlarge` and
> stage 2's picture quality are not measured.
> - **"Total day ≀ ~675.5 GPU-s"**: the quota ledger for UTC 2026-08-25 reads ≀ 1528.672 GPU-s
> over 20 runner calls. Adding the ≀ ~93.3 GPU-s of manual calls bounded by hand in
> [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md) Β§8 puts the day's bound at
> ~1622 GPU-s (wall Γ— 2 upper bounds, not billed seconds), about 84 % of the 1920 stop line.
> The evening's manual calls and all of 2026-08-26/27 were never recorded.
> - **"The post-fix eleven-mode matrix goes in `SPACE_MODE_MATRIX.md` Β§10"**: it ran that
> evening but was never written up, and its video and audio files are gone.
> - **Β§3 "82.2 GB of 150 [OK]"**: read before ZeroGPU packs (the packed term was 0.0). On the
> 2026-11-13 boot the platform's own progress bar read `40.6G` packed; the disk peak has never been
> measured.
> - **Β§5 "On a Space"**: "deploying is a push to that remote's `main`" is wrong as stated. The
> Space repo has its own root commit, so pushing a work commit is rejected as non-fast-forward;
> every deploy so far was a merge commit on the live Space head
> (`git commit-tree <work>^{tree} -p <work> -p <Space head>`) pushed as a fast-forward. Never
> `--force`. The 72.19 GB download happened at startup, not at build time: the 2026-11-13 cold
> start took 13 min 59 s, about 11.5 min of it the download. `tests/test_deploy_drift.py`
> does not assert `startup_duration_timeout`.
> - **Where the weights come from** (Β§5; Β§3's "private-Space fetch path"; the deploy table's
> `startup_duration_timeout` and token rows): the Space is public now, and the code has the
> image bring the weights. `preload_from_hub` in the README's front matter fetches the 35
> model files the int8 Space loads (71.92 GB; the temporal upsampler is no longer fetched) and
> the prompt enhancer's 8 files (2.49 GB) before the app starts. `preflight.ensure_weights()`
> then checks the cache, downloads only what is missing, and refuses before writing a byte a
> download the 150 GB disk could not hold. How long a cold start takes this way: deploy 2's
> check on the Space, on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32).
> - **Β§5 "On hinode"**: hinode is retired. `scripts/sync_to_hinode.sh`, `scripts/hinode_run.sh`
> and every `scripts/host_*.py` were removed on 2026-11-16 (`git show 2f356b5:scripts/<name>`
> restores one).
> - **`auto` as the GPU size**: `LTX25_ZEROGPU_SIZE` now defaults to `xlarge` in code.
> - **"local MPS bf16" sign-off**: those artifacts came from tooling outside this repo, which had
> no Apple-silicon path. One was added on 2026-11-16 and verified on real weights (see
> `README.md`).
>
> Where the project stands: [`../README.md`](../README.md) (what the app does, how to run it,
> what it costs), [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) Β§10 (what has been
> verified, and how) and [`RELEASE_NOTES.md`](RELEASE_NOTES.md) (what each deploy contains).
> Deploy 1 (Space commit `0cd54f2`, the tree of `1e9fd20`, 2026-11-16) ran all eleven modes on
> the Space, each looked at (`SPACE_MODE_MATRIX.md` Β§10.1); deploy 2 (2026-11-17) replaced it,
> and its own check on the Space is reported on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32).
> **What you have, what you do not have, and what to do next.** Written for
> someone who was not here. [`PLAN.md`](PLAN.md) records how things were
> decided; [`TEST_PLAN.md`](TEST_PLAN.md) records what was run; this tells you
> where the project actually stands.
**Every load-bearing number below is marked with where it came from:**
βœ… *measured on hinode* Β· πŸ“„ *read off a pinned source or config* Β·
⚠️ *inferred* · ⏸️ *not yet established*.
If a number carries no mark, treat that as a defect in this document.
> ## β›” READ FIRST β€” 2026-08-25, evening: the deployed Space has NEVER produced a real image
>
> **Every generation result this document attributes to the Space was produced by
> a broken transformer.** Every Space sha up to and including `ff142d9` ran
> blanket bitsandbytes int8 with no `llm_int8_skip_modules`, so bnb quantised the
> LTX-2.5 **conditioning path** β€” the eight top-level `LTX2AdaLayerNormSingle`
> modulation heads and the in/out projections β€” along with everything else. The
> denoiser stopped being steered by the timestep and the prompt, and every Space
> artifact ever banked is **colour static or a flat field**: correctly shaped,
> high variance, fully deterministic, and not a picture.
>
> βœ… **Fixed at `eb1aa9f`** (316 of 1660 Linears spared, names read off the real
> checkpoint), **deployed as Space sha `e3b5333`**, and proven there β€” the same
> seed, prompt and geometry that produced colour static now produce a
> photographic frame. Before/after:
> `~/.soma/leads/ltx25-studio/frames/COMPARE_before_after_skipfix.png` πŸ“„.
>
> **βœ… MECHANICAL claims survive, all of them**: all eleven modes execute on the
> Space; both fork-boundary directions carry their cargo; determinism and
> byte-reproducibility hold; tiled decode enables 1216Γ—704Γ—121; stage 2 runs and
> doubles geometry with a ceiling between 25 and 49 frames; LoRA applies,
> detaches, and both routes agree; the disk, RAM and quota arithmetic.
>
> **β›” OUTPUT claims do not survive, none of them.** Every "content-verified"
> Space artifact in this file was verified **non-empty and non-frozen, never as
> an image**. Specifically: the conditioning-fidelity conclusion (H4b), the LoRA
> inertness check (H12b) and the stage-2 pass were all measured on garbage β€” they
> show the code paths ran, not that the output was correct.
>
> **β›” `b78803cc` is byte-reproducible NOISE.** The determinism claim is intact
> and its artifact is worthless.
>
> **β›” Β§2 is UNAFFECTED, and this is the load-bearing distinction in this
> document.** hinode ran **bf16, unquantised** (Β§2.1's `base.dtype` row records
> it), and the operator's only visual sign-off was on **local MPS bf16**
> artifacts served over tailscale. Every bf16 path anyone has eyeballed looked
> right; the only path ever eyeballed on the Space was broken. Β§3's discipline
> was correct all along, and the annotations that merged the two evidence bases
> were wrong: **they merged across a dtype boundary no run ever crossed.**