Spaces:
Running on Zero
Running on Zero
docs: every sentence stays true after the deploy: README and the release notes say this is deploy 2's code and point to issue #32 for what the Space shows, and no doc says the Space still runs deploy 1, is not deployed, or has not been redeployed since August (#32)
abc9150 unverified |
Download docs/OPERATOR_HANDOFF.md from techfreakworm/LTX2.5-Studio: direct link, hf CLI and curl.
- Browser
- Download file 68.9 kB
-
https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio/resolve/main/docs/OPERATOR_HANDOFF.md
- Command line
-
hf download hf://spaces/techfreakworm/LTX2.5-Studio/docs/OPERATOR_HANDOFF.md
-
curl -L -o OPERATOR_HANDOFF.md https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio/resolve/main/docs/OPERATOR_HANDOFF.md
68.9 kB
# Operator handoff β LTX-2.5 Studio
> ## Status on 2026-11-16
>
> This handoff is the August 2026 record. Its body is kept as written; it no longer says where
> the project stands. Superseded since:
>
> - **Stage 2's "ceiling between 25 and 49 frames"** (the two blocks below): the 49-frame failure
> was the 47.375 GiB `large` partition. 704Γ480Γ49 passes on `xlarge` (55.906 GiB reserved,
> [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md) Β§2). The ceiling on `xlarge` and
> stage 2's picture quality are not measured.
> - **"Total day β€ ~675.5 GPU-s"**: the quota ledger for UTC 2026-08-25 reads β€ 1528.672 GPU-s
> over 20 runner calls. Adding the β€ ~93.3 GPU-s of manual calls bounded by hand in
> [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md) Β§8 puts the day's bound at
> ~1622 GPU-s (wall Γ 2 upper bounds, not billed seconds), about 84 % of the 1920 stop line.
> The evening's manual calls and all of 2026-08-26/27 were never recorded.
> - **"The post-fix eleven-mode matrix goes in `SPACE_MODE_MATRIX.md` Β§10"**: it ran that
> evening but was never written up, and its video and audio files are gone.
> - **Β§3 "82.2 GB of 150 [OK]"**: read before ZeroGPU packs (the packed term was 0.0). On the
> 2026-11-13 boot the platform's own progress bar read `40.6G` packed; the disk peak has never been
> measured.
> - **Β§5 "On a Space"**: "deploying is a push to that remote's `main`" is wrong as stated. The
> Space repo has its own root commit, so pushing a work commit is rejected as non-fast-forward;
> every deploy so far was a merge commit on the live Space head
> (`git commit-tree <work>^{tree} -p <work> -p <Space head>`) pushed as a fast-forward. Never
> `--force`. The 72.19 GB download happened at startup, not at build time: the 2026-11-13 cold
> start took 13 min 59 s, about 11.5 min of it the download. `tests/test_deploy_drift.py`
> does not assert `startup_duration_timeout`.
> - **Where the weights come from** (Β§5; Β§3's "private-Space fetch path"; the deploy table's
> `startup_duration_timeout` and token rows): the Space is public now, and the code has the
> image bring the weights. `preload_from_hub` in the README's front matter fetches the 35
> model files the int8 Space loads (71.92 GB; the temporal upsampler is no longer fetched) and
> the prompt enhancer's 8 files (2.49 GB) before the app starts. `preflight.ensure_weights()`
> then checks the cache, downloads only what is missing, and refuses before writing a byte a
> download the 150 GB disk could not hold. How long a cold start takes this way: deploy 2's
> check on the Space, on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32).
> - **Β§5 "On hinode"**: hinode is retired. `scripts/sync_to_hinode.sh`, `scripts/hinode_run.sh`
> and every `scripts/host_*.py` were removed on 2026-11-16 (`git show 2f356b5:scripts/<name>`
> restores one).
> - **`auto` as the GPU size**: `LTX25_ZEROGPU_SIZE` now defaults to `xlarge` in code.
> - **"local MPS bf16" sign-off**: those artifacts came from tooling outside this repo, which had
> no Apple-silicon path. One was added on 2026-11-16 and verified on real weights (see
> `README.md`).
>
> Where the project stands: [`../README.md`](../README.md) (what the app does, how to run it,
> what it costs), [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) Β§10 (what has been
> verified, and how) and [`RELEASE_NOTES.md`](RELEASE_NOTES.md) (what each deploy contains).
> Deploy 1 (Space commit `0cd54f2`, the tree of `1e9fd20`, 2026-11-16) ran all eleven modes on
> the Space, each looked at (`SPACE_MODE_MATRIX.md` Β§10.1); deploy 2 (2026-11-17) replaced it,
> and its own check on the Space is reported on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32).
> **What you have, what you do not have, and what to do next.** Written for
> someone who was not here. [`PLAN.md`](PLAN.md) records how things were
> decided; [`TEST_PLAN.md`](TEST_PLAN.md) records what was run; this tells you
> where the project actually stands.
**Every load-bearing number below is marked with where it came from:**
β
*measured on hinode* Β· π *read off a pinned source or config* Β·
β οΈ *inferred* Β· βΈοΈ *not yet established*.
If a number carries no mark, treat that as a defect in this document.
> ## β READ FIRST β 2026-08-25, evening: the deployed Space has NEVER produced a real image
>
> **Every generation result this document attributes to the Space was produced by
> a broken transformer.** Every Space sha up to and including `ff142d9` ran
> blanket bitsandbytes int8 with no `llm_int8_skip_modules`, so bnb quantised the
> LTX-2.5 **conditioning path** β the eight top-level `LTX2AdaLayerNormSingle`
> modulation heads and the in/out projections β along with everything else. The
> denoiser stopped being steered by the timestep and the prompt, and every Space
> artifact ever banked is **colour static or a flat field**: correctly shaped,
> high variance, fully deterministic, and not a picture.
>
> β
**Fixed at `eb1aa9f`** (316 of 1660 Linears spared, names read off the real
> checkpoint), **deployed as Space sha `e3b5333`**, and proven there β the same
> seed, prompt and geometry that produced colour static now produce a
> photographic frame. Before/after:
> `~/.soma/leads/ltx25-studio/frames/COMPARE_before_after_skipfix.png` π.
>
> **β
MECHANICAL claims survive, all of them**: all eleven modes execute on the
> Space; both fork-boundary directions carry their cargo; determinism and
> byte-reproducibility hold; tiled decode enables 1216Γ704Γ121; stage 2 runs and
> doubles geometry with a ceiling between 25 and 49 frames; LoRA applies,
> detaches, and both routes agree; the disk, RAM and quota arithmetic.
>
> **β OUTPUT claims do not survive, none of them.** Every "content-verified"
> Space artifact in this file was verified **non-empty and non-frozen, never as
> an image**. Specifically: the conditioning-fidelity conclusion (H4b), the LoRA
> inertness check (H12b) and the stage-2 pass were all measured on garbage β they
> show the code paths ran, not that the output was correct.
>
> **β `b78803cc` is byte-reproducible NOISE.** The determinism claim is intact
> and its artifact is worthless.
>
> **β Β§2 is UNAFFECTED, and this is the load-bearing distinction in this
> document.** hinode ran **bf16, unquantised** (Β§2.1's `base.dtype` row records
> it), and the operator's only visual sign-off was on **local MPS bf16**
> artifacts served over tailscale. Every bf16 path anyone has eyeballed looked
> right; the only path ever eyeballed on the Space was broken. Β§3's discipline
> was correct all along, and the annotations that merged the two evidence bases
> were wrong: **they merged across a dtype boundary no run ever crossed.**
>
> **β The method failure, as policy.** Statistical gates cannot distinguish a
> render from mush. They are a **pre-filter, never a verdict β they demote
> artifacts, they never promote one.** No artifact is a PASS until a human or
> model eye has seen a frame at native resolution, and every report claiming an
> output is good carries the frame. This binds the structure detector added at
> `e54477c` too: it FAILS the flat grey (SR 0.5298) and **PASSES colour static**
> (SR 0.9183, against 0.9961β0.9968 for the real photograph), so even it cannot
> promote.
>
> **β The cheap-config lesson.** The 256Γ256Γ9 config used for most of the
> campaign produced output too small and too blobby for anyone to judge by eye.
> That is how three weeks of confident nonsense got manufactured. **A config too
> small to look at is a config that cannot be verified.** Β§2.2's own "Structure
> is not content" and Β§4.1's `verify_streams`-exit-0-on-a-silent-file had both
> already written down why β and "content-verified" was allowed to mean
> non-empty bytes anyway.
>
> Full record: the invalidation blocks at the top of
> [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) and
> [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md). The post-fix
> eleven-mode matrix goes in `SPACE_MODE_MATRIX.md` Β§10.
> ## β
State as of 2026-08-25 β read [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md)
>
> A full GPU campaign ran on the Space that day and settled most of what this
> document lists as open: **byte-level reproducibility sha256-VERIFIED** (three
> containers, three code shas), **1216Γ704Γ121 generates** (tiled decode,
> `cdb3211`), **stage 2 ran on the Space** (PASS at 704Γ480Γ25 input β
> 1408Γ960; FAIL at Γ49 β its ceiling is between those), **both LoRA
> directions proven with a passing inertness check** (H12b's claim,
> delivered), **conditioning shown to drive the output** (H4b's question,
> answered by clip-swap), and the fork-boundary error fix **proven live in
> both classes**. Total day β€ ~675.5 GPU-s of 2400.
>
> β ***Corrected the same evening β see the block above.** Every run in that
> campaign used the broken int8 build. The mechanical results stand exactly as
> written (reproducibility, geometry, ceilings, error paths, cost). The two
> OUTPUT claims in this paragraph do not: **"both LoRA directions proven with a
> passing inertness check (H12b's claim, delivered)"** β the adapter changed one
> garbage render into a different one, and H12b is NOT delivered β and
> **"conditioning shown to drive the output (H4b's question, answered by
> clip-swap)"** β both clip-swap outputs were colour static, and H4b is OPEN
> again. H10b is not the last discriminator standing; all three are.*
>
> **What genuinely remains open after that session:** the stage-2 ceiling
> (diffusion-decoder chunking β its own investigation), the `video_to_audio`
> β19.5 dBFS observation (now consistent-with the conditioning finding, still
> not directly probed β β *that "consistent-with" is withdrawn: it derived from
> the conditioning finding, which is withdrawn; and β19.5 dBFS was audio
> generated by a broken transformer*), the 2026-08-20 SIGKILL history
> (documented-unexplained, by operator ruling) β and, still never exercised on
> the platform: `auto_duration` and the **H10b** audio-input discriminator.
---
## 1. What this is
A Gradio studio for **LTX-2.5** (Lightricks' 22B video+audio model) covering
**all eleven generation modes**, with **user-uploadable LoRA validated at
inference time**. One model load serves every mode. Python 3.12, `diffusers`
pinned to an exact SHA, weights pinned to an exact revision.
Destination is a private HuggingFace ZeroGPU Space. β
**It runs and generates
there** β startup, one clean GPU worker, and a content-verified video, at Space
sha `8fb292d6`.
> β ***Corrected (2026-08-25, evening): "a content-verified video" was a
> non-empty, non-frozen file.** It was colour static β `8fb292d6` ran the
> blanket int8 that quantised the conditioning path. What stands: the Space
> **runs**, startup completes, a GPU worker forks cleanly, and a well-formed
> artifact comes back. What does not: that anything it returned was a video in
> any sense a user means. Fixed at `eb1aa9f`, deployed `e3b5333`. Banner at the
> top of this file.*
βΈοΈ **That is one mode, once.** Ten of the eleven have never executed on that
platform; Β§2's eleven-of-eleven is a **hinode** result and stays one. Β§3 is the
section that keeps those two evidence bases apart β read it before treating
either as the other.
> β ***The annotation below is itself corrected (2026-08-25, evening):**
> "content-verified" meant non-empty and non-frozen, and sha `287ba9c` ran the
> broken int8. All eleven modes **execute** on the Space β that half stands and
> is the useful half. Nothing about their output does. Banner at the top.*
>
> β
**SUPERSEDED (run 2026-08-22, annotated 2026-08-25): all eleven modes now
> generate on the live Space**, content-verified, at sha `287ba9c` β
> [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) is the record. "One mode, once"
> was true when written and is no longer the state: eleven-of-eleven is now
> **also** a Space claim, βΈοΈ at one configuration only (256Γ256Γ9, stage 2 off,
> no LoRA, no `auto_duration`). Stage 2, LoRA, and every shippable resolution
> remain hinode-only. Β§3's separation of the two evidence bases is what made
> the merge legible β its annotation carries the detail.
>
> *Narrowed further on 2026-08-25: stage 2 (to its 25β49-frame ceiling), LoRA
> (both directions) and 1216Γ704Γ121 have now run on the Space too β see the
> top-of-file note. Still hinode-only: the duration head.*
---
## 2. HOST-PROVEN
Everything in this section ran on **hinode**: an **unpartitioned RTX PRO 6000
with 124 GB RAM** β
β through `get_pipeline` exactly as `app.py` calls it, not
through a test harness.
### 2.1 Startup gates
| Gate | Result | |
|---|---|---|
| Weights present | **37 / 37** files, 72.185 GB | β
|
| Full repo, for contrast | 56 files, **164,192,096,189 B (164.19 GB)** | π manifest |
| Components shared between the two pipelines | **9 / 9 identical by `is`** | β
|
| `base.dtype` after deriving | **`torch.bfloat16`**, unchanged | β
|
| CUDA cost of the second pipeline | **β1024 bytes** | β
|
| Total for both pipelines | **65.370 GiB** | β
|
The **β1024 bytes** is the number to re-check after any change to pipeline
construction: it is what proves the second workflow shares the first's weights
rather than loading its own copy. A positive figure in the tens of GB means the
sharing has broken.
### 2.2 Modes
Baseline is **704 Γ 480 Γ 121 frames**, seed pinned to **1234**. "Streams" is
what `tests/verify_streams.py` observed in the container, not a human reading.
| # | Mode | Result | Streams observed | Size | Time |
|---|---|---|---|---|---|
| H1 | `text_to_video` | β
| h264 704Γ480@24, f=121 | 791,782 B | 8.4 s |
| H2 | `image_to_video` | β
| h264 704Γ480@24, f=121 | 307,583 B | 7.8 s |
| H3 | `image_text_to_video` | β
| h264 704Γ480@24, f=121 | 775,213 B | 8.0 s |
| H4 | `video_to_video` | β
| h264 704Γ480@24, f=121 | 708,885 B | 8.6 s |
| H5 | `text_to_audio_video` | β
| h264 + **aac 48000 Hz 2 ch** | 792,803 B | 7.8 s |
| H6 | `image_to_audio_video` | β
| h264 + aac 48000 Hz 2 ch | 374,184 B | 7.9 s |
| H7 | `image_text_to_audio_video` | β
| h264 + aac 48000 Hz 2 ch | 995,843 B | 8.2 s |
| H8 | `text_to_audio` | β
| **pcm_s16le 48000 Hz 2 ch**, no video | 961,998 B | 1.8 s |
| H9 | `video_to_audio` | β
| pcm_s16le 48000 Hz 2 ch, no video | 961,998 B | 2.0 s |
| H10 | `audio_to_audio` | β
| pcm_s16le 48000 Hz 2 ch, **β10.4 dBFS** | 961,998 B | 2.8 s |
| H11 | `audio_to_video` | β
| h264 704Γ480@24, f=121, no audio | 193,825 B | 8.0 s |
**Eleven of eleven.** All β
measured on the real `get_pipeline` path, no bypass
and no harness accommodation; `verify_streams` returned 0 on every one.
| beyond the baseline | | |
|---|---|---|
| **H14** stage 2 | 1408 Γ 960, f=121 | 1,807,646 B Β· 35.2 s |
| **H15** `duration_head` | 704 Γ 480, **f=97** not 121 | 593,527 B Β· 5.9 s |
| **H16** high resolution | **1216 Γ 704** + aac stereo | 1,775,743 B Β· 21.0 s |
| **H17** mel round-trip | 4.000 s in β 4.010 s out, **4/4 tones** | strict gate |
| **H12 / H13a / H13b** | LoRA accepted / rejected / rejected | stage 6.5 |
> **Wall-clock excludes the model load**, which is **14.0 s once** per batch β
.
> ### β οΈ Structure is not content
>
> Every run here passed `verify_streams`. **The one defect that mattered most
> was invisible to it**: `video_to_audio` came back 50 dB below its siblings
> with correct codec, channels, rate, duration and a non-empty file. **Silence
> and a healthy signal have identical stream layout.**
>
> It was found because someone measured RMS on an artifact that had already
> been signed off, and it led to a defect affecting **all twelve modes** (gap
> 7). A green table says these are well-formed files. **It is a weaker claim
> than "correct output", and the gap between the two is where this build spent
> its afternoon.**
>
> H8 and H9 are byte-identical in size because uncompressed PCM of equal
> duration always is. That is why size proves nothing about content, and why it
> took an RMS measurement to notice.
>
> **On audio levels generally.** Measured levels sort by *condition type*, not
> by mode β
β text-only β5.2 to β6.2 dBFS, image-conditioned β8.2 to β15.2,
> video-conditioned at strength 0.8 β54.5 to β55.0. The text-only group peaks at
> exactly **1.0000 with no samples exceeding**, so the export is faithful and
> **the saturation is upstream, in the model, not in our writer**. Those files
> are consistent with each other and sit at the rail; βΈοΈ whether that is correct
> for this model has not been established.
### 2.5 What the recipe fix bought β before / after
Same seed, same geometry on both sides of every pair; **only the distilled
recipe (gap 7) differs.**
| # | mode | before | after | wall-clock |
|---|---:|---:|---:|---|
| H1 | `text_to_video` | 886,265 | 791,782 | 25.7 β **8.4 s** |
| H2 | `image_to_video` | 432,486 | 307,583 | 26.3 β **7.8 s** |
| H3 | `image_text_to_video` | 1,130,178 | 775,213 | 26.6 β **8.0 s** |
| H4 | `video_to_video` | 727,490 | 708,885 | 27.0 β **8.6 s** |
| H5 | `text_to_audio_video` | 1,091,468 | 792,803 | 25.0 β **7.8 s** |
| H6 | `image_to_audio_video` | 620,537 | 374,184 | 26.6 β **7.9 s** |
| H7 | `image_text_to_audio_video` | 784,282 | **995,843** | 26.7 β **8.2 s** |
| H8 | `text_to_audio` | 961,998 | 961,998 | 5.6 β **1.8 s** |
| H9 | `video_to_audio` | 961,998 | 961,998 | 6.0 β **2.0 s** |
| H14 | stage 2 | 2,254,799 | 1,807,646 | 84.3 β **35.2 s** |
| H15 | `duration_head` | 684,374 | 593,527 | 19.4 β **5.9 s** |
| H16 | 1216 Γ 704 | 2,331,528 | 1,775,743 | 68.7 β **21.0 s** |
**Geometry is identical on both sides of every pair** β H14 stays 1408 Γ 960,
H15 stays 97 frames, the rest 704 Γ 480 Γ 121. *Sizes changed; geometry did
not* is the shape of the result.
Three things a reader needs:
- **H7 is the only artifact that grew.** h264 size tracks content complexity,
so that is neither good nor bad on its own. Flagged, not interpreted.
- **H8 and H9 are byte-identical** because uncompressed PCM at one duration,
rate and channel count always is. **Their change is entirely in level**, not
in bytes β see below β so reading this table alone would call them unchanged
when they are two of the three that moved most.
- **~3Γ faster everywhere.** At `guidance_scale=1.0` with STG and modality
zeroed there are no classifier-free branches, so it is one forward pass per
step instead of two or three. **This is the one column here that seeding does
not confound** β wall-clock is not drawn from a distribution β and it follows
from the mechanism rather than merely accompanying it.
> ### β οΈ These pairs are not yet controlled β the audio evidence is
>
> **The seed never reached the model** until `4ebb8b9` (gap 8), so both halves
> of every pair above were unconditionally random. **The pixel differences
> cannot evidence that the recipe fix caused them** β two runs of one mode
> differed anyway. β
Seeding is fixed, and verified byte-identical **for
> `audio_to_audio`**.
>
> β **The seeded gallery that was to supersede this table is DROPPED.** It lived
> on a host that no longer exists, and nothing replaced it. **So this table is
> permanently unattributable, not temporarily so** β these figures remain as the
> record of what the defect cost, and that is now all they will ever be.
>
> **The audio levels do stand on their own**, because they are systematic and
> orders of magnitude outside run-to-run variation:
>
> | | before | after |
> |---|---:|---:|
> | H9 `video_to_audio` | β55.0 dBFS | **β17.8 dBFS** |
> | H8 `text_to_audio` | β5.2 dBFS, **peak 1.0000** (railed) | **β15.3 dBFS**, peak 0.6953 |
> | highest peak, any mode | **1.7811** | **1.0050** |
>
> A 50 dB gap and a clipped peak going away are not a seed.
>
> β **And the replacement will not arrive.** Bitwise reproducibility is verified
> for `audio_to_audio`. The **video** path has since been *evidenced* on the
> Space β a run returned bytes identical to one banked the previous day on a
> different container at the same seed β but βΈοΈ **evidenced is not proven**: no
> two-run frame-hash comparison exists. See `docs/SPACE_MODE_MATRIX.md` Β§5. The
> gallery that was to close this is dropped with the host it lived on.
>
> β *Correction (2026-08-25) on "returned bytes identical": that overstates
> what was measured. The cross-day match was **file size** (39,146 B) plus
> **every decoded statistic to three decimals** β no hash of the day-1 artifact
> was ever recorded, and that file is gone, so byte identity against it is
> permanently unverifiable. The "evidenced is not proven" hedge above was
> right, and even understated: no byte comparison existed either. The planned
> close is a same-seed re-run sha256-compared against the surviving 2026-08-22
> artifact β `SPACE_MODE_MATRIX.md` Β§5's correction carries the hash.*
> ### β
Sizes differed between the two code paths β almost certainly just the seed
>
> Three modes were run through a **bypass** (`_base_pipeline` plus an injected
> `StageRunner`) and through the **real** `get_pipeline` path, **all pre-fix**:
>
> | | bypass | real path |
> |---|---:|---:|
> | H1 | 865,447 B | 886,265 B |
> | H2 | **577,051 B** | **432,486 B** β 25% smaller |
> | H3 | 1,040,007 B | 1,130,178 B |
>
> This was recorded as an open question β *"GPU non-determinism, or a genuine
> difference between the routes?"* β and described as *"at the same seed."*
>
> **It was not at the same seed. The seed reached nothing** (gap 8). These are
> six unconditionally random runs, and two runs of one mode differed by this
> much anyway. **There was never an anomaly to explain**, and the phrase *"at
> the same seed"* was doing all the work: it made an ordinary difference look
> like a finding.
>
> βΈοΈ Cheap to settle properly now β run one mode both ways at a fixed seed.
> Nobody has, so it is *probably* the seed rather than certainly.
>
> The advice it produced stands on its own and is worth more than the question:
> **assert stream layout, never an exact size.**
### 2.3 Beyond the baseline
| Run | What it proves | Result |
|---|---|---|
| **H16** high resolution | usability, not just correctness | β
`text_to_audio_video` at **1216 Γ 704** @24, f=121 + aac 48000 Hz 2 ch β 1,775,743 B, **21.0 s** |
| **H15** `duration_head` | the model chose the length, rather than a default leaking through | β
**f=97**, not 121 β 704 Γ 480, 593,527 B, 5.9 s |
| **H14** stage 2 | upsample β 3-step schedule β diffusion decode | β
**1408 Γ 960**, f=121 β 1,807,646 B, 35.2 s |
*Post-fix figures, matching Β§2.2. Β§2.5 carries the pre-fix column.*
H15 is worth understanding: `num_frames` is *omitted* and the duration head
picks. Had it returned 121 β the default β the run would have looked identical
to a pass while proving the opposite. **97 is the whole result.**
**H14's proof is its resolution, not a comparison.** β
1408 Γ 960 against H1's
704 Γ 480 is **exactly 2Γ in both dimensions** β a size only the spatial
upsampler can produce, and **the one claim here that no vintage or seed can
affect**, since geometry is identical across every before/after pair. That is stronger evidence than the
same-seed diff the matrix planned, because a same-seed comparison shows *a*
difference while this shows *the specific* difference stage 2 exists to make.
### 2.4 The LoRA validator, both directions
| Run | Adapter | Result |
|---|---|---|
| **H12** | `joyfox/LTX-2.3-Transition-LORA` | β
**ACCEPTED**, **576 modules** across transformer + connectors |
| **H13a** | `charlied98/β¦` `lora_weights_step_03000` | β
**REJECTED at stage 6.5**, `LoRADimensionError` |
| **H13b** | same repo, `comfy_lora_weights_step_03000` | β
**REJECTED at stage 6.5**, same mechanism |
H12 carries **mixed dimensions** β 4096 at attention, 2048 at `audio_ff`, 16384
at `ff.net.2` β
β and none of that is a mismatch. An earlier "expect hidden dim
4096" rule would have **rejected a perfectly good adapter**; the false-positive
guard is what stops that. H13a and H13b differ only in key prefix
(`transformer.` vs `diffusion_model.`) and reach the same verdict by the same
mechanism, which is what makes the rejection trustworthy rather than lucky.
---
## 3. SPACE-PROVEN vs HOST-PROVEN β read this before believing Β§2
> ### β
The Space runs and generates. βΈοΈ That is **one mode, once.**
>
> `text_to_video` at 256Γ256Γ9 produced a real, content-verified video on the
> live private Space β
. **Ten of the eleven modes have never executed there.**
>
> β ***Corrected (2026-08-25, evening): it was not a real video. It was colour
> static** β the origin claim of the entire Space evidence base, and it was
> wrong from the first run. "Content-verified" meant non-empty and non-frozen.
> What survives: the Space runs, and one mode executed there. Banner at the
> top.*
>
> A green table on a dedicated host and a working deployment are different
> claims. **Both have now been made β about different things.** That is exactly
> why this section still exists: the two evidence bases have diverged rather
> than merged, and Β§2's eleven-of-eleven remains a **hinode** claim.
> β
**SUPERSEDED (run 2026-08-22, annotated 2026-08-25).** The mode matrix ran
> on the live Space and **all eleven modes generate there, content-verified**,
> at sha `287ba9c` β [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md). The two
> evidence bases have **merged at exactly one configuration**: 256Γ256Γ9,
> stage 2 off, no LoRA, no `auto_duration`.
>
> β ***THE MERGE NEVER HAPPENED (corrected 2026-08-25, evening).** hinode ran
> **bf16, unquantised** β Β§2.1's `base.dtype` row records exactly that β and the
> Space ran blanket int8. The two evidence bases were separated by a dtype
> boundary no run ever crossed, and that difference was the whole defect. **This
> section's original discipline was right and this annotation was the error:**
> it merged on "the same mode name at the same geometry" when the thing that
> differed was the quantisation of the conditioning path. Keep Β§3. Banner at the
> top.* What is still hinode-only: stage 2
> (1408Γ960), the duration head, LoRA in both directions, the mel round-trip at
> full length, and every shippable resolution. This section's discipline is
> unchanged β it now keeps *configurations* apart instead of *platforms*.
>
> *That hinode-only list shrank again on 2026-08-25: stage 2 reached 1408Γ960
> on the Space (from β€25-frame input; its decoder ceiling is between 25 and 49
> frames), LoRA ran in both directions, and 1216Γ704Γ121 passed via the tiled
> decode. Remaining hinode-only: the duration head, and full-length audio
> work. `SPACE_SESSION_2026-08-25.md`.*
**Detailed record: [`SPACE_GENERATION_FINDING.md`](SPACE_GENERATION_FINDING.md)**
β the first deploy, the killed workers, the falsified ceiling, and the run that
generated. Not restated here on purpose.
hinode is an **unpartitioned** card with **124 GB RAM** and an ordinary
filesystem. ZeroGPU is none of those things. **The first deploy answered several
of these questions and changed the answer to one of them:**
- β
**Tensor packing** β exercised. Packing ran and the post-pack prune fired,
freeing 65.25 GB.
- β
**The 150 GB ephemeral-disk ceiling** β never the binding constraint. The
disk line read **82.2 GB of 150 [OK]** while the workload died of something
else. Arithmetic in [`ZEROGPU_DISK.md`](ZEROGPU_DISK.md).
- β **RAM β the ~68 GB figure is FALSIFIED.** See the finding below; this is the
single most important correction in this document.
- β
**The private-Space fetch path** β works. 37/37 files at the pinned revision,
downloaded at runtime because `preload_from_hub` 401s on a private repo.
- β
**The fork boundary** β crossed. A worker forked, ran, and released cleanly
(`fail=false` β 200).
- β
**The `size` kwarg's *plumbing*** β the name survived
`gpu_decorator_kwargs`'s signature filter and raised **no import-time
`TypeError`**. That is the risk PLAN Β§5.8 actually named β the platform pins
its own `spaces` and lags PyPI β so this is a real β
**on its own terms**.
- β **An explicit `size` value β never passed, and THAT WAS THE BUG.**
**β RESOLVED (2026-08-27), and the inference above was wrong.** `app.py` sent
`size=None` whenever `LTX25_ZEROGPU_SIZE` was `auto`, which was its default and
what the Space ran. The reasoning recorded here β *"ZeroGPU chose `xlarge` by
inference, packed 51.1 GB exceeds the 42.00 GiB threshold"* β was the fourth
document in this repo to assert it and none of the four had followed the
promoted value past the function that sets it. It never reaches the scheduler:
`default_gpu_size()` is consumed only by `get_duration_seconds`
(`client.py:109`) and feeds only the duration multiplier, while `gpuSize` is
sent to the platform only when the *decorator's* size is not None
(`api.py:112-113`). β
**`auto` therefore ran the Space on `large` β
47.375 GiB β for its entire life, instead of `xlarge`'s 94.971 GiB.** Measured
from inside the worker, both before and after. `LTX25_ZEROGPU_SIZE` is now
`xlarge`. Full trace: [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md) Β§1.
β οΈ Note the exact shape of the error, because it is the interesting part: the
*reservation* really was computed as `xlarge` (no 1.5Γ `duration_factor`). The
hardware was `large`. We were priced as one and allocated the other.
- βΈοΈ **Quota metering** β still unreadable. All three candidate endpoints 404, so
billed seconds can only be **bounded**, never measured.
- βΈοΈ **Ten of eleven modes** β no conditioning, no stage 2, no LoRA, and **no
audio path** has run on ZeroGPU. The audio modes exercise the mel front-end,
the `av` loader and the 16 kHz/48 kHz asymmetry; none of that has executed
there once.
β *Annotation (2026-08-25): mostly falsified by the 2026-08-22 mode matrix.
Image, audio and video conditioning **have** run on ZeroGPU; both audio-input
modes passed; the mel front-end and the 16 kHz/48 kHz asymmetry were
exercised there and passed (`SPACE_GENERATION_FINDING.md` Β§11 item 10,
`SPACE_MODE_MATRIX.md`). Still true, and only these: **stage 2, LoRA and
`auto_duration` have never run on ZeroGPU.***
β *β And that annotation is corrected in turn (2026-08-25, evening): "passed"
meant a well-formed, non-empty, non-frozen file from a transformer whose
conditioning path had been quantised away. Those paths **executed** on ZeroGPU
β the mel front-end, the `av` loader, the 16 kHz/48 kHz asymmetry all really
ran, and that half stands. The βΈοΈ this replaced was cautioning about something
real, and it should not have been retired on the strength of "passed". Banner
at the top.*
---
### The three findings from the first deploy
> #### 1. β The **~68 GB** RAM ceiling is **FALSIFIED**
>
> It was a community cgroup observation β **inherited, never ours**, and
> load-bearing in planning since before any deploy. Startup reached
> **87.13 GiB = 93.56 GB** peak RSS and **was not killed**.
>
> β
**Replace it with our own measured lower bound: the ceiling is above
> 93.56 GB.** Every headroom figure computed against ~68 GB was wrong by at
> least 25.56 GB.
> #### 2. β οΈ Quantising **more** raised peak RSS by **31.24 GiB**
>
> Adding the text encoder to `components_to_quantize` moved RSS **up**, from
> 55.89 to 87.13 GiB, in a change made to bring it down. `safetensors` mmaps
> lazily; **quantisation materialises those weights as anonymous memory**, so
> quantising a component destroys the laziness that was keeping it cheap.
>
> This is the **same single mechanism** `ZEROGPU_DISK.md` Β§4.2 already used to
> settle two other questions, now deciding a third β in the opposite direction
> from what we predicted.
>
> β ***Annotation (2026-08-25, evening): this is the change that broke
> generation, and this document records it as a RAM result and nothing else.**
> Widening `components_to_quantize` is the blanket int8 that quantised the
> conditioning path; from here until `eb1aa9f` the Space produced colour static
> in every mode. The RSS measurement above is correct and unaffected. What is
> missing β and was missing everywhere, `README.md`'s int8 tier note included β
> is any sentence connecting quantisation to output fidelity. Nobody wrote one,
> so nobody looked.*
> #### 3. βΈοΈ The `SIGKILL` is **still unexplained**
>
> cgroup-OOM is **dead** as an explanation: with the ceiling above 93.56 GB
> there was never the narrow margin the hypothesis required. β
**Route C β
> instrument the worker β is now the LEADING path, not the fallback.** It is the
> only route that returns evidence rather than another inference, and every
> inference made about this kill has been wrong at least once.
> #### β And the fix is **UNATTRIBUTED**
>
> Two things changed between the failing and succeeding runs: the quantisation
> widening **and a full container rebuild**. n=1 against n=1 with two variables.
> **"The Space now serves" is established; "the widening is why" is not** β and
> the hypothesis that motivated the widening is itself dead. Do not let any
> document claim the widening fixed generation.
>
> β
***VINDICATED, in the strongest available direction (2026-08-25, evening).**
> That last sentence was written as a caution against over-crediting the
> widening. The widening **broke** generation: blanket int8 quantised the
> conditioning path, and every Space artifact from here to `eb1aa9f` is colour
> static. "The Space now serves" was established and was exactly as narrow as
> this paragraph insisted.*
**Cost so far: β€ 243.58 GPU-seconds** of 2400/day β οΈ *a bound, not a
measurement* β quota endpoint 404s.
> **Superseded (annotated 2026-08-25).** That figure covered the
> first-generation session only. The mode-matrix day β UTC quota day
> **2026-08-21** β closed at **β€ ~413 GPU-s** of 2400 (`SPACE_MODE_MATRIX.md`
> Β§7, matrix plus re-run; still a bound, never a measurement). Quota days are
> per-UTC-date, so as of 2026-08-25 the day is fresh and neither number
> describes current consumption.
**One hinode number with a direct ZeroGPU consequence.** Stage 2 peaked at
**83.897 GiB** β
, against 70.497 GiB for a single-stage run. So **stage 2 cannot
run on a `large` 48 GB MIG slice at all**, and on `xlarge` (96 GB) it leaves
roughly **11 GiB** before any packing overhead. That is measured on an
unpartitioned card; the packing overhead is βΈοΈ unknown and comes off that margin.
One live risk that Part A *has* surfaced: the stage-2 diffusion decoder resolves
a **NATTEN kernel from the Hub at generation time** π, which is the only
network dependency that fires *after* the user presses Generate. Whether it
caches per image or per cold worker is βΈοΈ unknown and decides its severity.
---
## 4. Known gaps, stated plainly
0. **~~The `size` kwarg has never carried a value.~~ β CLOSED, AND IT WAS THE
MOST EXPENSIVE ITEM IN THIS LIST (2026-08-27).** The gap was written as *"if
anyone sets `LTX25_ZEROGPU_SIZE` to force a size, they are exercising an
untested path"* β with the unstated assumption that leaving it at `auto` was
the safe, tested default. It was the opposite. `auto` sends **no** size, and
ZeroGPU's own promotion never reaches the scheduler, so `auto` **was** the
`large` branch: 47.375 GiB. β
The note above about `large` being a slice
"that Β§3's own stage-2 measurement says cannot hold stage 2 at all" was
therefore describing the partition the Space was already running on β and it
predicted R4b's failure precisely, a day before anyone connected the two.
`LTX25_ZEROGPU_SIZE=xlarge` is now set and β
verified from inside the worker
at 94.971 GiB. Stage 2 at 704Γ480Γ49 passes on it: 53.08 s, 1408Γ960, peak
55.906 GiB reserved. See [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md).
1. **H10 / H11 β the audio-input modes β have not passed.** βΈοΈ The path they
need was connected to nothing until very recently: the UI collected a file,
the encoder existed and was exported, and **nothing called it**. It is built
and tested now, and it has not yet produced an artifact on the card.
**The acceptance test is H10b**, not H10: two `noise_scale` values at the
same pinned seed. *Identical audio means the input is still not reaching the
model*, whatever the stream check says β because `noise_scale=1.0` discards
the input by design, so a bug that drops it entirely is indistinguishable
from correct operation.
β *Annotation (2026-08-25): the first sentence was overtaken twice and this
paragraph was never updated. Both modes then passed on the card β Β§2.2's own
rows record H10 β
(961,998 B at β10.4 dBFS) and H11 β
(193,825 B) β and
both passed again on the live Space (2026-08-22, `SPACE_MODE_MATRIX.md`).
What stands is the acceptance-test framing: **H10b has still never run**, so
"the user's audio reaches the model" remains βΈοΈ unestablished; everything
that has passed is consistent with the input being silently discarded.*
2. **Stage 2 works; its NATTEN fetch is characterised.** β
**Resolved once per
process, not per generation** β two stage-2 generations in one process, one
kernel fetch. So a Space needs outbound network at a worker's **first**
stage-2 use rather than on every press of Generate. Still a real dependency,
because **each cold worker pays it once**. Retained below for the mechanism.
β
H14 ran and
produced 1408 Γ 960. But the diffusion decoder resolves its NATTEN kernel
through a **version β ref lookup that goes over the network even when the
kernel files are cached** π, and **whether that fires once per process or
once per generation is βΈοΈ unknown**. It cannot be grepped after the fact β
the run log contains **zero** natten mentions β so answering it needs
deliberate instrumentation. On a private Space this is the 401 class again,
except firing *after* the user presses Generate.
3. **`connectors: 0` on every accepted adapter.** β
observed. Stage 7 asserts
*total* modules attached > 0, which passes on the transformer alone β so an
adapter that *ought* to touch `connectors` and silently does not would still
be reported as attached. Probably correct for the three adapters tested
(2.3/2B-era), so **not currently a defect** β but the assertion as written
cannot distinguish "legitimately unused" from "silently missed". A
per-component expectation would close it. See PLAN Β§4.4.
4. **`temporal_latent_upsampler` has no known role.** π It is the same class as
the spatial upsampler, differing only by axis, and upstream's component table
and both documented recipes mention only the spatial one. We download it
(0.2619 GB) and do **not** wire it in. Not a defect; an open question.
5. ### β
`video_to_audio` near-silence β **fixed, and the first diagnosis was wrong**
H9 came back at **β55.0 dBFS** against `text_to_audio`'s β5.2 β
. **The cause
was the distilled recipe reaching only the stage-2 path (item 7).** With that
corrected, H9 runs at **β17.8 dBFS** β 2.5 dB from its siblings β at the
**unchanged** condition strength of 0.8.
| | before | after |
|---|---:|---:|
| H9 `video_to_audio` | β55.0 dBFS, peak 0.0146 | **β17.8 dBFS**, peak 0.5078 |
| H8 `text_to_audio` | β5.2 dBFS, **peak 1.0000** (railed) | **β15.3 dBFS**, peak 0.6953 |
| gap | **~50 dB** | **2.5 dB** |
**One mode 50 dB down and another driven into the rail were the same
misconfiguration seen from opposite ends.**
β οΈ **The first diagnosis blamed condition strength and was wrong.** A
single-lever sweep at fixed seed gave 0.8 β β55.0 and 0.5 β β5.1, and that
was read as the cause. It was a real interaction **between condition strength
and the broken guidance** β a genuine effect inside a pipeline that no longer
exists. A per-mode strength cap was written, committed and reverted.
> **A single-lever same-seed sweep is strong evidence that something moves,
> and no evidence about why.**
The source citation said throughout that a video condition *cannot* attenuate
audio in the denoise loop β the conditioning mask is applied to
`noise_pred_video` and never touches `noise_pred_audio`. It was correct, and
a clean correlation was believed over it.
6. **The audio input rate is 16000, the vocoder output is 48000.** π They are
not symmetric. Anything that assumes one rate for both directions will
pitch-shift audio without erroring.
7. ### β
The distilled recipe reached only the stage-2 path β **fixed**
Eleven of twelve runs sent **no sigmas at all**, so upstream built its own
schedule and five guidance parameters took their **SFT** defaults β
`audio_guidance_scale` at **7.0** against an intended 1.0.
```
sent: [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125]
intended: [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]
```
β
All twelve artifacts differed before and after, with magnitudes ordering
by conditioning strength (text-only 37β59 mean |diff|, image-conditioned
15β16, video-conditioned **3.71**).
βΈοΈ **Neither of those establishes the fix caused them, and the ordering is
not the corroboration it looks like.** Both halves were unseeded (gap 8), so
two runs of one mode differed anyway β and a **more strongly conditioned mode
is more constrained, so its run-to-run variance is smaller too.** The
ordering is exactly what random variation would produce. It cannot separate
the two explanations, and an earlier draft of this section claimed it could.
**What does stand, unconfounded:**
- **the timings.** 25.7 β 8.4 s is not run-to-run variation, and it follows
directly from the mechanism: unguided means one forward pass per step
instead of two or three. **The defect cost 3Γ the compute**, and that is
measured rather than inferred.
- **the audio levels and peaks** β β55.0 β β17.8 dBFS, highest peak
1.7811 β 1.0050. Systematic and orders of magnitude outside variation.
- π **that the schedule and guidance were wrong at all**, which is a source
fact and needs no run.
**It cost 3Γ the compute as well as the quality.** At `guidance_scale=1.0`
with STG and modality zeroed there are no classifier-free branches, so it is
one forward pass per step rather than two or three:
| | before | after |
|---|---:|---:|
| H1 `text_to_video` | 25.7 s | **8.4 s** |
| H8 `text_to_audio` | 5.6 s | **1.8 s** |
| H14 stage 2 | 84.3 s | **35.2 s** |
| H16 **1216Γ704 + stereo** | 68.7 s | **21.0 s** |
Clipping went with it β highest peak **1.7811 β 1.0050** β
β so no limiter is
needed and `outputs._write_audio`'s *"what comes back is what the vocoder
produced"* principle stands. **Our path never added gain; the material was
arriving at Β±1.0.** Stage 2's brightness falloff was this too, not stage 2:
mean `99.48 β 34.44` collapsing before, `82.52 β 110.47` stable after.
8. ### β
The seed reached only the sidecar β **fixed, and verified byte-identical**
`_resolve_seed` ran and the number went into the encoder metadata. **There
was no generator anywhere in the package**, so every run was unconditionally
random and every *"same seed"* comparison was two independent runs.
β
**Fixed** in `4ebb8b9`: a generator seeded from the request, built on the
pipeline's own device, threaded to the pipeline **and** to
`audio_vae.encode`'s `.sample()`.
β
**Verified on the card, for `audio_to_audio`.** Two runs at identical seed
and configuration:
```
run-a1d35aaaab06/output.wav md5 2988f01159a5322afb28abb6e9a73d26
run-b9dde07ed8bb/output.wav md5 2988f01159a5322afb28abb6e9a73d26
```
β
And the fix was complete when it landed, not repaired afterwards:
`git log -S 'generator' 4804da1..e4e8df1 -- ltx25_studio/` returns `4ebb8b9`
alone.
> **On this path, `seed` determines the output byte-for-byte, on GPU, with no
> deterministic-algorithm flags set.** Stronger than expected β the weaker
> *"seed-consistent but not bit-reproducible"* was drafted and never needed.
βΈοΈ **Scope, and it matters for Β§2.5:** that measurement is **one mode**. The
**video path is not shown to be bitwise reproducible** β it is *evidenced* on
the Space (byte-identical output across two days and two containers at one
seed, plus `text_to_audio_video` matching `text_to_video`'s video statistics
to three decimals) and no two-run frame-hash comparison has ever been made.
`docs/SPACE_MODE_MATRIX.md` Β§5 states that limit explicitly.
β *Correction (2026-08-25): "byte-identical output across two days" is too
strong β the cross-day evidence is **size-identical plus
statistics-identical**; no day-1 hash exists and the day-1 file is gone, so
byte identity was never checked and now cannot be. The conclusion of this
passage β not shown to be bitwise reproducible β is unchanged and was the
correct one. `SPACE_MODE_MATRIX.md` Β§5's correction records the planned
sha256-compared re-run against the surviving 2026-08-22 artifact.*
β **The twelve-mode pixel-diff gallery that depended on it is DROPPED** β the
host it ran on is gone. **So the consequence is permanent rather than
pending**: the Β§2.5 comparison table was measured *before* seeding worked, its
pixel diffs cannot attribute the change to the recipe fix, and nothing is
coming to replace it. Everything structural was never affected β H14's exact
2Γ resolution, H15's 97 frames, H17's frequency recovery, and every audio
level.
> The lever that should have caught it was ours: `decode_kwargs` took a
> `generator` argument, **with a docstring explaining why the diffusion
> decoder must be seeded**, and read a key nothing ever set. See PLAN Β§9
> rule 12.
9. ### βΈοΈ `noise_scale` and `sigmas[0]` disagree β open, deliberately unfixed
π Upstream infers one from the other β *"first (largest) sigma value if
using custom sigmas"* β and our `stage_2_kwargs` honours it. The
audio-input modes do not: they send `noise_scale` while `sigmas[0]` stays
1.0.
**It was diagnosed as the cause of a silent `audio_to_audio` and it was
not** (Β§4.1). A fix was written, committed and **withdrawn**: truncating the
schedule at `noise_scale` takes an eight-step run to **two** steps at 0.5
and **one** at 0.1 β values that measured healthy on eight.
**Agreeing with a contract is not evidence that a change improves
anything.** A test pins the current behaviour and explains why, so the next
attempt arrives with a measurement rather than with agreement.
---
## 4.1 β οΈ Known limitation β unusual audio input produces a silent file
**Not a defect we can properly fix, and the most likely way a user meets a
disappointing result.**
`audio_vae.encode` is **never called by the pipeline**. Our
`encode_waveform_to_latents` is the only producer of those latents, and
**nothing anywhere validates their distribution** β not upstream, not us.
So audio that is far from what the model was trained on encodes to latents the
model cannot use, and the result is **a structurally perfect, inaudible file
with no error and nothing to act on.** β
A real instance, and it cost a day of
misdiagnosis: a two-tone synthetic fixture at **β14.8 dBFS in** produced
**β87.9 dBFS out**, with correct codec, channels, rate and duration, and
`verify_streams` exit 0. The same mode fed a real recording produces **β10.4
dBFS**.
**Likely triggers:** pure tones, sine sweeps, test signals, near-silence,
heavily synthesised material. **Ordinary recorded audio is fine** β every
realistic input tried has worked.
β **Do not "fix" this with an out-of-distribution detector.** Reliable OOD
detection at that boundary is not something this project can do, and a
detector that is wrong in either direction is worse than the limitation.
βΈοΈ **A narrower thing is worth doing and is deliberately not done yet.** We
measure the input level and can measure the output level, so **a file that
comes back essentially silent could say so** in `GenerationResult.warnings`.
The decision that makes it honest is that it must be a **measurement, not a
diagnosis**: *"this output is β87.9 dBFS, effectively silent"* is a fact about
the artifact, checkable and cause-free. *"Your audio was out of distribution"*
is a claim we cannot support.
It is not built because it needs a threshold, a threshold needs evidence, and
the PR opens with **one** named defect. **Adding an unvalidated heuristic to
the export path hours beforehand is the same move as shipping a schedule change
on a contract reading** β which was reverted for that reason. It belongs in the
next pass, with a measured floor.
---
## 5. How to run it
### On hinode
```bash
./scripts/sync_to_hinode.sh # ships git archive HEAD, not the working tree
```
`rsync` is deliberately not used: it is not installed on the host and there is
no sudo to add it. Needs an HF token with access to the gated model repo. Then
`scripts/hinode_run.sh`, or `scripts/host_matrix.py` for the full matrix.
**Do not set `HF_HUB_OFFLINE=1`.** Construction loads from the local snapshot
directory, so nothing is fetched at load time β and stage 2 needs outbound
network for the NATTEN kernel. Forcing offline breaks the second without helping
the first.
### Locally, with no GPU
The whole test suite runs with **no GPU, no weights and no network**. That is a
deliberate property, not a coincidence, and it is worth preserving: it is what
makes it possible to catch a defect before it costs a run.
```bash
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/python -m pytest -q
```
`torch`, `diffusers` and `torchaudio` are **not** needed for the suite. `av` and
`numpy` are, because they are how real containers get written and read.
### On a Space
β
**This works.** The repo *is* the Space β `README.md`'s YAML front matter is
the Space config, and the `space` remote points at it:
```bash
git remote -v | grep space
# space https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio
```
Deploying is a push to that remote's `main`. β **This team does not push** β not
to `origin`, not to `space`. The orchestrator does it; that is a deliberate
boundary, not a missing capability.
**What the deploy depends on, all of it already load-bearing:**
| | |
|---|---|
| `app_file: app.py`, `sdk: gradio 6.24.0`, `python_version: "3.12"` | π front matter |
| `suggested_hardware: zero-a10g` | π **a legacy id string β not an A10G.** Changing it moves the Space onto paid dedicated hardware |
| `startup_duration_timeout: 1h` | π **required.** The default is 30 min and this Space downloads 72.19 GB at build time. When it blows, the failure looks like something else entirely |
| an HF token secret with access to the gated model repo | π weights fetch at **runtime**, because `preload_from_hub` 401s on a private Space |
`tests/test_deploy_drift.py` asserts these rather than trusting them β run the
suite before any deploy and the drift shows up locally instead of as a
40-minute build that dies at the end.
**Reading the startup log is the cheapest diagnostic available**, and it costs
zero GPU quota because packing and loading happen at module scope, before the
first `@spaces.GPU` call. The load line names its own configuration:
```
quantised transformer, text_encoder as int8 | peak RSS 0.91 -> 87.13 GiB
```
β οΈ **That peak is the number to watch.** It is a *peak across the load*, not a
steady state β we have never measured steady state, and the RAM questions are
about steady state. See [`SPACE_GENERATION_FINDING.md`](SPACE_GENERATION_FINDING.md) Β§7.
---
## 6. The lessons worth carrying
Not a diary. These are the ones that would change how the next person works.
### A fake derived from the mapping can never catch a wrong mapping
The audio VAE's six parameters were read from `audio_vae.config` **by names that
do not exist there** π β four live under different keys, two are instance
attributes absent from `config.json` entirely. Every runtime path through that
module raised on real weights, and every offline test passed, because the fake
used *our* names. It agreed with the code and with nothing else.
The instinctive repair β build the fake from the same mapping the code uses β is
a trap for exactly the same reason. The fix is a **recorded fact about reality**:
the real config's key set, checked into the tests with its provenance. The fake
checks semantics; the recorded keys check the world.
### The fixture that makes a test possible is itself untested by that test
Three instances in one day, all in tests that were otherwise excellent:
1. `test_encoder_contract.py` drove the **real** encoder through **real**
containers to a real file β and built its audio as `np.float32`. The model
produces `bfloat16`. **Six of eleven modes were broken at the last step and
1926 tests passed** β
*both counts measured β the modes on hinode, the tests
in this repo at the commit before the fix*.
2. The replacement fake's `.float()` returned an `ndarray` where torch returns a
*tensor*. Written the convenient way, it would have passed a broken fix.
3. The audio-VAE fake used our own key names β the case above.
The mitigation that worked: **check the fake against the real thing.** In (2)
that check failed within a minute of existing.
### A search that cannot see the claim reports that the claim is gone
Three defects in this build share one shape: **the instrument returned a
confident negative it was structurally incapable of turning positive.**
- **A wrapped line.** Β§7 said *"The Space has evicted twice and has never run"*
as one line. Β§1 said the same thing **split across a line break**, so a grep
for the phrase found the occurrences that fit on one line and pronounced the
audit clean.
- **A shell modifier.** `"$c:ltx25_studio/..."` is parsed by zsh as the `:l`
lowercase modifier, so `git show` got a mangled path and **every** commit
answered *"constant absent"* β including the one that demonstrably contains it.
- **A stale marker.** A `β οΈ (inherited)` pointing at a section that had been
rewritten to say **FALSIFIED** still scanned as *merely unverified*.
**None of the three raised an error.** Each returned a clean, plausible, wrong
answer, and two of them nearly closed an audit.
The defence is not a better search string. It is **running the instrument
against a case whose answer you already know**: if the check cannot go positive
on a known positive, its negatives carry no information. The zsh case was caught
only because *"absent at HEAD too"* was impossible on its face.
### A claim moving known β unknown is nearly always a regression
Documents accrete facts and rarely lose them. When an edit is about to record
something as uncertain, search first for it being *known* β the likeliest
explanation is that someone established it and the current author has not read
it. This has since caught its own inverse: a correction issued against this
document's history that the history did not actually contain.
### Checks over comments
Twice, a constraint written down did not survive the next function written by
its own author β a `dtype` requirement violated two commits after the β οΈ comment
demanding it, and a 44.33 GB refetch reintroduced *while reading the docstring
that predicts it*. A comment is a message to a future stranger; it has no
purchase on the author five minutes later. **Make the wrong thing unsayable
instead:** derive the value once and share it, so the two callers cannot drift.
### The instruments encode the failure you imagined, not the failures available
`host-runner`'s generalisation, in their words:
> *all three instruments encoded the failure I imagined rather than the failures
> available*
β a `pkill -f` pattern that matched the killer, a wait loop with no failure
exit, and a fixture guard that handled a fixture being *absent* but not one that
*raises*. **The three had different authors, including the lead.** That is the
point of recording it: it is not one person's blind spot, it is what happens
whenever the person who imagines the failure is the person who writes the check
for it.
### We can prove a function ran. We cannot prove it ran with the right arguments.
**The gap this build ends on**, and the one worth closing next.
`DISTILLED_SIGMA_VALUES` was defined, documented as mandatory, quoted in its
module's docstring, and **referenced** β so every check we own passed. It was
referenced from `stage_1_kwargs()`, which `StageRunner` applies **only when
stage 2 is enabled**. Eleven of twelve host runs were generated with upstream's
linear fallback and its **SFT** guidance defaults instead:
```
sent: [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125]
intended: [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]
```
A reference check passes. A reachability check at module level passes. Even the
function-level sweep passes, unless someone thought to name that particular
function. **What nothing checked was the arguments the pipeline was actually
called with.**
The pattern that does work, used three times here, is to **record the vendor's
specification independently and assert against that** β `MODEL_CARD_DISTILLED`,
`REAL_AUDIO_VAE_CONFIG_KEYS`, `REAL_VOCODER_CONFIG_KEYS`. Each is a fact copied
from the source of truth rather than derived from our own constants, so changing
ours cannot quietly redefine what correct means. Without that independence the
test agrees with the code and with nothing else.
> **The escalation ladder, in the order this build climbed it:** is the module
> imported β is it *reached by the path that does the work* β did this specific
> *function* run β **were the arguments the ones the vendor specifies.** Only
> the last rung is still hand-built, one call site at a time.
### A module can be reached and still contain a dead function
The reachability sweep that exists to catch importer-count-zero missed one, for
two independent reasons: the tracer recorded which **file** ran and discarded
which **function** β the name was in the same frame object β and the probe drove
**one mode**, so a path reachable from no other mode was invisible by
construction. **Either fix alone would still have missed it.** "Sharpen the
instrument" was the obvious response and would not have worked.
### The one time source-reading and measurement confirmed each other
Every other pairing today went the other way: one corrected the other, and the
lesson was about which to trust when they disagreed. Once, they agreed.
A reproducibility control failed β two runs at one seed, 8.7% apart β and the
obvious reading was an unseeded draw somewhere. Instead of hunting, **every
noise source in the pinned pipeline was enumerated**: `_create_noised_state`,
the conditioning mask, `prepare_*_latents`, the decode timestep, the condition
VAE encode. All threaded the generator; the encode used `sample_mode="argmax"`
and was deterministic regardless. **Source said it should already be
reproducible.**
Rather than write either sentence β *"the seed is attached to nothing"* or
*"we are seed-consistent but not bit-reproducible"* β a scoping run was
requested. Before it ran, the control was re-checked: **it had been measured
against a tree that predated the seeding fix.** At the tip, two runs are
byte-identical.
> **The true claim was stronger than either draft**: on `audio_to_audio`, `seed`
> determines the output byte-for-byte, on GPU, with no deterministic-algorithm
> flags. Both drafted sentences were weaker *and* wrong, and both would have
> shipped if the reluctance to write an unmeasured claim had not held for one
> more exchange.
**And then the corrected sentence was written without its scope, twice more in
this document** β as a property of the build rather than of one mode. βΈοΈ The
video path is still unverified. Recording that here because the lesson is not
*"be careful"*: it is that **a claim narrowed in one place does not narrow
itself in the others**, and a document is exactly where that goes wrong. Β§4
gap 8 carries the scope; Β§2.5 and this paragraph each had to be found and
fixed separately.
**Refusing to write a sentence you cannot yet support is the same discipline as
refusing to change a threshold to make a test green** β and it is the cheaper
of the two, because nobody has to notice you did it.
### Staffing β two makers and a lead who verifies
**Not five, and not one.** This build ran briefly with five teammates and settled
at two: one implementer owning the code, one runner owning the hardware. The
three stood down were not underused β they were **a coordination cost with no
matching throughput**, and the collision they produced (two agents assigned
`outputs.py`; a cross-boundary change landing under a commit message about other
files) cost more than their output was worth. Two makers with **disjoint, stated
file ownership** is the shape that worked, and the line held all day under
pressure: the runner found four product defects and edited none of them.
**The scarce resource was execution on real hardware, not review capacity.** At
the point the suite reported **1926 passing** β
, six of eleven modes were broken,
the audio-input path was connected to nothing, stage 2 was unreachable, and every
runtime path through `audio.py` raised on real weights. None of that was found by
reviewing harder β it was found by an afternoon on a real GPU. **Another reviewer
would have added agreement, not information.** Budget for one more agent is
better spent on a second executor with *different* hardware than on a second
reviewer.
**A lead who cannot edit is forced to verify by observation rather than by
reading their own diff**, and that constraint turned out to be a feature. Every
claim the lead checked was checked by running something β `git merge-base
--is-ancestor` rather than trusting a commit message, `av` on the container
rather than trusting a verifier's exit code, `pgrep` on the box rather than
trusting a status report. That caught the box silently trailing the tip **three
times**, a stale bypass premise, and a wait loop that had frozen a teammate. A
lead who could edit would have been tempted to fix those, and would have stopped
looking.
**The default-REJECT verifier role was filled by reality rather than by an
agent.** The plan called for an independent verifier holding a default-REJECT
stance; in practice the hardware held it. It rejected the `from_pipe` cast, the
bf16 encode, the missing audio parameters and the unreachable stage 2 without
being asked, and without any possibility of being talked round. **Where a real
gate exists, prefer it to a reviewer agent β an agent can be persuaded and a
`RuntimeError` cannot.**
**What it cost.** Message-crossing: three times a blocking item was buried inside
a long message and lost a cycle behind work already in flight. The fix is
mechanical and worth adopting from the start β **one blocker, one message,
nothing else in it.** Long messages are for context; short ones are for
priority.
---
## 7. If you do one thing
**Re-run the comparisons seeded.** Everything works; the open item is that some
of the *evidence* was gathered before the seed reached the model.
### The runs that need repeating, and what each would then establish
| run | what it would establish | why it does not yet |
|---|---|---|
| **Β§2.5 before/after pairs** | that the recipe fix caused the pixel differences | both halves were unconditionally random |
| **H10b** β `noise_scale` 0.1 vs 0.9 | that the user's audio reaches the model | same, plus the audio encode drew unseeded |
| **H12b** β LoRA effect | that attaching the adapter changed the output | same |
| **H4b** β `strength` 0.3 vs 0.9 | that the conditioning clip reaches the model | same |
Add a **same-value control** to each β two runs at *identical* settings. Under
a working seed its spread should be **zero**, and that is what turns "the halves
differ" into evidence rather than a coin toss. An earlier control measured 1.8%
and was read as a bound on the variance; **an unseeded control cannot bound
anything**, which is the mistake worth not repeating.
> β **The first row of that table is dead (annotated 2026-08-25), and this
> file already said so elsewhere.** The Β§2.5 before/after pairs cannot be
> re-run: the seeded gallery is dropped with its host, and Β§2.5/Β§4 gap 8 record
> that as *permanent* β the table is unattributable forever, not pending. The
> live content of this instruction is the other three rows: **H10b, H12b and
> H4b are still runnable** β on the Space (none needs stage 2; each is one
> same-seed pair at 256Γ256Γ9-class cost) or on a future host β and each still
> establishes exactly what the table says.
>
> *Two of the three were delivered on the Space later on 2026-08-25: H12b's
> claim (the adapter changed the output β `20a3a84bβ¦` β `b78803ccβ¦` at the
> same seed) and H4b's question (the conditioning clip reaches the model β
> answered by clip-swap: outputs 17.277 vs 0.864, mirroring their inputs,
> against a byte-identical control). **H10b is the last discriminator
> standing.** `SPACE_SESSION_2026-08-25.md` Β§6βΒ§7.*
>
> β ***BOTH DELIVERIES ARE WITHDRAWN (2026-08-25, evening).** Every run cited
> there used the broken int8 build. H12b's two hashes differ because one garbage
> render differs from another; H4b's 17.277-vs-0.864 compared the motion energy
> of two colour statics. The mechanical readings stand β the adapter injection
> path ran, the conditioning path carries a scalar, the noise floor at a fixed
> seed is exactly zero β and **all three discriminators are open again: H10b,
> H12b and H4b.** They are also worth spending on now, because there is finally
> a build whose output is a picture (`e3b5333`).*
### What is already earned, and does not need repeating
**Eleven of eleven modes produce artifacts with the expected stream layout on
the real path.** β
H14's exact 2Γ resolution, H15's 97 frames, H17's round-trip
recovering 4/4 tones under a strict gate, both LoRA directions, and every audio
level β all structural or systematic, and none of them depends on seeding.
**And read Β§3 before treating any of this as a deployment.** β
The Space now
runs and generates β βΈοΈ **for one of the eleven modes.** Everything in this
section is a **hinode** result; the Space has proven `text_to_video` at
256Γ256Γ9 and nothing else. Those are different claims about different machines,
and Β§3 is where they are kept apart.
β *β Corrected (2026-08-25, evening): the Space had proven that `text_to_video`
**executes**; its output was colour static. **This paragraph's instinct β keep
Β§2's hinode results apart from anything the Space says β was the correct one**,
and every later annotation that softened it was wrong. Everything in this
section is hinode, bf16, unquantised, and unaffected.*
> β ***And that supersession is itself corrected (2026-08-25, evening):**
> "content-verified" meant non-empty and non-frozen, on the broken int8 build.
> All eleven modes **execute** on the Space; none of their output was ever
> verified as an image. Banner at the top of this file.*
>
> β
**Superseded (annotated 2026-08-25): the Space has since proven all eleven
> modes at 256Γ256Γ9** β content-verified, sha `287ba9c`, 2026-08-22
> (`SPACE_MODE_MATRIX.md`). What it has still proven nothing about: stage 2,
> LoRA, `auto_duration`, and any resolution anyone would ship. Β§3's annotation
> carries the merged state.
>
> *And later on 2026-08-25 that residue shrank to `auto_duration` alone (plus
> the stage-2 ceiling's exact position): stage 2, LoRA and 1216Γ704Γ121 all
> ran on the Space that day. Top-of-file note; `SPACE_SESSION_2026-08-25.md`.*
# Operator handoff β LTX-2.5 Studio
> ## Status on 2026-11-16
>
> This handoff is the August 2026 record. Its body is kept as written; it no longer says where
> the project stands. Superseded since:
>
> - **Stage 2's "ceiling between 25 and 49 frames"** (the two blocks below): the 49-frame failure
> was the 47.375 GiB `large` partition. 704Γ480Γ49 passes on `xlarge` (55.906 GiB reserved,
> [`MIG_ALLOCATOR_SIGHTINGS.md`](MIG_ALLOCATOR_SIGHTINGS.md) Β§2). The ceiling on `xlarge` and
> stage 2's picture quality are not measured.
> - **"Total day β€ ~675.5 GPU-s"**: the quota ledger for UTC 2026-08-25 reads β€ 1528.672 GPU-s
> over 20 runner calls. Adding the β€ ~93.3 GPU-s of manual calls bounded by hand in
> [`SPACE_SESSION_2026-08-25.md`](SPACE_SESSION_2026-08-25.md) Β§8 puts the day's bound at
> ~1622 GPU-s (wall Γ 2 upper bounds, not billed seconds), about 84 % of the 1920 stop line.
> The evening's manual calls and all of 2026-08-26/27 were never recorded.
> - **"The post-fix eleven-mode matrix goes in `SPACE_MODE_MATRIX.md` Β§10"**: it ran that
> evening but was never written up, and its video and audio files are gone.
> - **Β§3 "82.2 GB of 150 [OK]"**: read before ZeroGPU packs (the packed term was 0.0). On the
> 2026-11-13 boot the platform's own progress bar read `40.6G` packed; the disk peak has never been
> measured.
> - **Β§5 "On a Space"**: "deploying is a push to that remote's `main`" is wrong as stated. The
> Space repo has its own root commit, so pushing a work commit is rejected as non-fast-forward;
> every deploy so far was a merge commit on the live Space head
> (`git commit-tree <work>^{tree} -p <work> -p <Space head>`) pushed as a fast-forward. Never
> `--force`. The 72.19 GB download happened at startup, not at build time: the 2026-11-13 cold
> start took 13 min 59 s, about 11.5 min of it the download. `tests/test_deploy_drift.py`
> does not assert `startup_duration_timeout`.
> - **Where the weights come from** (Β§5; Β§3's "private-Space fetch path"; the deploy table's
> `startup_duration_timeout` and token rows): the Space is public now, and the code has the
> image bring the weights. `preload_from_hub` in the README's front matter fetches the 35
> model files the int8 Space loads (71.92 GB; the temporal upsampler is no longer fetched) and
> the prompt enhancer's 8 files (2.49 GB) before the app starts. `preflight.ensure_weights()`
> then checks the cache, downloads only what is missing, and refuses before writing a byte a
> download the 150 GB disk could not hold. How long a cold start takes this way: deploy 2's
> check on the Space, on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32).
> - **Β§5 "On hinode"**: hinode is retired. `scripts/sync_to_hinode.sh`, `scripts/hinode_run.sh`
> and every `scripts/host_*.py` were removed on 2026-11-16 (`git show 2f356b5:scripts/<name>`
> restores one).
> - **`auto` as the GPU size**: `LTX25_ZEROGPU_SIZE` now defaults to `xlarge` in code.
> - **"local MPS bf16" sign-off**: those artifacts came from tooling outside this repo, which had
> no Apple-silicon path. One was added on 2026-11-16 and verified on real weights (see
> `README.md`).
>
> Where the project stands: [`../README.md`](../README.md) (what the app does, how to run it,
> what it costs), [`SPACE_MODE_MATRIX.md`](SPACE_MODE_MATRIX.md) Β§10 (what has been
> verified, and how) and [`RELEASE_NOTES.md`](RELEASE_NOTES.md) (what each deploy contains).
> Deploy 1 (Space commit `0cd54f2`, the tree of `1e9fd20`, 2026-11-16) ran all eleven modes on
> the Space, each looked at (`SPACE_MODE_MATRIX.md` Β§10.1); deploy 2 (2026-11-17) replaced it,
> and its own check on the Space is reported on [issue #32](https://github.com/techfreakworm/ltx2.5-studio/issues/32).
> **What you have, what you do not have, and what to do next.** Written for
> someone who was not here. [`PLAN.md`](PLAN.md) records how things were
> decided; [`TEST_PLAN.md`](TEST_PLAN.md) records what was run; this tells you
> where the project actually stands.
**Every load-bearing number below is marked with where it came from:**
β
*measured on hinode* Β· π *read off a pinned source or config* Β·
β οΈ *inferred* Β· βΈοΈ *not yet established*.
If a number carries no mark, treat that as a defect in this document.
> ## β READ FIRST β 2026-08-25, evening: the deployed Space has NEVER produced a real image
>
> **Every generation result this document attributes to the Space was produced by
> a broken transformer.** Every Space sha up to and including `ff142d9` ran
> blanket bitsandbytes int8 with no `llm_int8_skip_modules`, so bnb quantised the
> LTX-2.5 **conditioning path** β the eight top-level `LTX2AdaLayerNormSingle`
> modulation heads and the in/out projections β along with everything else. The
> denoiser stopped being steered by the timestep and the prompt, and every Space
> artifact ever banked is **colour static or a flat field**: correctly shaped,
> high variance, fully deterministic, and not a picture.
>
> β
**Fixed at `eb1aa9f`** (316 of 1660 Linears spared, names read off the real
> checkpoint), **deployed as Space sha `e3b5333`**, and proven there β the same
> seed, prompt and geometry that produced colour static now produce a
> photographic frame. Before/after:
> `~/.soma/leads/ltx25-studio/frames/COMPARE_before_after_skipfix.png` π.
>
> **β
MECHANICAL claims survive, all of them**: all eleven modes execute on the
> Space; both fork-boundary directions carry their cargo; determinism and
> byte-reproducibility hold; tiled decode enables 1216Γ704Γ121; stage 2 runs and
> doubles geometry with a ceiling between 25 and 49 frames; LoRA applies,
> detaches, and both routes agree; the disk, RAM and quota arithmetic.
>
> **β OUTPUT claims do not survive, none of them.** Every "content-verified"
> Space artifact in this file was verified **non-empty and non-frozen, never as
> an image**. Specifically: the conditioning-fidelity conclusion (H4b), the LoRA
> inertness check (H12b) and the stage-2 pass were all measured on garbage β they
> show the code paths ran, not that the output was correct.
>
> **β `b78803cc` is byte-reproducible NOISE.** The determinism claim is intact
> and its artifact is worthless.
>
> **β Β§2 is UNAFFECTED, and this is the load-bearing distinction in this
> document.** hinode ran **bf16, unquantised** (Β§2.1's `base.dtype` row records
> it), and the operator's only visual sign-off was on **local MPS bf16**
> artifacts served over tailscale. Every bf16 path anyone has eyeballed looked
> right; the only path ever eyeballed on the Space was broken. Β§3's discipline
> was correct all along, and the annotations that merged the two evidence bases
> were wrong: **they merged across a dtype boundary no run ever crossed.**