LTX2.5-Studio / docs /OPERATOR_HANDOFF.md
techfreakworm's picture
docs: every sentence stays true after the deploy: README and the release notes say this is deploy 2's code and point to issue #32 for what the Space shows, and no doc says the Space still runs deploy 1, is not deployed, or has not been redeployed since August (#32)
abc9150 unverified
|
Raw History Blame Contribute Delete
68.9 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Operator handoff β€” LTX-2.5 Studio

Status on 2026-11-16

This handoff is the August 2026 record. Its body is kept as written; it no longer says where the project stands. Superseded since:

  • Stage 2's "ceiling between 25 and 49 frames" (the two blocks below): the 49-frame failure was the 47.375 GiB large partition. 704Γ—480Γ—49 passes on xlarge (55.906 GiB reserved, MIG_ALLOCATOR_SIGHTINGS.md Β§2). The ceiling on xlarge and stage 2's picture quality are not measured.
  • "Total day ≀ ~675.5 GPU-s": the quota ledger for UTC 2026-08-25 reads ≀ 1528.672 GPU-s over 20 runner calls. Adding the ≀ ~93.3 GPU-s of manual calls bounded by hand in SPACE_SESSION_2026-08-25.md Β§8 puts the day's bound at ~1622 GPU-s (wall Γ— 2 upper bounds, not billed seconds), about 84 % of the 1920 stop line. The evening's manual calls and all of 2026-08-26/27 were never recorded.
  • "The post-fix eleven-mode matrix goes in SPACE_MODE_MATRIX.md Β§10": it ran that evening but was never written up, and its video and audio files are gone.
  • Β§3 "82.2 GB of 150 [OK]": read before ZeroGPU packs (the packed term was 0.0). On the 2026-11-13 boot the platform's own progress bar read 40.6G packed; the disk peak has never been measured.
  • Β§5 "On a Space": "deploying is a push to that remote's main" is wrong as stated. The Space repo has its own root commit, so pushing a work commit is rejected as non-fast-forward; every deploy so far was a merge commit on the live Space head (git commit-tree <work>^{tree} -p <work> -p <Space head>) pushed as a fast-forward. Never --force. The 72.19 GB download happened at startup, not at build time: the 2026-11-13 cold start took 13 min 59 s, about 11.5 min of it the download. tests/test_deploy_drift.py does not assert startup_duration_timeout.
  • Where the weights come from (Β§5; Β§3's "private-Space fetch path"; the deploy table's startup_duration_timeout and token rows): the Space is public now, and the code has the image bring the weights. preload_from_hub in the README's front matter fetches the 35 model files the int8 Space loads (71.92 GB; the temporal upsampler is no longer fetched) and the prompt enhancer's 8 files (2.49 GB) before the app starts. preflight.ensure_weights() then checks the cache, downloads only what is missing, and refuses before writing a byte a download the 150 GB disk could not hold. How long a cold start takes this way: deploy 2's check on the Space, on issue #32.
  • Β§5 "On hinode": hinode is retired. scripts/sync_to_hinode.sh, scripts/hinode_run.sh and every scripts/host_*.py were removed on 2026-11-16 (git show 2f356b5:scripts/<name> restores one).
  • auto as the GPU size: LTX25_ZEROGPU_SIZE now defaults to xlarge in code.
  • "local MPS bf16" sign-off: those artifacts came from tooling outside this repo, which had no Apple-silicon path. One was added on 2026-11-16 and verified on real weights (see README.md).

Where the project stands: ../README.md (what the app does, how to run it, what it costs), SPACE_MODE_MATRIX.md Β§10 (what has been verified, and how) and RELEASE_NOTES.md (what each deploy contains). Deploy 1 (Space commit 0cd54f2, the tree of 1e9fd20, 2026-11-16) ran all eleven modes on the Space, each looked at (SPACE_MODE_MATRIX.md Β§10.1); deploy 2 (2026-11-17) replaced it, and its own check on the Space is reported on issue #32.

What you have, what you do not have, and what to do next. Written for someone who was not here. PLAN.md records how things were decided; TEST_PLAN.md records what was run; this tells you where the project actually stands.

Every load-bearing number below is marked with where it came from: βœ… measured on hinode Β· πŸ“„ read off a pinned source or config Β· ⚠️ inferred Β· ⏸️ not yet established. If a number carries no mark, treat that as a defect in this document.

β›” READ FIRST β€” 2026-08-25, evening: the deployed Space has NEVER produced a real image

Every generation result this document attributes to the Space was produced by a broken transformer. Every Space sha up to and including ff142d9 ran blanket bitsandbytes int8 with no llm_int8_skip_modules, so bnb quantised the LTX-2.5 conditioning path β€” the eight top-level LTX2AdaLayerNormSingle modulation heads and the in/out projections β€” along with everything else. The denoiser stopped being steered by the timestep and the prompt, and every Space artifact ever banked is colour static or a flat field: correctly shaped, high variance, fully deterministic, and not a picture.

βœ… Fixed at eb1aa9f (316 of 1660 Linears spared, names read off the real checkpoint), deployed as Space sha e3b5333, and proven there β€” the same seed, prompt and geometry that produced colour static now produce a photographic frame. Before/after: ~/.soma/leads/ltx25-studio/frames/COMPARE_before_after_skipfix.png πŸ“„.

βœ… MECHANICAL claims survive, all of them: all eleven modes execute on the Space; both fork-boundary directions carry their cargo; determinism and byte-reproducibility hold; tiled decode enables 1216Γ—704Γ—121; stage 2 runs and doubles geometry with a ceiling between 25 and 49 frames; LoRA applies, detaches, and both routes agree; the disk, RAM and quota arithmetic.

β›” OUTPUT claims do not survive, none of them. Every "content-verified" Space artifact in this file was verified non-empty and non-frozen, never as an image. Specifically: the conditioning-fidelity conclusion (H4b), the LoRA inertness check (H12b) and the stage-2 pass were all measured on garbage β€” they show the code paths ran, not that the output was correct.

β›” b78803cc is byte-reproducible NOISE. The determinism claim is intact and its artifact is worthless.

β›” Β§2 is UNAFFECTED, and this is the load-bearing distinction in this document. hinode ran bf16, unquantised (Β§2.1's base.dtype row records it), and the operator's only visual sign-off was on local MPS bf16 artifacts served over tailscale. Every bf16 path anyone has eyeballed looked right; the only path ever eyeballed on the Space was broken. Β§3's discipline was correct all along, and the annotations that merged the two evidence bases were wrong: they merged across a dtype boundary no run ever crossed.

β›” The method failure, as policy. Statistical gates cannot distinguish a render from mush. They are a pre-filter, never a verdict β€” they demote artifacts, they never promote one. No artifact is a PASS until a human or model eye has seen a frame at native resolution, and every report claiming an output is good carries the frame. This binds the structure detector added at e54477c too: it FAILS the flat grey (SR 0.5298) and PASSES colour static (SR 0.9183, against 0.9961–0.9968 for the real photograph), so even it cannot promote.

β›” The cheap-config lesson. The 256Γ—256Γ—9 config used for most of the campaign produced output too small and too blobby for anyone to judge by eye. That is how three weeks of confident nonsense got manufactured. A config too small to look at is a config that cannot be verified. Β§2.2's own "Structure is not content" and Β§4.1's verify_streams-exit-0-on-a-silent-file had both already written down why β€” and "content-verified" was allowed to mean non-empty bytes anyway.

Full record: the invalidation blocks at the top of SPACE_MODE_MATRIX.md and SPACE_SESSION_2026-08-25.md. The post-fix eleven-mode matrix goes in SPACE_MODE_MATRIX.md Β§10.

βœ… State as of 2026-08-25 β€” read SPACE_SESSION_2026-08-25.md

A full GPU campaign ran on the Space that day and settled most of what this document lists as open: byte-level reproducibility sha256-VERIFIED (three containers, three code shas), 1216Γ—704Γ—121 generates (tiled decode, cdb3211), stage 2 ran on the Space (PASS at 704Γ—480Γ—25 input β†’ 1408Γ—960; FAIL at Γ—49 β€” its ceiling is between those), both LoRA directions proven with a passing inertness check (H12b's claim, delivered), conditioning shown to drive the output (H4b's question, answered by clip-swap), and the fork-boundary error fix proven live in both classes. Total day ≀ ~675.5 GPU-s of 2400.

β›” Corrected the same evening β€” see the block above. Every run in that campaign used the broken int8 build. The mechanical results stand exactly as written (reproducibility, geometry, ceilings, error paths, cost). The two OUTPUT claims in this paragraph do not: "both LoRA directions proven with a passing inertness check (H12b's claim, delivered)" β€” the adapter changed one garbage render into a different one, and H12b is NOT delivered β€” and "conditioning shown to drive the output (H4b's question, answered by clip-swap)" β€” both clip-swap outputs were colour static, and H4b is OPEN again. H10b is not the last discriminator standing; all three are.

What genuinely remains open after that session: the stage-2 ceiling (diffusion-decoder chunking β€” its own investigation), the video_to_audio βˆ’19.5 dBFS observation (now consistent-with the conditioning finding, still not directly probed β€” β›” that "consistent-with" is withdrawn: it derived from the conditioning finding, which is withdrawn; and βˆ’19.5 dBFS was audio generated by a broken transformer), the 2026-08-20 SIGKILL history (documented-unexplained, by operator ruling) β€” and, still never exercised on the platform: auto_duration and the H10b audio-input discriminator.


1. What this is

A Gradio studio for LTX-2.5 (Lightricks' 22B video+audio model) covering all eleven generation modes, with user-uploadable LoRA validated at inference time. One model load serves every mode. Python 3.12, diffusers pinned to an exact SHA, weights pinned to an exact revision.

Destination is a private HuggingFace ZeroGPU Space. βœ… It runs and generates there β€” startup, one clean GPU worker, and a content-verified video, at Space sha 8fb292d6.

β›” Corrected (2026-08-25, evening): "a content-verified video" was a non-empty, non-frozen file. It was colour static β€” 8fb292d6 ran the blanket int8 that quantised the conditioning path. What stands: the Space runs, startup completes, a GPU worker forks cleanly, and a well-formed artifact comes back. What does not: that anything it returned was a video in any sense a user means. Fixed at eb1aa9f, deployed e3b5333. Banner at the top of this file.

⏸️ That is one mode, once. Ten of the eleven have never executed on that platform; Β§2's eleven-of-eleven is a hinode result and stays one. Β§3 is the section that keeps those two evidence bases apart β€” read it before treating either as the other.

β›” The annotation below is itself corrected (2026-08-25, evening): "content-verified" meant non-empty and non-frozen, and sha 287ba9c ran the broken int8. All eleven modes execute on the Space β€” that half stands and is the useful half. Nothing about their output does. Banner at the top.

βœ… SUPERSEDED (run 2026-08-22, annotated 2026-08-25): all eleven modes now generate on the live Space, content-verified, at sha 287ba9c β€” SPACE_MODE_MATRIX.md is the record. "One mode, once" was true when written and is no longer the state: eleven-of-eleven is now also a Space claim, ⏸️ at one configuration only (256Γ—256Γ—9, stage 2 off, no LoRA, no auto_duration). Stage 2, LoRA, and every shippable resolution remain hinode-only. Β§3's separation of the two evidence bases is what made the merge legible β€” its annotation carries the detail.

Narrowed further on 2026-08-25: stage 2 (to its 25↔49-frame ceiling), LoRA (both directions) and 1216Γ—704Γ—121 have now run on the Space too β€” see the top-of-file note. Still hinode-only: the duration head.


2. HOST-PROVEN

Everything in this section ran on hinode: an unpartitioned RTX PRO 6000 with 124 GB RAM βœ… β€” through get_pipeline exactly as app.py calls it, not through a test harness.

2.1 Startup gates

Gate Result
Weights present 37 / 37 files, 72.185 GB βœ…
Full repo, for contrast 56 files, 164,192,096,189 B (164.19 GB) πŸ“„ manifest
Components shared between the two pipelines 9 / 9 identical by is βœ…
base.dtype after deriving torch.bfloat16, unchanged βœ…
CUDA cost of the second pipeline βˆ’1024 bytes βœ…
Total for both pipelines 65.370 GiB βœ…

The βˆ’1024 bytes is the number to re-check after any change to pipeline construction: it is what proves the second workflow shares the first's weights rather than loading its own copy. A positive figure in the tens of GB means the sharing has broken.

2.2 Modes

Baseline is 704 Γ— 480 Γ— 121 frames, seed pinned to 1234. "Streams" is what tests/verify_streams.py observed in the container, not a human reading.

# Mode Result Streams observed Size Time
H1 text_to_video βœ… h264 704Γ—480@24, f=121 791,782 B 8.4 s
H2 image_to_video βœ… h264 704Γ—480@24, f=121 307,583 B 7.8 s
H3 image_text_to_video βœ… h264 704Γ—480@24, f=121 775,213 B 8.0 s
H4 video_to_video βœ… h264 704Γ—480@24, f=121 708,885 B 8.6 s
H5 text_to_audio_video βœ… h264 + aac 48000 Hz 2 ch 792,803 B 7.8 s
H6 image_to_audio_video βœ… h264 + aac 48000 Hz 2 ch 374,184 B 7.9 s
H7 image_text_to_audio_video βœ… h264 + aac 48000 Hz 2 ch 995,843 B 8.2 s
H8 text_to_audio βœ… pcm_s16le 48000 Hz 2 ch, no video 961,998 B 1.8 s
H9 video_to_audio βœ… pcm_s16le 48000 Hz 2 ch, no video 961,998 B 2.0 s
H10 audio_to_audio βœ… pcm_s16le 48000 Hz 2 ch, βˆ’10.4 dBFS 961,998 B 2.8 s
H11 audio_to_video βœ… h264 704Γ—480@24, f=121, no audio 193,825 B 8.0 s

Eleven of eleven. All βœ… measured on the real get_pipeline path, no bypass and no harness accommodation; verify_streams returned 0 on every one.

beyond the baseline
H14 stage 2 1408 Γ— 960, f=121 1,807,646 B Β· 35.2 s
H15 duration_head 704 Γ— 480, f=97 not 121 593,527 B Β· 5.9 s
H16 high resolution 1216 Γ— 704 + aac stereo 1,775,743 B Β· 21.0 s
H17 mel round-trip 4.000 s in β†’ 4.010 s out, 4/4 tones strict gate
H12 / H13a / H13b LoRA accepted / rejected / rejected stage 6.5

Wall-clock excludes the model load, which is 14.0 s once per batch βœ….

⚠️ Structure is not content

Every run here passed verify_streams. The one defect that mattered most was invisible to it: video_to_audio came back 50 dB below its siblings with correct codec, channels, rate, duration and a non-empty file. Silence and a healthy signal have identical stream layout.

It was found because someone measured RMS on an artifact that had already been signed off, and it led to a defect affecting all twelve modes (gap 7). A green table says these are well-formed files. It is a weaker claim than "correct output", and the gap between the two is where this build spent its afternoon.

H8 and H9 are byte-identical in size because uncompressed PCM of equal duration always is. That is why size proves nothing about content, and why it took an RMS measurement to notice.

On audio levels generally. Measured levels sort by condition type, not by mode βœ… β€” text-only βˆ’5.2 to βˆ’6.2 dBFS, image-conditioned βˆ’8.2 to βˆ’15.2, video-conditioned at strength 0.8 βˆ’54.5 to βˆ’55.0. The text-only group peaks at exactly 1.0000 with no samples exceeding, so the export is faithful and the saturation is upstream, in the model, not in our writer. Those files are consistent with each other and sit at the rail; ⏸️ whether that is correct for this model has not been established.

2.5 What the recipe fix bought β€” before / after

Same seed, same geometry on both sides of every pair; only the distilled recipe (gap 7) differs.

# mode before after wall-clock
H1 text_to_video 886,265 791,782 25.7 β†’ 8.4 s
H2 image_to_video 432,486 307,583 26.3 β†’ 7.8 s
H3 image_text_to_video 1,130,178 775,213 26.6 β†’ 8.0 s
H4 video_to_video 727,490 708,885 27.0 β†’ 8.6 s
H5 text_to_audio_video 1,091,468 792,803 25.0 β†’ 7.8 s
H6 image_to_audio_video 620,537 374,184 26.6 β†’ 7.9 s
H7 image_text_to_audio_video 784,282 995,843 26.7 β†’ 8.2 s
H8 text_to_audio 961,998 961,998 5.6 β†’ 1.8 s
H9 video_to_audio 961,998 961,998 6.0 β†’ 2.0 s
H14 stage 2 2,254,799 1,807,646 84.3 β†’ 35.2 s
H15 duration_head 684,374 593,527 19.4 β†’ 5.9 s
H16 1216 Γ— 704 2,331,528 1,775,743 68.7 β†’ 21.0 s

Geometry is identical on both sides of every pair β€” H14 stays 1408 Γ— 960, H15 stays 97 frames, the rest 704 Γ— 480 Γ— 121. Sizes changed; geometry did not is the shape of the result.

Three things a reader needs:

  • H7 is the only artifact that grew. h264 size tracks content complexity, so that is neither good nor bad on its own. Flagged, not interpreted.
  • H8 and H9 are byte-identical because uncompressed PCM at one duration, rate and channel count always is. Their change is entirely in level, not in bytes β€” see below β€” so reading this table alone would call them unchanged when they are two of the three that moved most.
  • ~3Γ— faster everywhere. At guidance_scale=1.0 with STG and modality zeroed there are no classifier-free branches, so it is one forward pass per step instead of two or three. This is the one column here that seeding does not confound β€” wall-clock is not drawn from a distribution β€” and it follows from the mechanism rather than merely accompanying it.

⚠️ These pairs are not yet controlled β€” the audio evidence is

The seed never reached the model until 4ebb8b9 (gap 8), so both halves of every pair above were unconditionally random. The pixel differences cannot evidence that the recipe fix caused them β€” two runs of one mode differed anyway. βœ… Seeding is fixed, and verified byte-identical for audio_to_audio.

β›” The seeded gallery that was to supersede this table is DROPPED. It lived on a host that no longer exists, and nothing replaced it. So this table is permanently unattributable, not temporarily so β€” these figures remain as the record of what the defect cost, and that is now all they will ever be.

The audio levels do stand on their own, because they are systematic and orders of magnitude outside run-to-run variation:

before after
H9 video_to_audio βˆ’55.0 dBFS βˆ’17.8 dBFS
H8 text_to_audio βˆ’5.2 dBFS, peak 1.0000 (railed) βˆ’15.3 dBFS, peak 0.6953
highest peak, any mode 1.7811 1.0050

A 50 dB gap and a clipped peak going away are not a seed.

β›” And the replacement will not arrive. Bitwise reproducibility is verified for audio_to_audio. The video path has since been evidenced on the Space β€” a run returned bytes identical to one banked the previous day on a different container at the same seed β€” but ⏸️ evidenced is not proven: no two-run frame-hash comparison exists. See docs/SPACE_MODE_MATRIX.md Β§5. The gallery that was to close this is dropped with the host it lived on.

β›” Correction (2026-08-25) on "returned bytes identical": that overstates what was measured. The cross-day match was file size (39,146 B) plus every decoded statistic to three decimals β€” no hash of the day-1 artifact was ever recorded, and that file is gone, so byte identity against it is permanently unverifiable. The "evidenced is not proven" hedge above was right, and even understated: no byte comparison existed either. The planned close is a same-seed re-run sha256-compared against the surviving 2026-08-22 artifact β€” SPACE_MODE_MATRIX.md Β§5's correction carries the hash.

βœ… Sizes differed between the two code paths β€” almost certainly just the seed

Three modes were run through a bypass (_base_pipeline plus an injected StageRunner) and through the real get_pipeline path, all pre-fix:

bypass real path
H1 865,447 B 886,265 B
H2 577,051 B 432,486 B β€” 25% smaller
H3 1,040,007 B 1,130,178 B

This was recorded as an open question β€” "GPU non-determinism, or a genuine difference between the routes?" β€” and described as "at the same seed."

It was not at the same seed. The seed reached nothing (gap 8). These are six unconditionally random runs, and two runs of one mode differed by this much anyway. There was never an anomaly to explain, and the phrase "at the same seed" was doing all the work: it made an ordinary difference look like a finding.

⏸️ Cheap to settle properly now β€” run one mode both ways at a fixed seed. Nobody has, so it is probably the seed rather than certainly.

The advice it produced stands on its own and is worth more than the question: assert stream layout, never an exact size.

2.3 Beyond the baseline

Run What it proves Result
H16 high resolution usability, not just correctness βœ… text_to_audio_video at 1216 Γ— 704 @24, f=121 + aac 48000 Hz 2 ch β€” 1,775,743 B, 21.0 s
H15 duration_head the model chose the length, rather than a default leaking through βœ… f=97, not 121 β€” 704 Γ— 480, 593,527 B, 5.9 s
H14 stage 2 upsample β†’ 3-step schedule β†’ diffusion decode βœ… 1408 Γ— 960, f=121 β€” 1,807,646 B, 35.2 s

Post-fix figures, matching Β§2.2. Β§2.5 carries the pre-fix column.

H15 is worth understanding: num_frames is omitted and the duration head picks. Had it returned 121 β€” the default β€” the run would have looked identical to a pass while proving the opposite. 97 is the whole result.

H14's proof is its resolution, not a comparison. βœ… 1408 Γ— 960 against H1's 704 Γ— 480 is exactly 2Γ— in both dimensions β€” a size only the spatial upsampler can produce, and the one claim here that no vintage or seed can affect, since geometry is identical across every before/after pair. That is stronger evidence than the same-seed diff the matrix planned, because a same-seed comparison shows a difference while this shows the specific difference stage 2 exists to make.

2.4 The LoRA validator, both directions

Run Adapter Result
H12 joyfox/LTX-2.3-Transition-LORA βœ… ACCEPTED, 576 modules across transformer + connectors
H13a charlied98/… lora_weights_step_03000 βœ… REJECTED at stage 6.5, LoRADimensionError
H13b same repo, comfy_lora_weights_step_03000 βœ… REJECTED at stage 6.5, same mechanism

H12 carries mixed dimensions β€” 4096 at attention, 2048 at audio_ff, 16384 at ff.net.2 βœ… β€” and none of that is a mismatch. An earlier "expect hidden dim 4096" rule would have rejected a perfectly good adapter; the false-positive guard is what stops that. H13a and H13b differ only in key prefix (transformer. vs diffusion_model.) and reach the same verdict by the same mechanism, which is what makes the rejection trustworthy rather than lucky.


3. SPACE-PROVEN vs HOST-PROVEN β€” read this before believing Β§2

βœ… The Space runs and generates. ⏸️ That is one mode, once.

text_to_video at 256Γ—256Γ—9 produced a real, content-verified video on the live private Space βœ…. Ten of the eleven modes have never executed there.

β›” Corrected (2026-08-25, evening): it was not a real video. It was colour static β€” the origin claim of the entire Space evidence base, and it was wrong from the first run. "Content-verified" meant non-empty and non-frozen. What survives: the Space runs, and one mode executed there. Banner at the top.

A green table on a dedicated host and a working deployment are different claims. Both have now been made β€” about different things. That is exactly why this section still exists: the two evidence bases have diverged rather than merged, and Β§2's eleven-of-eleven remains a hinode claim.

βœ… SUPERSEDED (run 2026-08-22, annotated 2026-08-25). The mode matrix ran on the live Space and all eleven modes generate there, content-verified, at sha 287ba9c β€” SPACE_MODE_MATRIX.md. The two evidence bases have merged at exactly one configuration: 256Γ—256Γ—9, stage 2 off, no LoRA, no auto_duration.

β›” THE MERGE NEVER HAPPENED (corrected 2026-08-25, evening). hinode ran bf16, unquantised β€” Β§2.1's base.dtype row records exactly that β€” and the Space ran blanket int8. The two evidence bases were separated by a dtype boundary no run ever crossed, and that difference was the whole defect. This section's original discipline was right and this annotation was the error: it merged on "the same mode name at the same geometry" when the thing that differed was the quantisation of the conditioning path. Keep Β§3. Banner at the top. What is still hinode-only: stage 2 (1408Γ—960), the duration head, LoRA in both directions, the mel round-trip at full length, and every shippable resolution. This section's discipline is unchanged β€” it now keeps configurations apart instead of platforms.

That hinode-only list shrank again on 2026-08-25: stage 2 reached 1408Γ—960 on the Space (from ≀25-frame input; its decoder ceiling is between 25 and 49 frames), LoRA ran in both directions, and 1216Γ—704Γ—121 passed via the tiled decode. Remaining hinode-only: the duration head, and full-length audio work. SPACE_SESSION_2026-08-25.md.

Detailed record: SPACE_GENERATION_FINDING.md β€” the first deploy, the killed workers, the falsified ceiling, and the run that generated. Not restated here on purpose.

hinode is an unpartitioned card with 124 GB RAM and an ordinary filesystem. ZeroGPU is none of those things. The first deploy answered several of these questions and changed the answer to one of them:

  • βœ… Tensor packing β€” exercised. Packing ran and the post-pack prune fired, freeing 65.25 GB.

  • βœ… The 150 GB ephemeral-disk ceiling β€” never the binding constraint. The disk line read 82.2 GB of 150 [OK] while the workload died of something else. Arithmetic in ZEROGPU_DISK.md.

  • β›” RAM β€” the ~68 GB figure is FALSIFIED. See the finding below; this is the single most important correction in this document.

  • βœ… The private-Space fetch path β€” works. 37/37 files at the pinned revision, downloaded at runtime because preload_from_hub 401s on a private repo.

  • βœ… The fork boundary β€” crossed. A worker forked, ran, and released cleanly (fail=false β†’ 200).

  • βœ… The size kwarg's plumbing β€” the name survived gpu_decorator_kwargs's signature filter and raised no import-time TypeError. That is the risk PLAN Β§5.8 actually named β€” the platform pins its own spaces and lags PyPI β€” so this is a real βœ… on its own terms.

  • β›” An explicit size value β€” never passed, and THAT WAS THE BUG. β†’ RESOLVED (2026-08-27), and the inference above was wrong. app.py sent size=None whenever LTX25_ZEROGPU_SIZE was auto, which was its default and what the Space ran. The reasoning recorded here β€” "ZeroGPU chose xlarge by inference, packed 51.1 GB exceeds the 42.00 GiB threshold" β€” was the fourth document in this repo to assert it and none of the four had followed the promoted value past the function that sets it. It never reaches the scheduler: default_gpu_size() is consumed only by get_duration_seconds (client.py:109) and feeds only the duration multiplier, while gpuSize is sent to the platform only when the decorator's size is not None (api.py:112-113). βœ… auto therefore ran the Space on large β€” 47.375 GiB β€” for its entire life, instead of xlarge's 94.971 GiB. Measured from inside the worker, both before and after. LTX25_ZEROGPU_SIZE is now xlarge. Full trace: MIG_ALLOCATOR_SIGHTINGS.md Β§1. ⚠️ Note the exact shape of the error, because it is the interesting part: the reservation really was computed as xlarge (no 1.5Γ— duration_factor). The hardware was large. We were priced as one and allocated the other.

  • ⏸️ Quota metering β€” still unreadable. All three candidate endpoints 404, so billed seconds can only be bounded, never measured.

  • ⏸️ Ten of eleven modes β€” no conditioning, no stage 2, no LoRA, and no audio path has run on ZeroGPU. The audio modes exercise the mel front-end, the av loader and the 16 kHz/48 kHz asymmetry; none of that has executed there once.

    β€” Annotation (2026-08-25): mostly falsified by the 2026-08-22 mode matrix. Image, audio and video conditioning have run on ZeroGPU; both audio-input modes passed; the mel front-end and the 16 kHz/48 kHz asymmetry were exercised there and passed (SPACE_GENERATION_FINDING.md Β§11 item 10, SPACE_MODE_MATRIX.md). Still true, and only these: stage 2, LoRA and auto_duration have never run on ZeroGPU.

    β€” β›” And that annotation is corrected in turn (2026-08-25, evening): "passed" meant a well-formed, non-empty, non-frozen file from a transformer whose conditioning path had been quantised away. Those paths executed on ZeroGPU β€” the mel front-end, the av loader, the 16 kHz/48 kHz asymmetry all really ran, and that half stands. The ⏸️ this replaced was cautioning about something real, and it should not have been retired on the strength of "passed". Banner at the top.


The three findings from the first deploy

1. β›” The ~68 GB RAM ceiling is FALSIFIED

It was a community cgroup observation β€” inherited, never ours, and load-bearing in planning since before any deploy. Startup reached 87.13 GiB = 93.56 GB peak RSS and was not killed.

βœ… Replace it with our own measured lower bound: the ceiling is above 93.56 GB. Every headroom figure computed against ~68 GB was wrong by at least 25.56 GB.

2. ⚠️ Quantising more raised peak RSS by 31.24 GiB

Adding the text encoder to components_to_quantize moved RSS up, from 55.89 to 87.13 GiB, in a change made to bring it down. safetensors mmaps lazily; quantisation materialises those weights as anonymous memory, so quantising a component destroys the laziness that was keeping it cheap.

This is the same single mechanism ZEROGPU_DISK.md Β§4.2 already used to settle two other questions, now deciding a third β€” in the opposite direction from what we predicted.

β›” Annotation (2026-08-25, evening): this is the change that broke generation, and this document records it as a RAM result and nothing else. Widening components_to_quantize is the blanket int8 that quantised the conditioning path; from here until eb1aa9f the Space produced colour static in every mode. The RSS measurement above is correct and unaffected. What is missing β€” and was missing everywhere, README.md's int8 tier note included β€” is any sentence connecting quantisation to output fidelity. Nobody wrote one, so nobody looked.

3. ⏸️ The SIGKILL is still unexplained

cgroup-OOM is dead as an explanation: with the ceiling above 93.56 GB there was never the narrow margin the hypothesis required. βœ… Route C β€” instrument the worker β€” is now the LEADING path, not the fallback. It is the only route that returns evidence rather than another inference, and every inference made about this kill has been wrong at least once.

β›” And the fix is UNATTRIBUTED

Two things changed between the failing and succeeding runs: the quantisation widening and a full container rebuild. n=1 against n=1 with two variables. "The Space now serves" is established; "the widening is why" is not β€” and the hypothesis that motivated the widening is itself dead. Do not let any document claim the widening fixed generation.

βœ… VINDICATED, in the strongest available direction (2026-08-25, evening). That last sentence was written as a caution against over-crediting the widening. The widening broke generation: blanket int8 quantised the conditioning path, and every Space artifact from here to eb1aa9f is colour static. "The Space now serves" was established and was exactly as narrow as this paragraph insisted.

Cost so far: ≀ 243.58 GPU-seconds of 2400/day ⚠️ a bound, not a measurement β€” quota endpoint 404s.

Superseded (annotated 2026-08-25). That figure covered the first-generation session only. The mode-matrix day β€” UTC quota day 2026-08-21 β€” closed at ≀ ~413 GPU-s of 2400 (SPACE_MODE_MATRIX.md Β§7, matrix plus re-run; still a bound, never a measurement). Quota days are per-UTC-date, so as of 2026-08-25 the day is fresh and neither number describes current consumption.

One hinode number with a direct ZeroGPU consequence. Stage 2 peaked at 83.897 GiB βœ…, against 70.497 GiB for a single-stage run. So stage 2 cannot run on a large 48 GB MIG slice at all, and on xlarge (96 GB) it leaves roughly 11 GiB before any packing overhead. That is measured on an unpartitioned card; the packing overhead is ⏸️ unknown and comes off that margin.

One live risk that Part A has surfaced: the stage-2 diffusion decoder resolves a NATTEN kernel from the Hub at generation time πŸ“„, which is the only network dependency that fires after the user presses Generate. Whether it caches per image or per cold worker is ⏸️ unknown and decides its severity.


4. Known gaps, stated plainly

  1. The size kwarg has never carried a value. β›” CLOSED, AND IT WAS THE MOST EXPENSIVE ITEM IN THIS LIST (2026-08-27). The gap was written as "if anyone sets LTX25_ZEROGPU_SIZE to force a size, they are exercising an untested path" β€” with the unstated assumption that leaving it at auto was the safe, tested default. It was the opposite. auto sends no size, and ZeroGPU's own promotion never reaches the scheduler, so auto was the large branch: 47.375 GiB. βœ… The note above about large being a slice "that Β§3's own stage-2 measurement says cannot hold stage 2 at all" was therefore describing the partition the Space was already running on β€” and it predicted R4b's failure precisely, a day before anyone connected the two. LTX25_ZEROGPU_SIZE=xlarge is now set and βœ… verified from inside the worker at 94.971 GiB. Stage 2 at 704Γ—480Γ—49 passes on it: 53.08 s, 1408Γ—960, peak 55.906 GiB reserved. See MIG_ALLOCATOR_SIGHTINGS.md.

  2. H10 / H11 β€” the audio-input modes β€” have not passed. ⏸️ The path they need was connected to nothing until very recently: the UI collected a file, the encoder existed and was exported, and nothing called it. It is built and tested now, and it has not yet produced an artifact on the card. The acceptance test is H10b, not H10: two noise_scale values at the same pinned seed. Identical audio means the input is still not reaching the model, whatever the stream check says β€” because noise_scale=1.0 discards the input by design, so a bug that drops it entirely is indistinguishable from correct operation.

    β€” Annotation (2026-08-25): the first sentence was overtaken twice and this paragraph was never updated. Both modes then passed on the card β€” Β§2.2's own rows record H10 βœ… (961,998 B at βˆ’10.4 dBFS) and H11 βœ… (193,825 B) β€” and both passed again on the live Space (2026-08-22, SPACE_MODE_MATRIX.md). What stands is the acceptance-test framing: H10b has still never run, so "the user's audio reaches the model" remains ⏸️ unestablished; everything that has passed is consistent with the input being silently discarded.

  3. Stage 2 works; its NATTEN fetch is characterised. βœ… Resolved once per process, not per generation β€” two stage-2 generations in one process, one kernel fetch. So a Space needs outbound network at a worker's first stage-2 use rather than on every press of Generate. Still a real dependency, because each cold worker pays it once. Retained below for the mechanism.

    βœ… H14 ran and produced 1408 Γ— 960. But the diffusion decoder resolves its NATTEN kernel through a version β†’ ref lookup that goes over the network even when the kernel files are cached πŸ“„, and whether that fires once per process or once per generation is ⏸️ unknown. It cannot be grepped after the fact β€” the run log contains zero natten mentions β€” so answering it needs deliberate instrumentation. On a private Space this is the 401 class again, except firing after the user presses Generate.

  4. connectors: 0 on every accepted adapter. βœ… observed. Stage 7 asserts total modules attached > 0, which passes on the transformer alone β€” so an adapter that ought to touch connectors and silently does not would still be reported as attached. Probably correct for the three adapters tested (2.3/2B-era), so not currently a defect β€” but the assertion as written cannot distinguish "legitimately unused" from "silently missed". A per-component expectation would close it. See PLAN Β§4.4.

  5. temporal_latent_upsampler has no known role. πŸ“„ It is the same class as the spatial upsampler, differing only by axis, and upstream's component table and both documented recipes mention only the spatial one. We download it (0.2619 GB) and do not wire it in. Not a defect; an open question.

  6. βœ… video_to_audio near-silence β€” fixed, and the first diagnosis was wrong

    H9 came back at βˆ’55.0 dBFS against text_to_audio's βˆ’5.2 βœ…. The cause was the distilled recipe reaching only the stage-2 path (item 7). With that corrected, H9 runs at βˆ’17.8 dBFS β€” 2.5 dB from its siblings β€” at the unchanged condition strength of 0.8.

    before after
    H9 video_to_audio βˆ’55.0 dBFS, peak 0.0146 βˆ’17.8 dBFS, peak 0.5078
    H8 text_to_audio βˆ’5.2 dBFS, peak 1.0000 (railed) βˆ’15.3 dBFS, peak 0.6953
    gap ~50 dB 2.5 dB

    One mode 50 dB down and another driven into the rail were the same misconfiguration seen from opposite ends.

    ⚠️ The first diagnosis blamed condition strength and was wrong. A single-lever sweep at fixed seed gave 0.8 β†’ βˆ’55.0 and 0.5 β†’ βˆ’5.1, and that was read as the cause. It was a real interaction between condition strength and the broken guidance β€” a genuine effect inside a pipeline that no longer exists. A per-mode strength cap was written, committed and reverted.

    A single-lever same-seed sweep is strong evidence that something moves, and no evidence about why.

    The source citation said throughout that a video condition cannot attenuate audio in the denoise loop β€” the conditioning mask is applied to noise_pred_video and never touches noise_pred_audio. It was correct, and a clean correlation was believed over it.

  7. The audio input rate is 16000, the vocoder output is 48000. πŸ“„ They are not symmetric. Anything that assumes one rate for both directions will pitch-shift audio without erroring.

  8. βœ… The distilled recipe reached only the stage-2 path β€” fixed

    Eleven of twelve runs sent no sigmas at all, so upstream built its own schedule and five guidance parameters took their SFT defaults β€” audio_guidance_scale at 7.0 against an intended 1.0.

    sent:      [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125]
    intended:  [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]
    

    βœ… All twelve artifacts differed before and after, with magnitudes ordering by conditioning strength (text-only 37–59 mean |diff|, image-conditioned 15–16, video-conditioned 3.71).

    ⏸️ Neither of those establishes the fix caused them, and the ordering is not the corroboration it looks like. Both halves were unseeded (gap 8), so two runs of one mode differed anyway β€” and a more strongly conditioned mode is more constrained, so its run-to-run variance is smaller too. The ordering is exactly what random variation would produce. It cannot separate the two explanations, and an earlier draft of this section claimed it could.

    What does stand, unconfounded:

    • the timings. 25.7 β†’ 8.4 s is not run-to-run variation, and it follows directly from the mechanism: unguided means one forward pass per step instead of two or three. The defect cost 3Γ— the compute, and that is measured rather than inferred.
    • the audio levels and peaks β€” βˆ’55.0 β†’ βˆ’17.8 dBFS, highest peak 1.7811 β†’ 1.0050. Systematic and orders of magnitude outside variation.
    • πŸ“„ that the schedule and guidance were wrong at all, which is a source fact and needs no run.

    It cost 3Γ— the compute as well as the quality. At guidance_scale=1.0 with STG and modality zeroed there are no classifier-free branches, so it is one forward pass per step rather than two or three:

    before after
    H1 text_to_video 25.7 s 8.4 s
    H8 text_to_audio 5.6 s 1.8 s
    H14 stage 2 84.3 s 35.2 s
    H16 1216Γ—704 + stereo 68.7 s 21.0 s

    Clipping went with it β€” highest peak 1.7811 β†’ 1.0050 βœ… β€” so no limiter is needed and outputs._write_audio's "what comes back is what the vocoder produced" principle stands. Our path never added gain; the material was arriving at Β±1.0. Stage 2's brightness falloff was this too, not stage 2: mean 99.48 β†’ 34.44 collapsing before, 82.52 β†’ 110.47 stable after.

  9. βœ… The seed reached only the sidecar β€” fixed, and verified byte-identical

    _resolve_seed ran and the number went into the encoder metadata. There was no generator anywhere in the package, so every run was unconditionally random and every "same seed" comparison was two independent runs.

    βœ… Fixed in 4ebb8b9: a generator seeded from the request, built on the pipeline's own device, threaded to the pipeline and to audio_vae.encode's .sample().

    βœ… Verified on the card, for audio_to_audio. Two runs at identical seed and configuration:

    run-a1d35aaaab06/output.wav   md5 2988f01159a5322afb28abb6e9a73d26
    run-b9dde07ed8bb/output.wav   md5 2988f01159a5322afb28abb6e9a73d26
    

    βœ… And the fix was complete when it landed, not repaired afterwards: git log -S 'generator' 4804da1..e4e8df1 -- ltx25_studio/ returns 4ebb8b9 alone.

    On this path, seed determines the output byte-for-byte, on GPU, with no deterministic-algorithm flags set. Stronger than expected β€” the weaker "seed-consistent but not bit-reproducible" was drafted and never needed.

    ⏸️ Scope, and it matters for Β§2.5: that measurement is one mode. The video path is not shown to be bitwise reproducible β€” it is evidenced on the Space (byte-identical output across two days and two containers at one seed, plus text_to_audio_video matching text_to_video's video statistics to three decimals) and no two-run frame-hash comparison has ever been made. docs/SPACE_MODE_MATRIX.md Β§5 states that limit explicitly.

    β€” Correction (2026-08-25): "byte-identical output across two days" is too strong β€” the cross-day evidence is size-identical plus statistics-identical; no day-1 hash exists and the day-1 file is gone, so byte identity was never checked and now cannot be. The conclusion of this passage β€” not shown to be bitwise reproducible β€” is unchanged and was the correct one. SPACE_MODE_MATRIX.md Β§5's correction records the planned sha256-compared re-run against the surviving 2026-08-22 artifact.

    β›” The twelve-mode pixel-diff gallery that depended on it is DROPPED β€” the host it ran on is gone. So the consequence is permanent rather than pending: the Β§2.5 comparison table was measured before seeding worked, its pixel diffs cannot attribute the change to the recipe fix, and nothing is coming to replace it. Everything structural was never affected β€” H14's exact 2Γ— resolution, H15's 97 frames, H17's frequency recovery, and every audio level.

    The lever that should have caught it was ours: decode_kwargs took a generator argument, with a docstring explaining why the diffusion decoder must be seeded, and read a key nothing ever set. See PLAN Β§9 rule 12.

  10. ⏸️ noise_scale and sigmas[0] disagree β€” open, deliberately unfixed

    πŸ“„ Upstream infers one from the other β€” "first (largest) sigma value if using custom sigmas" β€” and our stage_2_kwargs honours it. The audio-input modes do not: they send noise_scale while sigmas[0] stays 1.0.

    It was diagnosed as the cause of a silent audio_to_audio and it was not (Β§4.1). A fix was written, committed and withdrawn: truncating the schedule at noise_scale takes an eight-step run to two steps at 0.5 and one at 0.1 β€” values that measured healthy on eight.

    Agreeing with a contract is not evidence that a change improves anything. A test pins the current behaviour and explains why, so the next attempt arrives with a measurement rather than with agreement.


4.1 ⚠️ Known limitation β€” unusual audio input produces a silent file

Not a defect we can properly fix, and the most likely way a user meets a disappointing result.

audio_vae.encode is never called by the pipeline. Our encode_waveform_to_latents is the only producer of those latents, and nothing anywhere validates their distribution β€” not upstream, not us.

So audio that is far from what the model was trained on encodes to latents the model cannot use, and the result is a structurally perfect, inaudible file with no error and nothing to act on. βœ… A real instance, and it cost a day of misdiagnosis: a two-tone synthetic fixture at βˆ’14.8 dBFS in produced βˆ’87.9 dBFS out, with correct codec, channels, rate and duration, and verify_streams exit 0. The same mode fed a real recording produces βˆ’10.4 dBFS.

Likely triggers: pure tones, sine sweeps, test signals, near-silence, heavily synthesised material. Ordinary recorded audio is fine β€” every realistic input tried has worked.

β›” Do not "fix" this with an out-of-distribution detector. Reliable OOD detection at that boundary is not something this project can do, and a detector that is wrong in either direction is worse than the limitation.

⏸️ A narrower thing is worth doing and is deliberately not done yet. We measure the input level and can measure the output level, so a file that comes back essentially silent could say so in GenerationResult.warnings. The decision that makes it honest is that it must be a measurement, not a diagnosis: "this output is βˆ’87.9 dBFS, effectively silent" is a fact about the artifact, checkable and cause-free. "Your audio was out of distribution" is a claim we cannot support.

It is not built because it needs a threshold, a threshold needs evidence, and the PR opens with one named defect. Adding an unvalidated heuristic to the export path hours beforehand is the same move as shipping a schedule change on a contract reading β€” which was reverted for that reason. It belongs in the next pass, with a measured floor.


5. How to run it

On hinode

./scripts/sync_to_hinode.sh          # ships git archive HEAD, not the working tree

rsync is deliberately not used: it is not installed on the host and there is no sudo to add it. Needs an HF token with access to the gated model repo. Then scripts/hinode_run.sh, or scripts/host_matrix.py for the full matrix.

Do not set HF_HUB_OFFLINE=1. Construction loads from the local snapshot directory, so nothing is fetched at load time β€” and stage 2 needs outbound network for the NATTEN kernel. Forcing offline breaks the second without helping the first.

Locally, with no GPU

The whole test suite runs with no GPU, no weights and no network. That is a deliberate property, not a coincidence, and it is worth preserving: it is what makes it possible to catch a defect before it costs a run.

python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/python -m pytest -q

torch, diffusers and torchaudio are not needed for the suite. av and numpy are, because they are how real containers get written and read.

On a Space

βœ… This works. The repo is the Space β€” README.md's YAML front matter is the Space config, and the space remote points at it:

git remote -v | grep space
# space  https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio

Deploying is a push to that remote's main. β›” This team does not push β€” not to origin, not to space. The orchestrator does it; that is a deliberate boundary, not a missing capability.

What the deploy depends on, all of it already load-bearing:

app_file: app.py, sdk: gradio 6.24.0, python_version: "3.12" πŸ“„ front matter
suggested_hardware: zero-a10g πŸ“„ a legacy id string β€” not an A10G. Changing it moves the Space onto paid dedicated hardware
startup_duration_timeout: 1h πŸ“„ required. The default is 30 min and this Space downloads 72.19 GB at build time. When it blows, the failure looks like something else entirely
an HF token secret with access to the gated model repo πŸ“„ weights fetch at runtime, because preload_from_hub 401s on a private Space

tests/test_deploy_drift.py asserts these rather than trusting them β€” run the suite before any deploy and the drift shows up locally instead of as a 40-minute build that dies at the end.

Reading the startup log is the cheapest diagnostic available, and it costs zero GPU quota because packing and loading happen at module scope, before the first @spaces.GPU call. The load line names its own configuration:

quantised transformer, text_encoder as int8 | peak RSS 0.91 -> 87.13 GiB

⚠️ That peak is the number to watch. It is a peak across the load, not a steady state β€” we have never measured steady state, and the RAM questions are about steady state. See SPACE_GENERATION_FINDING.md Β§7.


6. The lessons worth carrying

Not a diary. These are the ones that would change how the next person works.

A fake derived from the mapping can never catch a wrong mapping

The audio VAE's six parameters were read from audio_vae.config by names that do not exist there πŸ“„ β€” four live under different keys, two are instance attributes absent from config.json entirely. Every runtime path through that module raised on real weights, and every offline test passed, because the fake used our names. It agreed with the code and with nothing else.

The instinctive repair β€” build the fake from the same mapping the code uses β€” is a trap for exactly the same reason. The fix is a recorded fact about reality: the real config's key set, checked into the tests with its provenance. The fake checks semantics; the recorded keys check the world.

The fixture that makes a test possible is itself untested by that test

Three instances in one day, all in tests that were otherwise excellent:

  1. test_encoder_contract.py drove the real encoder through real containers to a real file β€” and built its audio as np.float32. The model produces bfloat16. Six of eleven modes were broken at the last step and 1926 tests passed βœ… both counts measured β€” the modes on hinode, the tests in this repo at the commit before the fix.
  2. The replacement fake's .float() returned an ndarray where torch returns a tensor. Written the convenient way, it would have passed a broken fix.
  3. The audio-VAE fake used our own key names β€” the case above.

The mitigation that worked: check the fake against the real thing. In (2) that check failed within a minute of existing.

A search that cannot see the claim reports that the claim is gone

Three defects in this build share one shape: the instrument returned a confident negative it was structurally incapable of turning positive.

  • A wrapped line. Β§7 said "The Space has evicted twice and has never run" as one line. Β§1 said the same thing split across a line break, so a grep for the phrase found the occurrences that fit on one line and pronounced the audit clean.
  • A shell modifier. "$c:ltx25_studio/..." is parsed by zsh as the :l lowercase modifier, so git show got a mangled path and every commit answered "constant absent" β€” including the one that demonstrably contains it.
  • A stale marker. A ⚠️ (inherited) pointing at a section that had been rewritten to say FALSIFIED still scanned as merely unverified.

None of the three raised an error. Each returned a clean, plausible, wrong answer, and two of them nearly closed an audit.

The defence is not a better search string. It is running the instrument against a case whose answer you already know: if the check cannot go positive on a known positive, its negatives carry no information. The zsh case was caught only because "absent at HEAD too" was impossible on its face.

A claim moving known β†’ unknown is nearly always a regression

Documents accrete facts and rarely lose them. When an edit is about to record something as uncertain, search first for it being known β€” the likeliest explanation is that someone established it and the current author has not read it. This has since caught its own inverse: a correction issued against this document's history that the history did not actually contain.

Checks over comments

Twice, a constraint written down did not survive the next function written by its own author β€” a dtype requirement violated two commits after the ⚠️ comment demanding it, and a 44.33 GB refetch reintroduced while reading the docstring that predicts it. A comment is a message to a future stranger; it has no purchase on the author five minutes later. Make the wrong thing unsayable instead: derive the value once and share it, so the two callers cannot drift.

The instruments encode the failure you imagined, not the failures available

host-runner's generalisation, in their words:

all three instruments encoded the failure I imagined rather than the failures available

β€” a pkill -f pattern that matched the killer, a wait loop with no failure exit, and a fixture guard that handled a fixture being absent but not one that raises. The three had different authors, including the lead. That is the point of recording it: it is not one person's blind spot, it is what happens whenever the person who imagines the failure is the person who writes the check for it.

We can prove a function ran. We cannot prove it ran with the right arguments.

The gap this build ends on, and the one worth closing next.

DISTILLED_SIGMA_VALUES was defined, documented as mandatory, quoted in its module's docstring, and referenced β€” so every check we own passed. It was referenced from stage_1_kwargs(), which StageRunner applies only when stage 2 is enabled. Eleven of twelve host runs were generated with upstream's linear fallback and its SFT guidance defaults instead:

sent:      [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125]
intended:  [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]

A reference check passes. A reachability check at module level passes. Even the function-level sweep passes, unless someone thought to name that particular function. What nothing checked was the arguments the pipeline was actually called with.

The pattern that does work, used three times here, is to record the vendor's specification independently and assert against that β€” MODEL_CARD_DISTILLED, REAL_AUDIO_VAE_CONFIG_KEYS, REAL_VOCODER_CONFIG_KEYS. Each is a fact copied from the source of truth rather than derived from our own constants, so changing ours cannot quietly redefine what correct means. Without that independence the test agrees with the code and with nothing else.

The escalation ladder, in the order this build climbed it: is the module imported β†’ is it reached by the path that does the work β†’ did this specific function run β†’ were the arguments the ones the vendor specifies. Only the last rung is still hand-built, one call site at a time.

A module can be reached and still contain a dead function

The reachability sweep that exists to catch importer-count-zero missed one, for two independent reasons: the tracer recorded which file ran and discarded which function β€” the name was in the same frame object β€” and the probe drove one mode, so a path reachable from no other mode was invisible by construction. Either fix alone would still have missed it. "Sharpen the instrument" was the obvious response and would not have worked.

The one time source-reading and measurement confirmed each other

Every other pairing today went the other way: one corrected the other, and the lesson was about which to trust when they disagreed. Once, they agreed.

A reproducibility control failed β€” two runs at one seed, 8.7% apart β€” and the obvious reading was an unseeded draw somewhere. Instead of hunting, every noise source in the pinned pipeline was enumerated: _create_noised_state, the conditioning mask, prepare_*_latents, the decode timestep, the condition VAE encode. All threaded the generator; the encode used sample_mode="argmax" and was deterministic regardless. Source said it should already be reproducible.

Rather than write either sentence β€” "the seed is attached to nothing" or "we are seed-consistent but not bit-reproducible" β€” a scoping run was requested. Before it ran, the control was re-checked: it had been measured against a tree that predated the seeding fix. At the tip, two runs are byte-identical.

The true claim was stronger than either draft: on audio_to_audio, seed determines the output byte-for-byte, on GPU, with no deterministic-algorithm flags. Both drafted sentences were weaker and wrong, and both would have shipped if the reluctance to write an unmeasured claim had not held for one more exchange.

And then the corrected sentence was written without its scope, twice more in this document β€” as a property of the build rather than of one mode. ⏸️ The video path is still unverified. Recording that here because the lesson is not "be careful": it is that a claim narrowed in one place does not narrow itself in the others, and a document is exactly where that goes wrong. Β§4 gap 8 carries the scope; Β§2.5 and this paragraph each had to be found and fixed separately.

Refusing to write a sentence you cannot yet support is the same discipline as refusing to change a threshold to make a test green β€” and it is the cheaper of the two, because nobody has to notice you did it.

Staffing β€” two makers and a lead who verifies

Not five, and not one. This build ran briefly with five teammates and settled at two: one implementer owning the code, one runner owning the hardware. The three stood down were not underused β€” they were a coordination cost with no matching throughput, and the collision they produced (two agents assigned outputs.py; a cross-boundary change landing under a commit message about other files) cost more than their output was worth. Two makers with disjoint, stated file ownership is the shape that worked, and the line held all day under pressure: the runner found four product defects and edited none of them.

The scarce resource was execution on real hardware, not review capacity. At the point the suite reported 1926 passing βœ…, six of eleven modes were broken, the audio-input path was connected to nothing, stage 2 was unreachable, and every runtime path through audio.py raised on real weights. None of that was found by reviewing harder β€” it was found by an afternoon on a real GPU. Another reviewer would have added agreement, not information. Budget for one more agent is better spent on a second executor with different hardware than on a second reviewer.

A lead who cannot edit is forced to verify by observation rather than by reading their own diff, and that constraint turned out to be a feature. Every claim the lead checked was checked by running something β€” git merge-base --is-ancestor rather than trusting a commit message, av on the container rather than trusting a verifier's exit code, pgrep on the box rather than trusting a status report. That caught the box silently trailing the tip three times, a stale bypass premise, and a wait loop that had frozen a teammate. A lead who could edit would have been tempted to fix those, and would have stopped looking.

The default-REJECT verifier role was filled by reality rather than by an agent. The plan called for an independent verifier holding a default-REJECT stance; in practice the hardware held it. It rejected the from_pipe cast, the bf16 encode, the missing audio parameters and the unreachable stage 2 without being asked, and without any possibility of being talked round. Where a real gate exists, prefer it to a reviewer agent β€” an agent can be persuaded and a RuntimeError cannot.

What it cost. Message-crossing: three times a blocking item was buried inside a long message and lost a cycle behind work already in flight. The fix is mechanical and worth adopting from the start β€” one blocker, one message, nothing else in it. Long messages are for context; short ones are for priority.


7. If you do one thing

Re-run the comparisons seeded. Everything works; the open item is that some of the evidence was gathered before the seed reached the model.

The runs that need repeating, and what each would then establish

run what it would establish why it does not yet
Β§2.5 before/after pairs that the recipe fix caused the pixel differences both halves were unconditionally random
H10b β€” noise_scale 0.1 vs 0.9 that the user's audio reaches the model same, plus the audio encode drew unseeded
H12b β€” LoRA effect that attaching the adapter changed the output same
H4b β€” strength 0.3 vs 0.9 that the conditioning clip reaches the model same

Add a same-value control to each β€” two runs at identical settings. Under a working seed its spread should be zero, and that is what turns "the halves differ" into evidence rather than a coin toss. An earlier control measured 1.8% and was read as a bound on the variance; an unseeded control cannot bound anything, which is the mistake worth not repeating.

β›” The first row of that table is dead (annotated 2026-08-25), and this file already said so elsewhere. The Β§2.5 before/after pairs cannot be re-run: the seeded gallery is dropped with its host, and Β§2.5/Β§4 gap 8 record that as permanent β€” the table is unattributable forever, not pending. The live content of this instruction is the other three rows: H10b, H12b and H4b are still runnable β€” on the Space (none needs stage 2; each is one same-seed pair at 256Γ—256Γ—9-class cost) or on a future host β€” and each still establishes exactly what the table says.

Two of the three were delivered on the Space later on 2026-08-25: H12b's claim (the adapter changed the output β€” 20a3a84b… β‰  b78803cc… at the same seed) and H4b's question (the conditioning clip reaches the model β€” answered by clip-swap: outputs 17.277 vs 0.864, mirroring their inputs, against a byte-identical control). H10b is the last discriminator standing. SPACE_SESSION_2026-08-25.md Β§6–§7.

β›” BOTH DELIVERIES ARE WITHDRAWN (2026-08-25, evening). Every run cited there used the broken int8 build. H12b's two hashes differ because one garbage render differs from another; H4b's 17.277-vs-0.864 compared the motion energy of two colour statics. The mechanical readings stand β€” the adapter injection path ran, the conditioning path carries a scalar, the noise floor at a fixed seed is exactly zero β€” and all three discriminators are open again: H10b, H12b and H4b. They are also worth spending on now, because there is finally a build whose output is a picture (e3b5333).

What is already earned, and does not need repeating

Eleven of eleven modes produce artifacts with the expected stream layout on the real path. βœ… H14's exact 2Γ— resolution, H15's 97 frames, H17's round-trip recovering 4/4 tones under a strict gate, both LoRA directions, and every audio level β€” all structural or systematic, and none of them depends on seeding.

And read Β§3 before treating any of this as a deployment. βœ… The Space now runs and generates β€” ⏸️ for one of the eleven modes. Everything in this section is a hinode result; the Space has proven text_to_video at 256Γ—256Γ—9 and nothing else. Those are different claims about different machines, and Β§3 is where they are kept apart. β€” β›” Corrected (2026-08-25, evening): the Space had proven that text_to_video executes; its output was colour static. This paragraph's instinct β€” keep Β§2's hinode results apart from anything the Space says β€” was the correct one, and every later annotation that softened it was wrong. Everything in this section is hinode, bf16, unquantised, and unaffected.

β›” And that supersession is itself corrected (2026-08-25, evening): "content-verified" meant non-empty and non-frozen, on the broken int8 build. All eleven modes execute on the Space; none of their output was ever verified as an image. Banner at the top of this file.

βœ… Superseded (annotated 2026-08-25): the Space has since proven all eleven modes at 256Γ—256Γ—9 β€” content-verified, sha 287ba9c, 2026-08-22 (SPACE_MODE_MATRIX.md). What it has still proven nothing about: stage 2, LoRA, auto_duration, and any resolution anyone would ship. Β§3's annotation carries the merged state.

And later on 2026-08-25 that residue shrank to auto_duration alone (plus the stage-2 ceiling's exact position): stage 2, LoRA and 1216Γ—704Γ—121 all ran on the Space that day. Top-of-file note; SPACE_SESSION_2026-08-25.md.