Spaces:
Running on Zero
Download docs/OPERATOR_HANDOFF.md from techfreakworm/LTX2.5-Studio: direct link, hf CLI and curl.
- Browser
- Download file 68.9 kB
-
https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio/resolve/main/docs/OPERATOR_HANDOFF.md
- Command line
-
hf download hf://spaces/techfreakworm/LTX2.5-Studio/docs/OPERATOR_HANDOFF.md
-
curl -L -o OPERATOR_HANDOFF.md https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio/resolve/main/docs/OPERATOR_HANDOFF.md
A newer version of the Gradio SDK is available: 6.32.0
Operator handoff β LTX-2.5 Studio
Status on 2026-11-16
This handoff is the August 2026 record. Its body is kept as written; it no longer says where the project stands. Superseded since:
- Stage 2's "ceiling between 25 and 49 frames" (the two blocks below): the 49-frame failure was the 47.375 GiB
largepartition. 704Γ480Γ49 passes onxlarge(55.906 GiB reserved,MIG_ALLOCATOR_SIGHTINGS.mdΒ§2). The ceiling onxlargeand stage 2's picture quality are not measured.- "Total day β€ ~675.5 GPU-s": the quota ledger for UTC 2026-08-25 reads β€ 1528.672 GPU-s over 20 runner calls. Adding the β€ ~93.3 GPU-s of manual calls bounded by hand in
SPACE_SESSION_2026-08-25.mdΒ§8 puts the day's bound at ~1622 GPU-s (wall Γ 2 upper bounds, not billed seconds), about 84 % of the 1920 stop line. The evening's manual calls and all of 2026-08-26/27 were never recorded.- "The post-fix eleven-mode matrix goes in
SPACE_MODE_MATRIX.mdΒ§10": it ran that evening but was never written up, and its video and audio files are gone.- Β§3 "82.2 GB of 150 [OK]": read before ZeroGPU packs (the packed term was 0.0). On the 2026-11-13 boot the platform's own progress bar read
40.6Gpacked; the disk peak has never been measured.- Β§5 "On a Space": "deploying is a push to that remote's
main" is wrong as stated. The Space repo has its own root commit, so pushing a work commit is rejected as non-fast-forward; every deploy so far was a merge commit on the live Space head (git commit-tree <work>^{tree} -p <work> -p <Space head>) pushed as a fast-forward. Never--force. The 72.19 GB download happened at startup, not at build time: the 2026-11-13 cold start took 13 min 59 s, about 11.5 min of it the download.tests/test_deploy_drift.pydoes not assertstartup_duration_timeout.- Where the weights come from (Β§5; Β§3's "private-Space fetch path"; the deploy table's
startup_duration_timeoutand token rows): the Space is public now, and the code has the image bring the weights.preload_from_hubin the README's front matter fetches the 35 model files the int8 Space loads (71.92 GB; the temporal upsampler is no longer fetched) and the prompt enhancer's 8 files (2.49 GB) before the app starts.preflight.ensure_weights()then checks the cache, downloads only what is missing, and refuses before writing a byte a download the 150 GB disk could not hold. How long a cold start takes this way: deploy 2's check on the Space, on issue #32.- Β§5 "On hinode": hinode is retired.
scripts/sync_to_hinode.sh,scripts/hinode_run.shand everyscripts/host_*.pywere removed on 2026-11-16 (git show 2f356b5:scripts/<name>restores one).autoas the GPU size:LTX25_ZEROGPU_SIZEnow defaults toxlargein code.- "local MPS bf16" sign-off: those artifacts came from tooling outside this repo, which had no Apple-silicon path. One was added on 2026-11-16 and verified on real weights (see
README.md).Where the project stands:
../README.md(what the app does, how to run it, what it costs),SPACE_MODE_MATRIX.mdΒ§10 (what has been verified, and how) andRELEASE_NOTES.md(what each deploy contains). Deploy 1 (Space commit0cd54f2, the tree of1e9fd20, 2026-11-16) ran all eleven modes on the Space, each looked at (SPACE_MODE_MATRIX.mdΒ§10.1); deploy 2 (2026-11-17) replaced it, and its own check on the Space is reported on issue #32.
What you have, what you do not have, and what to do next. Written for someone who was not here.
PLAN.mdrecords how things were decided;TEST_PLAN.mdrecords what was run; this tells you where the project actually stands.
Every load-bearing number below is marked with where it came from: β measured on hinode Β· π read off a pinned source or config Β· β οΈ inferred Β· βΈοΈ not yet established. If a number carries no mark, treat that as a defect in this document.
β READ FIRST β 2026-08-25, evening: the deployed Space has NEVER produced a real image
Every generation result this document attributes to the Space was produced by a broken transformer. Every Space sha up to and including
ff142d9ran blanket bitsandbytes int8 with nollm_int8_skip_modules, so bnb quantised the LTX-2.5 conditioning path β the eight top-levelLTX2AdaLayerNormSinglemodulation heads and the in/out projections β along with everything else. The denoiser stopped being steered by the timestep and the prompt, and every Space artifact ever banked is colour static or a flat field: correctly shaped, high variance, fully deterministic, and not a picture.β Fixed at
eb1aa9f(316 of 1660 Linears spared, names read off the real checkpoint), deployed as Space shae3b5333, and proven there β the same seed, prompt and geometry that produced colour static now produce a photographic frame. Before/after:~/.soma/leads/ltx25-studio/frames/COMPARE_before_after_skipfix.pngπ.β MECHANICAL claims survive, all of them: all eleven modes execute on the Space; both fork-boundary directions carry their cargo; determinism and byte-reproducibility hold; tiled decode enables 1216Γ704Γ121; stage 2 runs and doubles geometry with a ceiling between 25 and 49 frames; LoRA applies, detaches, and both routes agree; the disk, RAM and quota arithmetic.
β OUTPUT claims do not survive, none of them. Every "content-verified" Space artifact in this file was verified non-empty and non-frozen, never as an image. Specifically: the conditioning-fidelity conclusion (H4b), the LoRA inertness check (H12b) and the stage-2 pass were all measured on garbage β they show the code paths ran, not that the output was correct.
β
b78803ccis byte-reproducible NOISE. The determinism claim is intact and its artifact is worthless.β Β§2 is UNAFFECTED, and this is the load-bearing distinction in this document. hinode ran bf16, unquantised (Β§2.1's
base.dtyperow records it), and the operator's only visual sign-off was on local MPS bf16 artifacts served over tailscale. Every bf16 path anyone has eyeballed looked right; the only path ever eyeballed on the Space was broken. Β§3's discipline was correct all along, and the annotations that merged the two evidence bases were wrong: they merged across a dtype boundary no run ever crossed.β The method failure, as policy. Statistical gates cannot distinguish a render from mush. They are a pre-filter, never a verdict β they demote artifacts, they never promote one. No artifact is a PASS until a human or model eye has seen a frame at native resolution, and every report claiming an output is good carries the frame. This binds the structure detector added at
e54477ctoo: it FAILS the flat grey (SR 0.5298) and PASSES colour static (SR 0.9183, against 0.9961β0.9968 for the real photograph), so even it cannot promote.β The cheap-config lesson. The 256Γ256Γ9 config used for most of the campaign produced output too small and too blobby for anyone to judge by eye. That is how three weeks of confident nonsense got manufactured. A config too small to look at is a config that cannot be verified. Β§2.2's own "Structure is not content" and Β§4.1's
verify_streams-exit-0-on-a-silent-file had both already written down why β and "content-verified" was allowed to mean non-empty bytes anyway.Full record: the invalidation blocks at the top of
SPACE_MODE_MATRIX.mdandSPACE_SESSION_2026-08-25.md. The post-fix eleven-mode matrix goes inSPACE_MODE_MATRIX.mdΒ§10.
β State as of 2026-08-25 β read
SPACE_SESSION_2026-08-25.mdA full GPU campaign ran on the Space that day and settled most of what this document lists as open: byte-level reproducibility sha256-VERIFIED (three containers, three code shas), 1216Γ704Γ121 generates (tiled decode,
cdb3211), stage 2 ran on the Space (PASS at 704Γ480Γ25 input β 1408Γ960; FAIL at Γ49 β its ceiling is between those), both LoRA directions proven with a passing inertness check (H12b's claim, delivered), conditioning shown to drive the output (H4b's question, answered by clip-swap), and the fork-boundary error fix proven live in both classes. Total day β€ ~675.5 GPU-s of 2400.β Corrected the same evening β see the block above. Every run in that campaign used the broken int8 build. The mechanical results stand exactly as written (reproducibility, geometry, ceilings, error paths, cost). The two OUTPUT claims in this paragraph do not: "both LoRA directions proven with a passing inertness check (H12b's claim, delivered)" β the adapter changed one garbage render into a different one, and H12b is NOT delivered β and "conditioning shown to drive the output (H4b's question, answered by clip-swap)" β both clip-swap outputs were colour static, and H4b is OPEN again. H10b is not the last discriminator standing; all three are.
What genuinely remains open after that session: the stage-2 ceiling (diffusion-decoder chunking β its own investigation), the
video_to_audioβ19.5 dBFS observation (now consistent-with the conditioning finding, still not directly probed β β that "consistent-with" is withdrawn: it derived from the conditioning finding, which is withdrawn; and β19.5 dBFS was audio generated by a broken transformer), the 2026-08-20 SIGKILL history (documented-unexplained, by operator ruling) β and, still never exercised on the platform:auto_durationand the H10b audio-input discriminator.
1. What this is
A Gradio studio for LTX-2.5 (Lightricks' 22B video+audio model) covering
all eleven generation modes, with user-uploadable LoRA validated at
inference time. One model load serves every mode. Python 3.12, diffusers
pinned to an exact SHA, weights pinned to an exact revision.
Destination is a private HuggingFace ZeroGPU Space. β
It runs and generates
there β startup, one clean GPU worker, and a content-verified video, at Space
sha 8fb292d6.
β Corrected (2026-08-25, evening): "a content-verified video" was a non-empty, non-frozen file. It was colour static β
8fb292d6ran the blanket int8 that quantised the conditioning path. What stands: the Space runs, startup completes, a GPU worker forks cleanly, and a well-formed artifact comes back. What does not: that anything it returned was a video in any sense a user means. Fixed ateb1aa9f, deployede3b5333. Banner at the top of this file.
βΈοΈ That is one mode, once. Ten of the eleven have never executed on that platform; Β§2's eleven-of-eleven is a hinode result and stays one. Β§3 is the section that keeps those two evidence bases apart β read it before treating either as the other.
β The annotation below is itself corrected (2026-08-25, evening): "content-verified" meant non-empty and non-frozen, and sha
287ba9cran the broken int8. All eleven modes execute on the Space β that half stands and is the useful half. Nothing about their output does. Banner at the top.β SUPERSEDED (run 2026-08-22, annotated 2026-08-25): all eleven modes now generate on the live Space, content-verified, at sha
287ba9cβSPACE_MODE_MATRIX.mdis the record. "One mode, once" was true when written and is no longer the state: eleven-of-eleven is now also a Space claim, βΈοΈ at one configuration only (256Γ256Γ9, stage 2 off, no LoRA, noauto_duration). Stage 2, LoRA, and every shippable resolution remain hinode-only. Β§3's separation of the two evidence bases is what made the merge legible β its annotation carries the detail.Narrowed further on 2026-08-25: stage 2 (to its 25β49-frame ceiling), LoRA (both directions) and 1216Γ704Γ121 have now run on the Space too β see the top-of-file note. Still hinode-only: the duration head.
2. HOST-PROVEN
Everything in this section ran on hinode: an unpartitioned RTX PRO 6000
with 124 GB RAM β
β through get_pipeline exactly as app.py calls it, not
through a test harness.
2.1 Startup gates
| Gate | Result | |
|---|---|---|
| Weights present | 37 / 37 files, 72.185 GB | β |
| Full repo, for contrast | 56 files, 164,192,096,189 B (164.19 GB) | π manifest |
| Components shared between the two pipelines | 9 / 9 identical by is |
β |
base.dtype after deriving |
torch.bfloat16, unchanged |
β |
| CUDA cost of the second pipeline | β1024 bytes | β |
| Total for both pipelines | 65.370 GiB | β |
The β1024 bytes is the number to re-check after any change to pipeline construction: it is what proves the second workflow shares the first's weights rather than loading its own copy. A positive figure in the tens of GB means the sharing has broken.
2.2 Modes
Baseline is 704 Γ 480 Γ 121 frames, seed pinned to 1234. "Streams" is
what tests/verify_streams.py observed in the container, not a human reading.
| # | Mode | Result | Streams observed | Size | Time |
|---|---|---|---|---|---|
| H1 | text_to_video |
β | h264 704Γ480@24, f=121 | 791,782 B | 8.4 s |
| H2 | image_to_video |
β | h264 704Γ480@24, f=121 | 307,583 B | 7.8 s |
| H3 | image_text_to_video |
β | h264 704Γ480@24, f=121 | 775,213 B | 8.0 s |
| H4 | video_to_video |
β | h264 704Γ480@24, f=121 | 708,885 B | 8.6 s |
| H5 | text_to_audio_video |
β | h264 + aac 48000 Hz 2 ch | 792,803 B | 7.8 s |
| H6 | image_to_audio_video |
β | h264 + aac 48000 Hz 2 ch | 374,184 B | 7.9 s |
| H7 | image_text_to_audio_video |
β | h264 + aac 48000 Hz 2 ch | 995,843 B | 8.2 s |
| H8 | text_to_audio |
β | pcm_s16le 48000 Hz 2 ch, no video | 961,998 B | 1.8 s |
| H9 | video_to_audio |
β | pcm_s16le 48000 Hz 2 ch, no video | 961,998 B | 2.0 s |
| H10 | audio_to_audio |
β | pcm_s16le 48000 Hz 2 ch, β10.4 dBFS | 961,998 B | 2.8 s |
| H11 | audio_to_video |
β | h264 704Γ480@24, f=121, no audio | 193,825 B | 8.0 s |
Eleven of eleven. All β
measured on the real get_pipeline path, no bypass
and no harness accommodation; verify_streams returned 0 on every one.
| beyond the baseline | ||
|---|---|---|
| H14 stage 2 | 1408 Γ 960, f=121 | 1,807,646 B Β· 35.2 s |
H15 duration_head |
704 Γ 480, f=97 not 121 | 593,527 B Β· 5.9 s |
| H16 high resolution | 1216 Γ 704 + aac stereo | 1,775,743 B Β· 21.0 s |
| H17 mel round-trip | 4.000 s in β 4.010 s out, 4/4 tones | strict gate |
| H12 / H13a / H13b | LoRA accepted / rejected / rejected | stage 6.5 |
Wall-clock excludes the model load, which is 14.0 s once per batch β .
β οΈ Structure is not content
Every run here passed
verify_streams. The one defect that mattered most was invisible to it:video_to_audiocame back 50 dB below its siblings with correct codec, channels, rate, duration and a non-empty file. Silence and a healthy signal have identical stream layout.It was found because someone measured RMS on an artifact that had already been signed off, and it led to a defect affecting all twelve modes (gap 7). A green table says these are well-formed files. It is a weaker claim than "correct output", and the gap between the two is where this build spent its afternoon.
H8 and H9 are byte-identical in size because uncompressed PCM of equal duration always is. That is why size proves nothing about content, and why it took an RMS measurement to notice.
On audio levels generally. Measured levels sort by condition type, not by mode β β text-only β5.2 to β6.2 dBFS, image-conditioned β8.2 to β15.2, video-conditioned at strength 0.8 β54.5 to β55.0. The text-only group peaks at exactly 1.0000 with no samples exceeding, so the export is faithful and the saturation is upstream, in the model, not in our writer. Those files are consistent with each other and sit at the rail; βΈοΈ whether that is correct for this model has not been established.
2.5 What the recipe fix bought β before / after
Same seed, same geometry on both sides of every pair; only the distilled recipe (gap 7) differs.
| # | mode | before | after | wall-clock |
|---|---|---|---|---|
| H1 | text_to_video |
886,265 | 791,782 | 25.7 β 8.4 s |
| H2 | image_to_video |
432,486 | 307,583 | 26.3 β 7.8 s |
| H3 | image_text_to_video |
1,130,178 | 775,213 | 26.6 β 8.0 s |
| H4 | video_to_video |
727,490 | 708,885 | 27.0 β 8.6 s |
| H5 | text_to_audio_video |
1,091,468 | 792,803 | 25.0 β 7.8 s |
| H6 | image_to_audio_video |
620,537 | 374,184 | 26.6 β 7.9 s |
| H7 | image_text_to_audio_video |
784,282 | 995,843 | 26.7 β 8.2 s |
| H8 | text_to_audio |
961,998 | 961,998 | 5.6 β 1.8 s |
| H9 | video_to_audio |
961,998 | 961,998 | 6.0 β 2.0 s |
| H14 | stage 2 | 2,254,799 | 1,807,646 | 84.3 β 35.2 s |
| H15 | duration_head |
684,374 | 593,527 | 19.4 β 5.9 s |
| H16 | 1216 Γ 704 | 2,331,528 | 1,775,743 | 68.7 β 21.0 s |
Geometry is identical on both sides of every pair β H14 stays 1408 Γ 960, H15 stays 97 frames, the rest 704 Γ 480 Γ 121. Sizes changed; geometry did not is the shape of the result.
Three things a reader needs:
- H7 is the only artifact that grew. h264 size tracks content complexity, so that is neither good nor bad on its own. Flagged, not interpreted.
- H8 and H9 are byte-identical because uncompressed PCM at one duration, rate and channel count always is. Their change is entirely in level, not in bytes β see below β so reading this table alone would call them unchanged when they are two of the three that moved most.
- ~3Γ faster everywhere. At
guidance_scale=1.0with STG and modality zeroed there are no classifier-free branches, so it is one forward pass per step instead of two or three. This is the one column here that seeding does not confound β wall-clock is not drawn from a distribution β and it follows from the mechanism rather than merely accompanying it.
β οΈ These pairs are not yet controlled β the audio evidence is
The seed never reached the model until
4ebb8b9(gap 8), so both halves of every pair above were unconditionally random. The pixel differences cannot evidence that the recipe fix caused them β two runs of one mode differed anyway. β Seeding is fixed, and verified byte-identical foraudio_to_audio.β The seeded gallery that was to supersede this table is DROPPED. It lived on a host that no longer exists, and nothing replaced it. So this table is permanently unattributable, not temporarily so β these figures remain as the record of what the defect cost, and that is now all they will ever be.
The audio levels do stand on their own, because they are systematic and orders of magnitude outside run-to-run variation:
before after H9 video_to_audioβ55.0 dBFS β17.8 dBFS H8 text_to_audioβ5.2 dBFS, peak 1.0000 (railed) β15.3 dBFS, peak 0.6953 highest peak, any mode 1.7811 1.0050 A 50 dB gap and a clipped peak going away are not a seed.
β And the replacement will not arrive. Bitwise reproducibility is verified for
audio_to_audio. The video path has since been evidenced on the Space β a run returned bytes identical to one banked the previous day on a different container at the same seed β but βΈοΈ evidenced is not proven: no two-run frame-hash comparison exists. Seedocs/SPACE_MODE_MATRIX.mdΒ§5. The gallery that was to close this is dropped with the host it lived on.β Correction (2026-08-25) on "returned bytes identical": that overstates what was measured. The cross-day match was file size (39,146 B) plus every decoded statistic to three decimals β no hash of the day-1 artifact was ever recorded, and that file is gone, so byte identity against it is permanently unverifiable. The "evidenced is not proven" hedge above was right, and even understated: no byte comparison existed either. The planned close is a same-seed re-run sha256-compared against the surviving 2026-08-22 artifact β
SPACE_MODE_MATRIX.mdΒ§5's correction carries the hash.
β Sizes differed between the two code paths β almost certainly just the seed
Three modes were run through a bypass (
_base_pipelineplus an injectedStageRunner) and through the realget_pipelinepath, all pre-fix:
bypass real path H1 865,447 B 886,265 B H2 577,051 B 432,486 B β 25% smaller H3 1,040,007 B 1,130,178 B This was recorded as an open question β "GPU non-determinism, or a genuine difference between the routes?" β and described as "at the same seed."
It was not at the same seed. The seed reached nothing (gap 8). These are six unconditionally random runs, and two runs of one mode differed by this much anyway. There was never an anomaly to explain, and the phrase "at the same seed" was doing all the work: it made an ordinary difference look like a finding.
βΈοΈ Cheap to settle properly now β run one mode both ways at a fixed seed. Nobody has, so it is probably the seed rather than certainly.
The advice it produced stands on its own and is worth more than the question: assert stream layout, never an exact size.
2.3 Beyond the baseline
| Run | What it proves | Result |
|---|---|---|
| H16 high resolution | usability, not just correctness | β
text_to_audio_video at 1216 Γ 704 @24, f=121 + aac 48000 Hz 2 ch β 1,775,743 B, 21.0 s |
H15 duration_head |
the model chose the length, rather than a default leaking through | β f=97, not 121 β 704 Γ 480, 593,527 B, 5.9 s |
| H14 stage 2 | upsample β 3-step schedule β diffusion decode | β 1408 Γ 960, f=121 β 1,807,646 B, 35.2 s |
Post-fix figures, matching Β§2.2. Β§2.5 carries the pre-fix column.
H15 is worth understanding: num_frames is omitted and the duration head
picks. Had it returned 121 β the default β the run would have looked identical
to a pass while proving the opposite. 97 is the whole result.
H14's proof is its resolution, not a comparison. β 1408 Γ 960 against H1's 704 Γ 480 is exactly 2Γ in both dimensions β a size only the spatial upsampler can produce, and the one claim here that no vintage or seed can affect, since geometry is identical across every before/after pair. That is stronger evidence than the same-seed diff the matrix planned, because a same-seed comparison shows a difference while this shows the specific difference stage 2 exists to make.
2.4 The LoRA validator, both directions
| Run | Adapter | Result |
|---|---|---|
| H12 | joyfox/LTX-2.3-Transition-LORA |
β ACCEPTED, 576 modules across transformer + connectors |
| H13a | charlied98/β¦ lora_weights_step_03000 |
β
REJECTED at stage 6.5, LoRADimensionError |
| H13b | same repo, comfy_lora_weights_step_03000 |
β REJECTED at stage 6.5, same mechanism |
H12 carries mixed dimensions β 4096 at attention, 2048 at audio_ff, 16384
at ff.net.2 β
β and none of that is a mismatch. An earlier "expect hidden dim
4096" rule would have rejected a perfectly good adapter; the false-positive
guard is what stops that. H13a and H13b differ only in key prefix
(transformer. vs diffusion_model.) and reach the same verdict by the same
mechanism, which is what makes the rejection trustworthy rather than lucky.
3. SPACE-PROVEN vs HOST-PROVEN β read this before believing Β§2
β The Space runs and generates. βΈοΈ That is one mode, once.
text_to_videoat 256Γ256Γ9 produced a real, content-verified video on the live private Space β . Ten of the eleven modes have never executed there.β Corrected (2026-08-25, evening): it was not a real video. It was colour static β the origin claim of the entire Space evidence base, and it was wrong from the first run. "Content-verified" meant non-empty and non-frozen. What survives: the Space runs, and one mode executed there. Banner at the top.
A green table on a dedicated host and a working deployment are different claims. Both have now been made β about different things. That is exactly why this section still exists: the two evidence bases have diverged rather than merged, and Β§2's eleven-of-eleven remains a hinode claim.
β SUPERSEDED (run 2026-08-22, annotated 2026-08-25). The mode matrix ran on the live Space and all eleven modes generate there, content-verified, at sha
287ba9cβSPACE_MODE_MATRIX.md. The two evidence bases have merged at exactly one configuration: 256Γ256Γ9, stage 2 off, no LoRA, noauto_duration.β THE MERGE NEVER HAPPENED (corrected 2026-08-25, evening). hinode ran bf16, unquantised β Β§2.1's
base.dtyperow records exactly that β and the Space ran blanket int8. The two evidence bases were separated by a dtype boundary no run ever crossed, and that difference was the whole defect. This section's original discipline was right and this annotation was the error: it merged on "the same mode name at the same geometry" when the thing that differed was the quantisation of the conditioning path. Keep Β§3. Banner at the top. What is still hinode-only: stage 2 (1408Γ960), the duration head, LoRA in both directions, the mel round-trip at full length, and every shippable resolution. This section's discipline is unchanged β it now keeps configurations apart instead of platforms.That hinode-only list shrank again on 2026-08-25: stage 2 reached 1408Γ960 on the Space (from β€25-frame input; its decoder ceiling is between 25 and 49 frames), LoRA ran in both directions, and 1216Γ704Γ121 passed via the tiled decode. Remaining hinode-only: the duration head, and full-length audio work.
SPACE_SESSION_2026-08-25.md.
Detailed record: SPACE_GENERATION_FINDING.md
β the first deploy, the killed workers, the falsified ceiling, and the run that
generated. Not restated here on purpose.
hinode is an unpartitioned card with 124 GB RAM and an ordinary filesystem. ZeroGPU is none of those things. The first deploy answered several of these questions and changed the answer to one of them:
β Tensor packing β exercised. Packing ran and the post-pack prune fired, freeing 65.25 GB.
β The 150 GB ephemeral-disk ceiling β never the binding constraint. The disk line read 82.2 GB of 150 [OK] while the workload died of something else. Arithmetic in
ZEROGPU_DISK.md.β RAM β the ~68 GB figure is FALSIFIED. See the finding below; this is the single most important correction in this document.
β The private-Space fetch path β works. 37/37 files at the pinned revision, downloaded at runtime because
preload_from_hub401s on a private repo.β The fork boundary β crossed. A worker forked, ran, and released cleanly (
fail=falseβ 200).β The
sizekwarg's plumbing β the name survivedgpu_decorator_kwargs's signature filter and raised no import-timeTypeError. That is the risk PLAN Β§5.8 actually named β the platform pins its ownspacesand lags PyPI β so this is a real β on its own terms.β An explicit
sizevalue β never passed, and THAT WAS THE BUG. β RESOLVED (2026-08-27), and the inference above was wrong.app.pysentsize=NonewheneverLTX25_ZEROGPU_SIZEwasauto, which was its default and what the Space ran. The reasoning recorded here β "ZeroGPU chosexlargeby inference, packed 51.1 GB exceeds the 42.00 GiB threshold" β was the fourth document in this repo to assert it and none of the four had followed the promoted value past the function that sets it. It never reaches the scheduler:default_gpu_size()is consumed only byget_duration_seconds(client.py:109) and feeds only the duration multiplier, whilegpuSizeis sent to the platform only when the decorator's size is not None (api.py:112-113). βautotherefore ran the Space onlargeβ 47.375 GiB β for its entire life, instead ofxlarge's 94.971 GiB. Measured from inside the worker, both before and after.LTX25_ZEROGPU_SIZEis nowxlarge. Full trace:MIG_ALLOCATOR_SIGHTINGS.mdΒ§1. β οΈ Note the exact shape of the error, because it is the interesting part: the reservation really was computed asxlarge(no 1.5Γduration_factor). The hardware waslarge. We were priced as one and allocated the other.βΈοΈ Quota metering β still unreadable. All three candidate endpoints 404, so billed seconds can only be bounded, never measured.
βΈοΈ Ten of eleven modes β no conditioning, no stage 2, no LoRA, and no audio path has run on ZeroGPU. The audio modes exercise the mel front-end, the
avloader and the 16 kHz/48 kHz asymmetry; none of that has executed there once.β Annotation (2026-08-25): mostly falsified by the 2026-08-22 mode matrix. Image, audio and video conditioning have run on ZeroGPU; both audio-input modes passed; the mel front-end and the 16 kHz/48 kHz asymmetry were exercised there and passed (
SPACE_GENERATION_FINDING.mdΒ§11 item 10,SPACE_MODE_MATRIX.md). Still true, and only these: stage 2, LoRA andauto_durationhave never run on ZeroGPU.β β And that annotation is corrected in turn (2026-08-25, evening): "passed" meant a well-formed, non-empty, non-frozen file from a transformer whose conditioning path had been quantised away. Those paths executed on ZeroGPU β the mel front-end, the
avloader, the 16 kHz/48 kHz asymmetry all really ran, and that half stands. The βΈοΈ this replaced was cautioning about something real, and it should not have been retired on the strength of "passed". Banner at the top.
The three findings from the first deploy
1. β The ~68 GB RAM ceiling is FALSIFIED
It was a community cgroup observation β inherited, never ours, and load-bearing in planning since before any deploy. Startup reached 87.13 GiB = 93.56 GB peak RSS and was not killed.
β Replace it with our own measured lower bound: the ceiling is above 93.56 GB. Every headroom figure computed against ~68 GB was wrong by at least 25.56 GB.
2. β οΈ Quantising more raised peak RSS by 31.24 GiB
Adding the text encoder to
components_to_quantizemoved RSS up, from 55.89 to 87.13 GiB, in a change made to bring it down.safetensorsmmaps lazily; quantisation materialises those weights as anonymous memory, so quantising a component destroys the laziness that was keeping it cheap.This is the same single mechanism
ZEROGPU_DISK.mdΒ§4.2 already used to settle two other questions, now deciding a third β in the opposite direction from what we predicted.β Annotation (2026-08-25, evening): this is the change that broke generation, and this document records it as a RAM result and nothing else. Widening
components_to_quantizeis the blanket int8 that quantised the conditioning path; from here untileb1aa9fthe Space produced colour static in every mode. The RSS measurement above is correct and unaffected. What is missing β and was missing everywhere,README.md's int8 tier note included β is any sentence connecting quantisation to output fidelity. Nobody wrote one, so nobody looked.
3. βΈοΈ The
SIGKILLis still unexplainedcgroup-OOM is dead as an explanation: with the ceiling above 93.56 GB there was never the narrow margin the hypothesis required. β Route C β instrument the worker β is now the LEADING path, not the fallback. It is the only route that returns evidence rather than another inference, and every inference made about this kill has been wrong at least once.
β And the fix is UNATTRIBUTED
Two things changed between the failing and succeeding runs: the quantisation widening and a full container rebuild. n=1 against n=1 with two variables. "The Space now serves" is established; "the widening is why" is not β and the hypothesis that motivated the widening is itself dead. Do not let any document claim the widening fixed generation.
β VINDICATED, in the strongest available direction (2026-08-25, evening). That last sentence was written as a caution against over-crediting the widening. The widening broke generation: blanket int8 quantised the conditioning path, and every Space artifact from here to
eb1aa9fis colour static. "The Space now serves" was established and was exactly as narrow as this paragraph insisted.
Cost so far: β€ 243.58 GPU-seconds of 2400/day β οΈ a bound, not a measurement β quota endpoint 404s.
Superseded (annotated 2026-08-25). That figure covered the first-generation session only. The mode-matrix day β UTC quota day 2026-08-21 β closed at β€ ~413 GPU-s of 2400 (
SPACE_MODE_MATRIX.mdΒ§7, matrix plus re-run; still a bound, never a measurement). Quota days are per-UTC-date, so as of 2026-08-25 the day is fresh and neither number describes current consumption.
One hinode number with a direct ZeroGPU consequence. Stage 2 peaked at
83.897 GiB β
, against 70.497 GiB for a single-stage run. So stage 2 cannot
run on a large 48 GB MIG slice at all, and on xlarge (96 GB) it leaves
roughly 11 GiB before any packing overhead. That is measured on an
unpartitioned card; the packing overhead is βΈοΈ unknown and comes off that margin.
One live risk that Part A has surfaced: the stage-2 diffusion decoder resolves a NATTEN kernel from the Hub at generation time π, which is the only network dependency that fires after the user presses Generate. Whether it caches per image or per cold worker is βΈοΈ unknown and decides its severity.
4. Known gaps, stated plainly
Theβ CLOSED, AND IT WAS THE MOST EXPENSIVE ITEM IN THIS LIST (2026-08-27). The gap was written as "if anyone setssizekwarg has never carried a value.LTX25_ZEROGPU_SIZEto force a size, they are exercising an untested path" β with the unstated assumption that leaving it atautowas the safe, tested default. It was the opposite.autosends no size, and ZeroGPU's own promotion never reaches the scheduler, soautowas thelargebranch: 47.375 GiB. β The note above aboutlargebeing a slice "that Β§3's own stage-2 measurement says cannot hold stage 2 at all" was therefore describing the partition the Space was already running on β and it predicted R4b's failure precisely, a day before anyone connected the two.LTX25_ZEROGPU_SIZE=xlargeis now set and β verified from inside the worker at 94.971 GiB. Stage 2 at 704Γ480Γ49 passes on it: 53.08 s, 1408Γ960, peak 55.906 GiB reserved. SeeMIG_ALLOCATOR_SIGHTINGS.md.H10 / H11 β the audio-input modes β have not passed. βΈοΈ The path they need was connected to nothing until very recently: the UI collected a file, the encoder existed and was exported, and nothing called it. It is built and tested now, and it has not yet produced an artifact on the card. The acceptance test is H10b, not H10: two
noise_scalevalues at the same pinned seed. Identical audio means the input is still not reaching the model, whatever the stream check says β becausenoise_scale=1.0discards the input by design, so a bug that drops it entirely is indistinguishable from correct operation.β Annotation (2026-08-25): the first sentence was overtaken twice and this paragraph was never updated. Both modes then passed on the card β Β§2.2's own rows record H10 β (961,998 B at β10.4 dBFS) and H11 β (193,825 B) β and both passed again on the live Space (2026-08-22,
SPACE_MODE_MATRIX.md). What stands is the acceptance-test framing: H10b has still never run, so "the user's audio reaches the model" remains βΈοΈ unestablished; everything that has passed is consistent with the input being silently discarded.Stage 2 works; its NATTEN fetch is characterised. β Resolved once per process, not per generation β two stage-2 generations in one process, one kernel fetch. So a Space needs outbound network at a worker's first stage-2 use rather than on every press of Generate. Still a real dependency, because each cold worker pays it once. Retained below for the mechanism.
β H14 ran and produced 1408 Γ 960. But the diffusion decoder resolves its NATTEN kernel through a version β ref lookup that goes over the network even when the kernel files are cached π, and whether that fires once per process or once per generation is βΈοΈ unknown. It cannot be grepped after the fact β the run log contains zero natten mentions β so answering it needs deliberate instrumentation. On a private Space this is the 401 class again, except firing after the user presses Generate.
connectors: 0on every accepted adapter. β observed. Stage 7 asserts total modules attached > 0, which passes on the transformer alone β so an adapter that ought to touchconnectorsand silently does not would still be reported as attached. Probably correct for the three adapters tested (2.3/2B-era), so not currently a defect β but the assertion as written cannot distinguish "legitimately unused" from "silently missed". A per-component expectation would close it. See PLAN Β§4.4.temporal_latent_upsamplerhas no known role. π It is the same class as the spatial upsampler, differing only by axis, and upstream's component table and both documented recipes mention only the spatial one. We download it (0.2619 GB) and do not wire it in. Not a defect; an open question.β
video_to_audionear-silence β fixed, and the first diagnosis was wrongH9 came back at β55.0 dBFS against
text_to_audio's β5.2 β . The cause was the distilled recipe reaching only the stage-2 path (item 7). With that corrected, H9 runs at β17.8 dBFS β 2.5 dB from its siblings β at the unchanged condition strength of 0.8.before after H9 video_to_audioβ55.0 dBFS, peak 0.0146 β17.8 dBFS, peak 0.5078 H8 text_to_audioβ5.2 dBFS, peak 1.0000 (railed) β15.3 dBFS, peak 0.6953 gap ~50 dB 2.5 dB One mode 50 dB down and another driven into the rail were the same misconfiguration seen from opposite ends.
β οΈ The first diagnosis blamed condition strength and was wrong. A single-lever sweep at fixed seed gave 0.8 β β55.0 and 0.5 β β5.1, and that was read as the cause. It was a real interaction between condition strength and the broken guidance β a genuine effect inside a pipeline that no longer exists. A per-mode strength cap was written, committed and reverted.
A single-lever same-seed sweep is strong evidence that something moves, and no evidence about why.
The source citation said throughout that a video condition cannot attenuate audio in the denoise loop β the conditioning mask is applied to
noise_pred_videoand never touchesnoise_pred_audio. It was correct, and a clean correlation was believed over it.The audio input rate is 16000, the vocoder output is 48000. π They are not symmetric. Anything that assumes one rate for both directions will pitch-shift audio without erroring.
β The distilled recipe reached only the stage-2 path β fixed
Eleven of twelve runs sent no sigmas at all, so upstream built its own schedule and five guidance parameters took their SFT defaults β
audio_guidance_scaleat 7.0 against an intended 1.0.sent: [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125] intended: [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]β All twelve artifacts differed before and after, with magnitudes ordering by conditioning strength (text-only 37β59 mean |diff|, image-conditioned 15β16, video-conditioned 3.71).
βΈοΈ Neither of those establishes the fix caused them, and the ordering is not the corroboration it looks like. Both halves were unseeded (gap 8), so two runs of one mode differed anyway β and a more strongly conditioned mode is more constrained, so its run-to-run variance is smaller too. The ordering is exactly what random variation would produce. It cannot separate the two explanations, and an earlier draft of this section claimed it could.
What does stand, unconfounded:
- the timings. 25.7 β 8.4 s is not run-to-run variation, and it follows directly from the mechanism: unguided means one forward pass per step instead of two or three. The defect cost 3Γ the compute, and that is measured rather than inferred.
- the audio levels and peaks β β55.0 β β17.8 dBFS, highest peak 1.7811 β 1.0050. Systematic and orders of magnitude outside variation.
- π that the schedule and guidance were wrong at all, which is a source fact and needs no run.
It cost 3Γ the compute as well as the quality. At
guidance_scale=1.0with STG and modality zeroed there are no classifier-free branches, so it is one forward pass per step rather than two or three:before after H1 text_to_video25.7 s 8.4 s H8 text_to_audio5.6 s 1.8 s H14 stage 2 84.3 s 35.2 s H16 1216Γ704 + stereo 68.7 s 21.0 s Clipping went with it β highest peak 1.7811 β 1.0050 β β so no limiter is needed and
outputs._write_audio's "what comes back is what the vocoder produced" principle stands. Our path never added gain; the material was arriving at Β±1.0. Stage 2's brightness falloff was this too, not stage 2: mean99.48 β 34.44collapsing before,82.52 β 110.47stable after.β The seed reached only the sidecar β fixed, and verified byte-identical
_resolve_seedran and the number went into the encoder metadata. There was no generator anywhere in the package, so every run was unconditionally random and every "same seed" comparison was two independent runs.β Fixed in
4ebb8b9: a generator seeded from the request, built on the pipeline's own device, threaded to the pipeline and toaudio_vae.encode's.sample().β Verified on the card, for
audio_to_audio. Two runs at identical seed and configuration:run-a1d35aaaab06/output.wav md5 2988f01159a5322afb28abb6e9a73d26 run-b9dde07ed8bb/output.wav md5 2988f01159a5322afb28abb6e9a73d26β And the fix was complete when it landed, not repaired afterwards:
git log -S 'generator' 4804da1..e4e8df1 -- ltx25_studio/returns4ebb8b9alone.On this path,
seeddetermines the output byte-for-byte, on GPU, with no deterministic-algorithm flags set. Stronger than expected β the weaker "seed-consistent but not bit-reproducible" was drafted and never needed.βΈοΈ Scope, and it matters for Β§2.5: that measurement is one mode. The video path is not shown to be bitwise reproducible β it is evidenced on the Space (byte-identical output across two days and two containers at one seed, plus
text_to_audio_videomatchingtext_to_video's video statistics to three decimals) and no two-run frame-hash comparison has ever been made.docs/SPACE_MODE_MATRIX.mdΒ§5 states that limit explicitly.β Correction (2026-08-25): "byte-identical output across two days" is too strong β the cross-day evidence is size-identical plus statistics-identical; no day-1 hash exists and the day-1 file is gone, so byte identity was never checked and now cannot be. The conclusion of this passage β not shown to be bitwise reproducible β is unchanged and was the correct one.
SPACE_MODE_MATRIX.mdΒ§5's correction records the planned sha256-compared re-run against the surviving 2026-08-22 artifact.β The twelve-mode pixel-diff gallery that depended on it is DROPPED β the host it ran on is gone. So the consequence is permanent rather than pending: the Β§2.5 comparison table was measured before seeding worked, its pixel diffs cannot attribute the change to the recipe fix, and nothing is coming to replace it. Everything structural was never affected β H14's exact 2Γ resolution, H15's 97 frames, H17's frequency recovery, and every audio level.
The lever that should have caught it was ours:
decode_kwargstook ageneratorargument, with a docstring explaining why the diffusion decoder must be seeded, and read a key nothing ever set. See PLAN Β§9 rule 12.βΈοΈ
noise_scaleandsigmas[0]disagree β open, deliberately unfixedπ Upstream infers one from the other β "first (largest) sigma value if using custom sigmas" β and our
stage_2_kwargshonours it. The audio-input modes do not: they sendnoise_scalewhilesigmas[0]stays 1.0.It was diagnosed as the cause of a silent
audio_to_audioand it was not (Β§4.1). A fix was written, committed and withdrawn: truncating the schedule atnoise_scaletakes an eight-step run to two steps at 0.5 and one at 0.1 β values that measured healthy on eight.Agreeing with a contract is not evidence that a change improves anything. A test pins the current behaviour and explains why, so the next attempt arrives with a measurement rather than with agreement.
4.1 β οΈ Known limitation β unusual audio input produces a silent file
Not a defect we can properly fix, and the most likely way a user meets a disappointing result.
audio_vae.encode is never called by the pipeline. Our
encode_waveform_to_latents is the only producer of those latents, and
nothing anywhere validates their distribution β not upstream, not us.
So audio that is far from what the model was trained on encodes to latents the
model cannot use, and the result is a structurally perfect, inaudible file
with no error and nothing to act on. β
A real instance, and it cost a day of
misdiagnosis: a two-tone synthetic fixture at β14.8 dBFS in produced
β87.9 dBFS out, with correct codec, channels, rate and duration, and
verify_streams exit 0. The same mode fed a real recording produces β10.4
dBFS.
Likely triggers: pure tones, sine sweeps, test signals, near-silence, heavily synthesised material. Ordinary recorded audio is fine β every realistic input tried has worked.
β Do not "fix" this with an out-of-distribution detector. Reliable OOD detection at that boundary is not something this project can do, and a detector that is wrong in either direction is worse than the limitation.
βΈοΈ A narrower thing is worth doing and is deliberately not done yet. We
measure the input level and can measure the output level, so a file that
comes back essentially silent could say so in GenerationResult.warnings.
The decision that makes it honest is that it must be a measurement, not a
diagnosis: "this output is β87.9 dBFS, effectively silent" is a fact about
the artifact, checkable and cause-free. "Your audio was out of distribution"
is a claim we cannot support.
It is not built because it needs a threshold, a threshold needs evidence, and the PR opens with one named defect. Adding an unvalidated heuristic to the export path hours beforehand is the same move as shipping a schedule change on a contract reading β which was reverted for that reason. It belongs in the next pass, with a measured floor.
5. How to run it
On hinode
./scripts/sync_to_hinode.sh # ships git archive HEAD, not the working tree
rsync is deliberately not used: it is not installed on the host and there is
no sudo to add it. Needs an HF token with access to the gated model repo. Then
scripts/hinode_run.sh, or scripts/host_matrix.py for the full matrix.
Do not set HF_HUB_OFFLINE=1. Construction loads from the local snapshot
directory, so nothing is fetched at load time β and stage 2 needs outbound
network for the NATTEN kernel. Forcing offline breaks the second without helping
the first.
Locally, with no GPU
The whole test suite runs with no GPU, no weights and no network. That is a deliberate property, not a coincidence, and it is worth preserving: it is what makes it possible to catch a defect before it costs a run.
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/python -m pytest -q
torch, diffusers and torchaudio are not needed for the suite. av and
numpy are, because they are how real containers get written and read.
On a Space
β
This works. The repo is the Space β README.md's YAML front matter is
the Space config, and the space remote points at it:
git remote -v | grep space
# space https://huggingface.co/spaces/techfreakworm/LTX2.5-Studio
Deploying is a push to that remote's main. β This team does not push β not
to origin, not to space. The orchestrator does it; that is a deliberate
boundary, not a missing capability.
What the deploy depends on, all of it already load-bearing:
app_file: app.py, sdk: gradio 6.24.0, python_version: "3.12" |
π front matter |
suggested_hardware: zero-a10g |
π a legacy id string β not an A10G. Changing it moves the Space onto paid dedicated hardware |
startup_duration_timeout: 1h |
π required. The default is 30 min and this Space downloads 72.19 GB at build time. When it blows, the failure looks like something else entirely |
| an HF token secret with access to the gated model repo | π weights fetch at runtime, because preload_from_hub 401s on a private Space |
tests/test_deploy_drift.py asserts these rather than trusting them β run the
suite before any deploy and the drift shows up locally instead of as a
40-minute build that dies at the end.
Reading the startup log is the cheapest diagnostic available, and it costs
zero GPU quota because packing and loading happen at module scope, before the
first @spaces.GPU call. The load line names its own configuration:
quantised transformer, text_encoder as int8 | peak RSS 0.91 -> 87.13 GiB
β οΈ That peak is the number to watch. It is a peak across the load, not a
steady state β we have never measured steady state, and the RAM questions are
about steady state. See SPACE_GENERATION_FINDING.md Β§7.
6. The lessons worth carrying
Not a diary. These are the ones that would change how the next person works.
A fake derived from the mapping can never catch a wrong mapping
The audio VAE's six parameters were read from audio_vae.config by names that
do not exist there π β four live under different keys, two are instance
attributes absent from config.json entirely. Every runtime path through that
module raised on real weights, and every offline test passed, because the fake
used our names. It agreed with the code and with nothing else.
The instinctive repair β build the fake from the same mapping the code uses β is a trap for exactly the same reason. The fix is a recorded fact about reality: the real config's key set, checked into the tests with its provenance. The fake checks semantics; the recorded keys check the world.
The fixture that makes a test possible is itself untested by that test
Three instances in one day, all in tests that were otherwise excellent:
test_encoder_contract.pydrove the real encoder through real containers to a real file β and built its audio asnp.float32. The model producesbfloat16. Six of eleven modes were broken at the last step and 1926 tests passed β both counts measured β the modes on hinode, the tests in this repo at the commit before the fix.- The replacement fake's
.float()returned anndarraywhere torch returns a tensor. Written the convenient way, it would have passed a broken fix. - The audio-VAE fake used our own key names β the case above.
The mitigation that worked: check the fake against the real thing. In (2) that check failed within a minute of existing.
A search that cannot see the claim reports that the claim is gone
Three defects in this build share one shape: the instrument returned a confident negative it was structurally incapable of turning positive.
- A wrapped line. Β§7 said "The Space has evicted twice and has never run" as one line. Β§1 said the same thing split across a line break, so a grep for the phrase found the occurrences that fit on one line and pronounced the audit clean.
- A shell modifier.
"$c:ltx25_studio/..."is parsed by zsh as the:llowercase modifier, sogit showgot a mangled path and every commit answered "constant absent" β including the one that demonstrably contains it. - A stale marker. A
β οΈ (inherited)pointing at a section that had been rewritten to say FALSIFIED still scanned as merely unverified.
None of the three raised an error. Each returned a clean, plausible, wrong answer, and two of them nearly closed an audit.
The defence is not a better search string. It is running the instrument against a case whose answer you already know: if the check cannot go positive on a known positive, its negatives carry no information. The zsh case was caught only because "absent at HEAD too" was impossible on its face.
A claim moving known β unknown is nearly always a regression
Documents accrete facts and rarely lose them. When an edit is about to record something as uncertain, search first for it being known β the likeliest explanation is that someone established it and the current author has not read it. This has since caught its own inverse: a correction issued against this document's history that the history did not actually contain.
Checks over comments
Twice, a constraint written down did not survive the next function written by
its own author β a dtype requirement violated two commits after the β οΈ comment
demanding it, and a 44.33 GB refetch reintroduced while reading the docstring
that predicts it. A comment is a message to a future stranger; it has no
purchase on the author five minutes later. Make the wrong thing unsayable
instead: derive the value once and share it, so the two callers cannot drift.
The instruments encode the failure you imagined, not the failures available
host-runner's generalisation, in their words:
all three instruments encoded the failure I imagined rather than the failures available
β a pkill -f pattern that matched the killer, a wait loop with no failure
exit, and a fixture guard that handled a fixture being absent but not one that
raises. The three had different authors, including the lead. That is the
point of recording it: it is not one person's blind spot, it is what happens
whenever the person who imagines the failure is the person who writes the check
for it.
We can prove a function ran. We cannot prove it ran with the right arguments.
The gap this build ends on, and the one worth closing next.
DISTILLED_SIGMA_VALUES was defined, documented as mandatory, quoted in its
module's docstring, and referenced β so every check we own passed. It was
referenced from stage_1_kwargs(), which StageRunner applies only when
stage 2 is enabled. Eleven of twelve host runs were generated with upstream's
linear fallback and its SFT guidance defaults instead:
sent: [1.0, 0.875, 0.75, 0.625, 0.5, 0.375, 0.25, 0.125]
intended: [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]
A reference check passes. A reachability check at module level passes. Even the function-level sweep passes, unless someone thought to name that particular function. What nothing checked was the arguments the pipeline was actually called with.
The pattern that does work, used three times here, is to record the vendor's
specification independently and assert against that β MODEL_CARD_DISTILLED,
REAL_AUDIO_VAE_CONFIG_KEYS, REAL_VOCODER_CONFIG_KEYS. Each is a fact copied
from the source of truth rather than derived from our own constants, so changing
ours cannot quietly redefine what correct means. Without that independence the
test agrees with the code and with nothing else.
The escalation ladder, in the order this build climbed it: is the module imported β is it reached by the path that does the work β did this specific function run β were the arguments the ones the vendor specifies. Only the last rung is still hand-built, one call site at a time.
A module can be reached and still contain a dead function
The reachability sweep that exists to catch importer-count-zero missed one, for two independent reasons: the tracer recorded which file ran and discarded which function β the name was in the same frame object β and the probe drove one mode, so a path reachable from no other mode was invisible by construction. Either fix alone would still have missed it. "Sharpen the instrument" was the obvious response and would not have worked.
The one time source-reading and measurement confirmed each other
Every other pairing today went the other way: one corrected the other, and the lesson was about which to trust when they disagreed. Once, they agreed.
A reproducibility control failed β two runs at one seed, 8.7% apart β and the
obvious reading was an unseeded draw somewhere. Instead of hunting, every
noise source in the pinned pipeline was enumerated: _create_noised_state,
the conditioning mask, prepare_*_latents, the decode timestep, the condition
VAE encode. All threaded the generator; the encode used sample_mode="argmax"
and was deterministic regardless. Source said it should already be
reproducible.
Rather than write either sentence β "the seed is attached to nothing" or "we are seed-consistent but not bit-reproducible" β a scoping run was requested. Before it ran, the control was re-checked: it had been measured against a tree that predated the seeding fix. At the tip, two runs are byte-identical.
The true claim was stronger than either draft: on
audio_to_audio,seeddetermines the output byte-for-byte, on GPU, with no deterministic-algorithm flags. Both drafted sentences were weaker and wrong, and both would have shipped if the reluctance to write an unmeasured claim had not held for one more exchange.
And then the corrected sentence was written without its scope, twice more in this document β as a property of the build rather than of one mode. βΈοΈ The video path is still unverified. Recording that here because the lesson is not "be careful": it is that a claim narrowed in one place does not narrow itself in the others, and a document is exactly where that goes wrong. Β§4 gap 8 carries the scope; Β§2.5 and this paragraph each had to be found and fixed separately.
Refusing to write a sentence you cannot yet support is the same discipline as refusing to change a threshold to make a test green β and it is the cheaper of the two, because nobody has to notice you did it.
Staffing β two makers and a lead who verifies
Not five, and not one. This build ran briefly with five teammates and settled
at two: one implementer owning the code, one runner owning the hardware. The
three stood down were not underused β they were a coordination cost with no
matching throughput, and the collision they produced (two agents assigned
outputs.py; a cross-boundary change landing under a commit message about other
files) cost more than their output was worth. Two makers with disjoint, stated
file ownership is the shape that worked, and the line held all day under
pressure: the runner found four product defects and edited none of them.
The scarce resource was execution on real hardware, not review capacity. At
the point the suite reported 1926 passing β
, six of eleven modes were broken,
the audio-input path was connected to nothing, stage 2 was unreachable, and every
runtime path through audio.py raised on real weights. None of that was found by
reviewing harder β it was found by an afternoon on a real GPU. Another reviewer
would have added agreement, not information. Budget for one more agent is
better spent on a second executor with different hardware than on a second
reviewer.
A lead who cannot edit is forced to verify by observation rather than by
reading their own diff, and that constraint turned out to be a feature. Every
claim the lead checked was checked by running something β git merge-base --is-ancestor rather than trusting a commit message, av on the container
rather than trusting a verifier's exit code, pgrep on the box rather than
trusting a status report. That caught the box silently trailing the tip three
times, a stale bypass premise, and a wait loop that had frozen a teammate. A
lead who could edit would have been tempted to fix those, and would have stopped
looking.
The default-REJECT verifier role was filled by reality rather than by an
agent. The plan called for an independent verifier holding a default-REJECT
stance; in practice the hardware held it. It rejected the from_pipe cast, the
bf16 encode, the missing audio parameters and the unreachable stage 2 without
being asked, and without any possibility of being talked round. Where a real
gate exists, prefer it to a reviewer agent β an agent can be persuaded and a
RuntimeError cannot.
What it cost. Message-crossing: three times a blocking item was buried inside a long message and lost a cycle behind work already in flight. The fix is mechanical and worth adopting from the start β one blocker, one message, nothing else in it. Long messages are for context; short ones are for priority.
7. If you do one thing
Re-run the comparisons seeded. Everything works; the open item is that some of the evidence was gathered before the seed reached the model.
The runs that need repeating, and what each would then establish
| run | what it would establish | why it does not yet |
|---|---|---|
| Β§2.5 before/after pairs | that the recipe fix caused the pixel differences | both halves were unconditionally random |
H10b β noise_scale 0.1 vs 0.9 |
that the user's audio reaches the model | same, plus the audio encode drew unseeded |
| H12b β LoRA effect | that attaching the adapter changed the output | same |
H4b β strength 0.3 vs 0.9 |
that the conditioning clip reaches the model | same |
Add a same-value control to each β two runs at identical settings. Under a working seed its spread should be zero, and that is what turns "the halves differ" into evidence rather than a coin toss. An earlier control measured 1.8% and was read as a bound on the variance; an unseeded control cannot bound anything, which is the mistake worth not repeating.
β The first row of that table is dead (annotated 2026-08-25), and this file already said so elsewhere. The Β§2.5 before/after pairs cannot be re-run: the seeded gallery is dropped with its host, and Β§2.5/Β§4 gap 8 record that as permanent β the table is unattributable forever, not pending. The live content of this instruction is the other three rows: H10b, H12b and H4b are still runnable β on the Space (none needs stage 2; each is one same-seed pair at 256Γ256Γ9-class cost) or on a future host β and each still establishes exactly what the table says.
Two of the three were delivered on the Space later on 2026-08-25: H12b's claim (the adapter changed the output β
20a3a84bβ¦βb78803ccβ¦at the same seed) and H4b's question (the conditioning clip reaches the model β answered by clip-swap: outputs 17.277 vs 0.864, mirroring their inputs, against a byte-identical control). H10b is the last discriminator standing.SPACE_SESSION_2026-08-25.mdΒ§6βΒ§7.β BOTH DELIVERIES ARE WITHDRAWN (2026-08-25, evening). Every run cited there used the broken int8 build. H12b's two hashes differ because one garbage render differs from another; H4b's 17.277-vs-0.864 compared the motion energy of two colour statics. The mechanical readings stand β the adapter injection path ran, the conditioning path carries a scalar, the noise floor at a fixed seed is exactly zero β and all three discriminators are open again: H10b, H12b and H4b. They are also worth spending on now, because there is finally a build whose output is a picture (
e3b5333).
What is already earned, and does not need repeating
Eleven of eleven modes produce artifacts with the expected stream layout on the real path. β H14's exact 2Γ resolution, H15's 97 frames, H17's round-trip recovering 4/4 tones under a strict gate, both LoRA directions, and every audio level β all structural or systematic, and none of them depends on seeding.
And read Β§3 before treating any of this as a deployment. β
The Space now
runs and generates β βΈοΈ for one of the eleven modes. Everything in this
section is a hinode result; the Space has proven text_to_video at
256Γ256Γ9 and nothing else. Those are different claims about different machines,
and Β§3 is where they are kept apart.
β β Corrected (2026-08-25, evening): the Space had proven that text_to_video
executes; its output was colour static. This paragraph's instinct β keep
Β§2's hinode results apart from anything the Space says β was the correct one,
and every later annotation that softened it was wrong. Everything in this
section is hinode, bf16, unquantised, and unaffected.
β And that supersession is itself corrected (2026-08-25, evening): "content-verified" meant non-empty and non-frozen, on the broken int8 build. All eleven modes execute on the Space; none of their output was ever verified as an image. Banner at the top of this file.
β Superseded (annotated 2026-08-25): the Space has since proven all eleven modes at 256Γ256Γ9 β content-verified, sha
287ba9c, 2026-08-22 (SPACE_MODE_MATRIX.md). What it has still proven nothing about: stage 2, LoRA,auto_duration, and any resolution anyone would ship. Β§3's annotation carries the merged state.And later on 2026-08-25 that residue shrank to
auto_durationalone (plus the stage-2 ceiling's exact position): stage 2, LoRA and 1216Γ704Γ121 all ran on the Space that day. Top-of-file note;SPACE_SESSION_2026-08-25.md.