Spaces:
Running on Zero
Running on Zero
Commit History
3 days ago
-
docs: every sentence stays true after the deploy: README and the release notes say this is deploy 2's code and point to issue #32 for what the Space shows, and no doc says the Space still runs deploy 1, is not deployed, or has not been redeployed since August (#32)
Unverified signatureThe commit we deploy is also what origin/main is fast-forwarded to, and nothing can be pushed after the deploy without restarting the public Space, so a sentence that a deploy makes false would stay false. README: "Where things stand, 2026-11-17" says the README describes deploy 2's code, which replaces deploy 1 on the Space, points to SPACE_MODE_MATRIX 10.1-10.3 and the release notes, and to issue #32 for deploy 2's own check on the Space. The cold-start time points to #32 instead of "has not been measured"; the longest accepted reservation is "not measured by these runs"; the browser runs of 2026-11-16 "did not finish in that session" instead of "have not been run ... yet". docs/RELEASE_NOTES.md: "Deploy 2 - 2026-11-17", the code of this commit's tree, the Space result on #32 (also under "On the Space"); deploy 1 "ran" as 0cd54f2 until deploy 2, and main "was fast-forwarded" to 1e9fd20 that day. The other docs, the same way: SPACE_MODE_MATRIX's header and 10 intro, 10.2 and 10.3; …abc9150 -
docs: the diffusion decoder is built once at start-up, beside the pipeline, and packed with it on the Space, with each run building its own only if that build failed: the README's decoder paragraph and two PLAN passages say so (#17)
Unverified signatured8bbbdb -
docs: the weights come from the image: the README, PLAN and the handoff say preload_from_hub fetches the 35 model files (71.92 GB, no temporal upsampler) and the prompt enhancer's 8 (2.49 GB) before the app starts, and start-up downloads only what is missing, if the disk can hold it (#23)
Unverified signature41dcf05, ff62c55 and 72b989d made the preload live: the README's front matter (B3's) fetches the 35 files the int8 Space loads at their pinned commits and the enhancer's 8 at its own, and preflight.ensure_weights checks the cache at start-up, downloads only the missing files after one warning, and refuses a download the disk cannot hold (missing bytes + ZeroGPU's packed copy + a 10 GB margin, under the 150 GB cap) before writing a byte. The docs still said the Space downloads 72 GB at every start because the preload 401s on a private Space. README: the enhancer paragraph says the image brings its weights on the Space and the start-up prefetch (CUDA only, app.py) is the fallback; the ZeroGPU table gets a Weights row. No cold-start time is claimed: none is measured with the preload. docs/PLAN.md: 2.3 notes the 35-file set, the deploy-shape table's Space and Weights rows are corrected in place (public; from the image), 6.2 and 6.2.1 open with a 2026-11-17 correction, and the enhancer-pre…38e36e9 -
docs: the reservation follows size and length: the README's table is the estimator's (the default run 60 s, needing 120 s of quota), the measured envelope is a stated limit, and SPACE_MODE_MATRIX §10.3 records the six size-and-length runs and the answer to the operator's question (#21)
Unverified signatureac68de5 made zerogpu.estimate_duration a fit of the Space's measured runs with a 1.3 margin per term; the docs still said a flat 180 s, x1.5 with Upscale, +60 with an adapter and +180 with Enhance. README: the reservation section says how a run is priced now and gives the estimator's own figures (computed from the code: the default 704x480, 5 s run reserves 60 s and needs 120 s free; +45 s with Upscale at that size, +40 s with adapters, +80 s with Enhance; 1280x704 at 10 s 145 s; 960x544 at 10 s with Upscale 240 s; a sound-only mode 45 s), that LTX25_GPU_DURATION_SECONDS is ignored and LTX25_GPU_DURATION_FLOOR_SECONDS sets a minimum; deploy 1's GPU windows stay as the history they are. The limits section states the envelope the Space refuses past (64,000 tokens per pass, 520 million returned pixel-frames) in the refusal's own words (tests/test_reservation.py). Enhance's "reserves 180 s more" is 80 s, Upscale's x1.5 is 45 s at the default size, and the Upscale paragraph gives what the …ca1f2a8
4 days ago
-
docs: the output folder is described: one outside the working directory and the temp dir is passed to Gradio as the one allowed path, and a result the page cannot show ends with a card, never on "Generating…"
Unverified signatureREADME, local settings: a row for LTX25_OUTPUT_DIR (outputs by default). A folder that resolves outside the working directory and the system temp dir is passed to launch as the only allowed_paths (e7cde1a), so the page can show what is written there; Gradio 6.24 reads GRADIO_ALLOWED_PATHS only when no allowed_paths is passed, so for that launch the folder takes its place. The README's closing rule (no code path where you press Generate and see nothing change) now names the case it missed until 55692f5: a file Gradio refused to serve left the page on "Generating…"; such a run ends with Generate back and a card saying the run finished but its result could not be shown, with the server's message. PLAN.md §2.7 gets the same as an "as built" note, with the one caveat from e7cde1a: a launcher that calls app.demo.launch() itself must pass the same arguments.c3c8672 -
docs: the local memory check is described as it now works: it waits for memory just freed to leave the gauge, reads a failing check again before it refuses, and its refusal is a kind of its own
Unverified signatureREADME, local section: a row for LTX25_MEMORY_SETTLE_SECONDS (2 by default, 0 to 60, 0 switches the wait off) and a paragraph on why the check waits. Freed GPU memory on the Mac leaves the gauge over a moment: the kernel gave the text encoder's 24 GB back over 0.1-0.35 s, so a reading taken at once was 3.5-23 GB too high, and with a browser and test suites running it refused a run that fitted. After a release the check now reads the gauge until two readings 50 ms apart agree within 0.25 GB; a failing check is read again every 0.1 s; both within that bound (memory.py at ba2d582 and a5caa41). On real weights the settled readings were 3 to 13 GB lower, in 0.16-0.36 s, and the three refused runs ran. A refusal is reported as its own kind, not as an app error (errors.MEMORY, 04c841e). The README's "a real-weights run through the UI on a Mac is not verified yet" is no longer true: through the page in a real browser, audio to video and text to video with an uploaded adapter finished; text to…68c528c -
docs: the local matrix records the final pass: every mode, the adapter, the enhancer and the model-chosen length run again on the final code, identical wherever compared with a run made after the RoPE fix
Unverified signatureSPACE_MODE_MATRIX.md §10.2 gets the lead's final pass (run log, 2026-11-16 19:16-20:16): fourteen runs on e59944e, each with the inputs, prompt, size and seed of an earlier run and compared with it, plus F11b_ocean and F11c_rail for mode 11's new sample and second example. Modes 01, 03, 04, 09, 10, 11, the adapter, the enhancer and the model-chosen length are identical to their references (SSIM 1.000, sound 1.000); 02, 05, 06 and 07, compared with runs from one commit before the float64 RoPE fix, show the same scenes framed slightly differently (SSIM 0.78-0.94), the same words and sound, and frame 0 of mode 02 at 0.7507 as before; mode 08 has the same sound. Lip sync on these outputs: +0.845 in mode 07, -0.379 in mode 11. Three runs refused on e59944e by the memory check's early reading (mode 08 twice, the enhancer, the model-chosen length at the denoise check) ran on ba2d582, with the settled readings 3 to 13 GB below the first ones; their peaks (77.0, 77.6, 84.5 GB) and the pass's (6…0300944 -
docs: running locally is described as one code path with the Space: the same stage graph, with the local runner supplying only what is resident when
Unverified signatureSince c712e7e the local runner is a StageRunner: the decode policy, stage 2 and its refusals, the audio conditioning, the clip variation and the run facts are one piece of code, and the local side adds residency through one seam. The README's local section and PLAN §7.1, §7.4 and §8.7 now say so, and what follows from it: off CUDA the default decode is the conv VAE and the run says why, and an explicit diffusion decode or Upscale x2 is refused before dispatch except at very small sizes. The measured memory and time figures stay, with the day's full range (peaks 57-72 GB at 704x480 or 480x704, up to 193 frames) and the enhancer's runs added. SPACE_MODE_MATRIX §10.2 says which of its runs predate the change.1f05c04 -
docs: the audio modes are described as measured after the audio-encoding fix, and the API's noise_scale default is said to snap to "a little"
Unverified signatureThe known-issue box about the 9 dB loss is gone: the cause (the mel transform's window taken from the vocoder's config) is fixed at 7cf1547. In its place the README states what the lead measured on real weights afterwards: audio to audio returns sound at the input's own level, graded by the choice (same events at the same times 0.97 / 0.95 / 0.71 on speech, transcript exact at "a little"); video to audio follows what is visible (on a silent clip of a shouting man, a voice whose loudness follows the clip's original track at 0.92); audio to video carries the recording exactly and takes its length from it, with or without a prompt and a first-frame image. Through the API, noise_scale has to be passed for video to video and audio to audio: the published default 0.5 snaps to 0.42 in both. SPACE_MODE_MATRIX §10.2 rows 10 and 11 carry the runs at 7cf1547 and the fixed defect, including the black-and-white pictures it caused; PLAN §5.4 says the analysis window is a declared convention, not a …67c1514 -
docs: video to video is described as verified on real weights: a variation of the clip at three levels, with the prompt acting at the highest
Unverified signatureThe README, PLAN §3.3 and the local matrix (SPACE_MODE_MATRIX §10.2) now say what the lead's runs of the new mechanism showed (V04_* at 85405e9, S04_summer_l091 at 7cf1547): SSIM to the clip 0.78 / 0.65 / 0.36-0.40 at "a little", "moderately" and "a lot"; the prompt acts only at "a lot"; it runs without a prompt; a sideways-stored phone clip comes out upright and a 60 fps clip is read in real time; the length is the clip's. All eleven modes are therefore verified locally, and every count of them says so. The follow-up comments the first docs pass left are gone. The Matroska/WebM length defect (F70) is not listed as a known issue: it is fixed at 16bbbd3.41d0e94 -
docs: PLAN says the enhancer prefetch failure was seen in one boot log, with a cause that applies to every boot
Unverified signatureThe correction added in f92f481 said the prefetch "failed on every boot". Only the 2026-11-13 boot log has been read; the cause (Gradio's process-wide tqdm patch) applies to every boot, and the sentence now says exactly that.c09685b -
docs: PLAN states the exact pins, the local Apple-silicon path, and corrects what the code and the measurements overtook
Unverified signature§5.1 now lists every requirement as the one version requirements.txt pins (transformers 5.23.0 with its companions, the torch trio, bitsandbytes, kernels, gradio, numpy, ...), says why `spaces` is absent (the platform installs it; the Space reports 0.51.3), names the fallback (transformers 5.22.2 with the same companions), and says which lines have no real-weights evidence yet: the CUDA-only ones, the int8 load under 5.23.0 in particular. §3.5 no longer calls 5.15.1 the floor. Corrected in place, each with the date and the source: - status: ten of eleven modes verified on real weights locally; none re-run on the Space; "all eleven generate at 287ba9c" was colour static; - the tier: `auto` never promoted; the code now defaults to xlarge; the quota arithmetic (180-510 s reservations, x2 documented not measured) and the "drop under 42 GiB" argument; VRAM was the constraint on `large`; - disk: "82.2 GB of 150" is a pre-pack reading; the bar read 40.6G packed; packing happens at l…f92f481
-
docs: the app ran its whole life on half a GPU — `auto` cannot promote, and four documents said otherwise
Unverified signature`MIG_ALLOCATOR_SIGHTINGS.md` was a register of three NVML allocator asserts with no explanation and a ruling that nobody chase them until somebody read the headroom. `0a4ddbb` read it. The number was **47.375 GiB** — ZeroGPU's `large` — where every document in this repo assumed `xlarge`'s 94.971 GiB. ⛔ THE ROOT CAUSE, IN FOUR HOPS, ALL FROM THE PLATFORM'S OWN PACKAGE: * `spaces/zero/__init__.py:29-35` — `startup()` packs, clears the 42.00 GiB `xlarge_threshold`, and sets the module global `_default_gpu_size='xlarge'`. The promotion is real and it does fire. * `spaces/zero/client.py:109` — `get_duration_seconds()` is the **only** consumer of `default_gpu_size()` anywhere in the package, and it rebinds a **local**. * `spaces/zero/client.py:111-112` — that local feeds one branch: `if gpu_size != 'xlarge': duration_seconds *= duration_factor`. The promotion's entire observable effect is skipping a 1.5× multiplier on the *requested duration*. * `client.py:144` → `api.py:112-…caf1083 -
fix: the enhancer's nine-minute wall — the rewrite moves onto the GPU it was priced for
Unverified signaturePrompt enhancement shipped in the web process, on the CPU, to stay out of ZeroGPU's packing window. The disk half of that argument was researched, checked against the packer's source, and is correct. The latency half was never measured. On the live Space, text_to_video at 256x256x9: enhance OFF 15.38 s enhance ON, cold 654.15 s (+638.8) enhance ON, warm 553.45 s (+538.1) The cold/warm split isolates it: the 2.46 GB fetch is 88 s and the weights load in under a second, so the download is not the cost. ~538 s is the CPU decode of 600 greedy tokens, about 0.9 s/token. The "1-2 s" that justified the CPU placement was a GPU number applied to a CPU placement — 600 tokens in 2 s is 300 tok/s, which no CPU does on a 5B-parameter model. "Where does it run" has two axes, disk and latency. One was optimised and the other was never measured. That is the transferable part and it is written into PLAN §3.5, UI_DESIGN and enhance.py rather than into this message alone…f967e30 -
feat: prompt enhancement ships — a real toggle, in the web process, at zero GPU cost
Unverified signatureThe build refused this twice, and both reasons were wrong. ⛔ "The pack ships it as absent, so `enable_prompt_enhancement=True` fails on our subset." `model_index.json` does carry `prompt_enhancer: [None, None]`, and that is not what it means. Upstream types the component `Gemma4ForConditionalGeneration | None` (pipeline_ltx2.py:249) and lists it in `_optional_components` (:235) — a plain transformers class. The empty slot is there because the component is CONSTRUCTED, not resolved from the snapshot. So it is an ordinary runtime `from_pretrained` into the HF cache, and nothing in model_files.py / preflight.py / preload_from_hub needs to know it exists. ⛔ "ZeroGPU charges it twice at the packing peak, ~20.6 GB." A placement artefact, not a property of the model. Packing runs once per container, at `demo.launch()`, over tensors registered while the module scope ran. A model loaded lazily inside a request handler, on the CPU, is in neither window. It is charged once, as a download. ✅ An…2f36d8a
-
docs: every Space generation result was garbage — the invalidation, stated at the top
Unverified signatureThe deployed Space produced colour static for the entire project. Blanket int8 quantised the LTX-2.5 conditioning path (the eight LTX2AdaLayerNormSingle modulation heads plus the in/out projections), the denoiser stopped being steered, and the output became a correctly-shaped sample from the wrong distribution. Fixed at eb1aa9f, deployed e3b5333, verified by opening a frame. SPACE_MODE_MATRIX and SPACE_SESSION_2026-08-25 get a dated invalidation block at the very top, above existing content, per the operator's ruling. House rule honoured: nothing is rewritten, every measurement stands as history, the corrections are dated on top. Each block says the same six things: - Every generation result in the document was produced by a broken transformer. Space shas up to and including ff142d9 ran blanket int8. - MECHANICAL CLAIMS SURVIVE: all eleven modes execute; both fork-boundary directions carry their cargo; determinism and byte-reproducibility hold; tiled decode enables 1216x704x12…898dad2 -
docs: the audit's stale claims annotated — measurements stand, corrections dated on top
Unverified signatureA four-agent audit swept every doc for claims events had falsified (~30, with file:line). Fixed per the house rule: history is annotated, never rewritten — dated blockquote corrections point at current reality with evidence. Six follow-up-table rows get in-cell appends with status flips (the file's own precedent for rows); no finding is deleted, no captured record reworded. The delicate one first. SPACE_MODE_MATRIX §5's 'byte-identical to the result banked the previous day' was never verified: no day-1 hash was ever recorded, the match was file size (39,146 B) plus every decoded statistic to three decimals, and the day-1 file is gone with its host — so byte identity is now unverifiable against it. The §5 caveat ('Do NOT upgrade this') was correct and reports upgraded past it; the error ran in that direction, not the doc's. Corrected claim: identical in size and in all decoded statistics, byte identity UNVERIFIED; a same-seed re-run sha256-compared against the surviving 2026-08-22 arti…fd85c87
-
docs: the ladder gains a rung, added by a failed deploy
Unverified signaturereached -> correct argument -> argument of the right TYPE for the callee. A quantization_config was built, threaded, and proven to reach from_pretrained by a test written for rung 4 -- and the pipeline rejected it, because a bare BitsAndBytesConfig is what a component takes while the pipeline requires PipelineQuantizationConfig. Both arrive equally well. The rung-4 test could not tell them apart and nothing at rung 4 could. Recorded with the shape that makes it useful rather than as another instance: this is the first time we were bitten one step PAST the rung the instrument was built for. An instrument answers the question it was designed for, and the next question is invisible to it precisely because that one looked answered.2eca23e
-
docs: two rules from the day's last two errors
Unverified signatureRule 11, in the lead's words: a number is not evidence for a hypothesis until you check which variable produced it. A -10.4 dBFS run was read as confirming the noise_scale/sigmas[0] diagnosis; the manifest showed the variable that moved was the audio fixture. A schedule change shipped on it. The 2x2 that settled it is recorded because row 1 is the whole answer -- realistic content at noise_scale=0.5 carries the FULL mismatch and is healthy, so content is sufficient and the mismatch is not necessary. The pre-registered single-variable test was consistent with both hypotheses and discriminated neither, which is the same failure as an experiment holding the suspected cause constant, seen from the other side. Rule 12: an advertised lever attached to nothing is not written by careless people. decode_kwargs(generator=...) came WITH a docstring explaining exactly why the decoder must be seeded, and read a key nothing ever set. Written by the person removing other people's inert levers the s…3d4ff9c -
docs: what H17 does not cover, and the four-attribute hand-off
Unverified signatureA.5.1 already said H10/H11 passing does not substitute for the round-trip. The reverse needed saying too: H17 passing does not clear the app's path. It ran through the to_mel= seam with the harness supplying wave.to(device=..., dtype=...), so it stepped over a hand-off the application did not do -- and H10/H11 still failed on CPU-vs-CUDA after H17 was green. A harness that moves a tensor the app does not is the same shape as a float32 fixture over a bfloat16 defect. Two claims, one green result. PLAN rule 10, in host-runner's framing: at a hand-off between our code and a model, every attribute that can disagree eventually will, and fixing them as they surface is not the same as enumerating them. One call site, four attributes, all four wrong in turn -- dtype outward, dtype inward, rank and axis order, device -- each found by running it and each fix correct and incomplete because it addressed the attribute that had crashed. The crashing end announces itself and the quiet end does not…de2596c -
docs: re-attribute H9, record the intervention, and rule 9
Unverified signatureThe handoff said the video_to_audio silence was caused by the strength=0.8 default. It was not, and leaving that in would have been the single most misleading sentence in the document -- it names a lever that is now back at its default and a cause that was withdrawn. The cause was the distilled recipe reaching only the stage-2 path. Corrected, H9 runs at -17.8 dBFS against H8's -15.3, at the unchanged strength of 0.8: one mode 50 dB down and another driven into the rail were the same misconfiguration seen from opposite ends. The wrong diagnosis is kept rather than deleted, because how it went wrong is the useful part: a clean single-lever sweep was believed over a source citation that was correct throughout. Gap 7 records the intervention as a result. All twelve modes moved, with magnitudes ordering by conditioning strength, and the defect cost 3x the compute as well as the quality -- unguided means one forward pass per step instead of two or three, so the flagship 1216x704 with ste…5d73ea4 -
docs: four rules the hardware taught, and the ladder's last rung
Unverified signature9.1.1-9.1.4 came from reading the code. These came from running it, and each cost a real defect. Rule 5, union not intersection: test_encoder_contract drove the REAL encoder through REAL containers to a real file and was still blind, because its fixture used np.float32 where the model produces bfloat16. The path was covered; the dtype was not. Rule 6, a fake that cannot reproduce the failure is not a test of it -- with the sharper form that a fake derived from the mapping can never catch a wrong mapping. Three instances in one day. The fix is a recorded fact about reality, copied from the vendor rather than derived from us, and where a fake stands in for something unavailable locally something must check the fake still behaves like the real thing. Rule 7, checks over comments: twice a constraint failed to survive the next function written by its own author. A comment is a message to a future stranger; make the wrong thing unsayable instead. Rule 8, a refutation from source is only …a7367c9 -
docs: we can prove a function ran, not that it ran with the right arguments
Unverified signatureThe gap this build ends on, and the operator asked for it in the PR body. DISTILLED_SIGMA_VALUES was defined, documented as mandatory, quoted in its module docstring and REFERENCED -- so every check we own passed. It was referenced from a function applied only when stage 2 was enabled. A reference check passes. A module-level reachability check passes. Even the function-level sweep passes unless someone names that particular function. Nothing checked the arguments the pipeline was actually called with. The pattern that works, used three times now, is to record the vendor's specification INDEPENDENTLY and assert against it -- MODEL_CARD_DISTILLED, REAL_AUDIO_VAE_CONFIG_KEYS, REAL_VOCODER_CONFIG_KEYS. Without that independence the test agrees with the code and with nothing else. Stated as a ladder: imported -> reached by the path that does the work -> this function ran -> the arguments were the vendor's. Only the last rung is still hand-built, one call site at a time. Also in this pa…1f07845 -
docs: the refetch fits the timing the packing peak does not
Unverified signatureZEROGPU_DISK 1.1: from_pretrained was refetching 44.33 GB against a complete cache, so a cold Space held 116.52 GB before packing began. That fits the observed death inside 'Loading checkpoint shards 2/4' -- the packed-copy explanation required packing to have started, and packing begins only after from_pretrained returns. Recorded as a candidate cause, not settled: the Space has not been re-run. The peak arithmetic stands as a separate and still-true constraint. local_files_only=True is recorded as NOT the fix. Noting for the record that it was never previously recorded here as one -- the only mentions in the tree are the two written today explaining why it is wrong. ZEROGPU_DISK 1.2: the full repo is 56 files, 164,192,096,189 bytes, read off the cached manifest. It does not fit the 150 GB cap with nothing else on disk at all, so the 37-file allow-list is what makes the Space possible rather than an economy. The excluded 92.01 GB is itemised and reconciles with the manifest total. …77286f0 -
docs: two rules whose shape does not depend on the author being right
Unverified signatureRule 3, the vacuity check: ask of every input whether the run would still pass if it were silently ignored. Where the answer is yes, a green run is not evidence -- five request fields on this build qualify. The rule caught itself: the six comparison runs it produced would all have passed vacuously because seed=-1 draws fresh, established by running it (777437012 vs 3499445304) rather than by reasoning about it. Rule 4, derive every consumer of a recorded fact: rule 1 is a property of copies, not of prose, and code is not exempt. The 37-file allow list was right on a machine nobody wrote it for because both consumers are generated from one recorded source. TEST_PLAN A.4.1 stays authoritative for the fields and the runs; PLAN states the rule once and A.4.1 now points at it, per rule 1.c5ba0bb -
feat: mechanical stream-layout verification; record two prose rules
Unverified signaturePart A's pass criterion is the expected stream layout per mode, which read by eye across sixteen runs is sixteen chances for a wrong-but-plausible pass — in the layer that produced every confidently-wrong artefact here. tests/verify_streams.py takes a mode and a file, resolves expectations from the REGISTRY on every call rather than a snapshot, probes the real container, and reports the actual layout on failure. Three properties it is built around: expectations never copied, so there is no second place the truth can drift; it fails in BOTH directions, including an undeclared stream present, which is the half the exports contract has historically been weak on; and could-not-probe exits 2, distinct from pass and fail, because startup_report once defaulted to 'nothing is missing' and reported 37/37 having checked nothing. Verified on real containers built with av, not on faked probe results: video-only correct, undeclared audio caught, joint correct, missing declared audio caught, mono …609c2a3 -
docs: add the ZeroGPU disk decision document; correct a false reason
Unverified signatureThe precision lever was deleted partly on the claim that the packer's handling of quantised tensor subclasses was unknown. It is not. spaces/zero/torch/bitsandbytes.py is a 43-line version check that no-ops for bnb >= 0.46.0, wired into the packing path at patching.py:328/355/445 — modern bitsandbytes needs no special handling at all. There is no equivalent module for torchao, and that asymmetry is what actually selects the library. The deletion stands, because the lever was wired to nothing, which is reason enough on its own. But getting a decision right for a wrong reason still leaves a wrong record, and this build has had enough of those. Corrected in config.py, model_files.py and PLAN §2.6, and technical risk #2 is resolved rather than open. docs/ZEROGPU_DISK.md is written for the operator choosing between four routes with money and quality at stake, not as a log. Every claim is labelled verified / inferred / assumed, because several earlier confident numbers turned out to have n…0e1b747 -
feat: fail loudly if the tensors stop being mmap-backed
Unverified signatureThe mmap precondition was a comment. It is now a check, because its failure mode is the worst shape this build has: a dtype cast or a clone materialises tensors into anonymous memory, ZeroGPU's post-pack prune finds nothing to unlink, and the Space dies at STEADY STATE with 72 GB it cannot account for and no error explaining why. The guard asserts on the same basis the packer prunes on rather than a proxy for it: walk /proc/self/maps, resolve each parameter's data_ptr back to its backing file, and require a match against the autoprune glob. File-backed is not sufficient — a tensor mapped from outside the pattern is just as unfreeable and looks fine to any check that only asks whether it is backed at all. Being unable to check is reported, never passed. There is no /proc/self/maps on macOS, and a quiet pass there would be the same defect as the startup report that defaulted to 37/37. Parsing and address resolution are pure functions, so the whole thing is testable offline with a synt…6fd68ea -
docs: reconcile the disk arithmetic; the peak precedes the prune
Unverified signaturePLAN contradicted itself on the fact everything else hangs on. §2.5 stated the auto-prune as known and read the margin as tight; risk #1 stated the same prune as unverified and the margin as negative. Two sections disagreeing is worse than either being wrong, because whichever one the next reader lands on is the one they believe. Settled in favour of the worse answer, verified against spaces 0.51.1: the prune is real, and the unlink runs strictly after _pack() returns, so it cannot prevent the peak it would have to prevent. §2.5's old '6 GB of margin is not much' never counted the image, which is not optional and is resident for the whole window. With it the total is 152-156 GB against a 150 GB cap. The margin is negative, not tight. Risk #1 is no longer a risk. It is a finding: bf16 on storage: None is structurally infeasible for this checkpoint set, and the three routes out are all operator decisions rather than knobs. Applies the two rules that would have prevented it. §2.5 is no…29e399d -
docs: the dtype is load-bearing for ZeroGPU's snapshot cleanup
Unverified signatureZeroGPU frees the 72 GB HF snapshot after packing by reading /proc/self/maps, mapping each tensor's data_ptr back to the blob it is mmap'd from, and unlinking those. So the cleanup is conditional on the tensors still BEING mmap'd — a dtype cast, a clone, an eager load or a non-safetensors path materialises them into anonymous memory and there is nothing left to capture. Then nothing is freed, 72 GB stays on disk forever, and there is no error and no log line saying so. torch_dtype has therefore stopped being a free parameter. The comment sits at the risk site in get_pipeline rather than only in the plan, because the next person to touch that line is the one who needs it and they will not be reading §8.3 at the time. Same class: the cache must stay under ~/.cache/huggingface/hub or the autoprune pattern stops matching. That is a sharper reason not to move HF_HOME than the preload one already recorded — it fails silently rather than loudly. TEST_PLAN corrections against the platform's…92aea11 -
docs: split the test plan by the question each host can answer
Unverified signatureTEST_PLAN was one matrix shaped entirely by a 2400 GPU-s/day budget that does not apply on hinode. Now two, with a table at the top making the difference structural rather than remembered: Part A asks whether the application is correct, Part B asks whether it fits ZeroGPU, and neither answers the other's question. Part A is sixteen runs covering all eleven modes at resolutions someone would actually use, plus LoRA in both directions, stage 2 and the duration head. The ZeroGPU matrix deliberately proved code paths and skipped modes whose paths were already covered; without a quota the right thing to buy is output we can look at, not more green ticks. So the run log has a required 'Output looks right?' column — every failure this build produced was something that ran and was wrong. A.0 lists the startup checks that must pass before any of it means anything, including module identity across the two pipelines, since equality passes with two copies loaded. A.2 records what hinode cannot t…66ca545 -
docs: the single-load fix is necessary, not sufficient
Unverified signatureMy previous note implied fixing the double load addressed the 150 GB ceiling. It does not, and writing that would have been the same confidently-wrong statement as the expected_weights=53.20GB line I deleted two commits ago — a plausible number pointing the next decision in the wrong direction. The packed copy is separate from the HF snapshot, so even the fixed single-pipeline case is 72.19 + 72.19 = 144.37 GB, leaving 5.63 GB for an image that needs 8-12. One pipeline does not fit either. The eviction was over-determined by three independent causes and the smallest absorbed two cycles. Names the unknown that decides it rather than assuming past it: whether ZeroGPU frees the snapshot after packing. If it does not, bf16 on storage: None is structurally infeasible for this checkpoint set, and the routes are persistent storage, a public Space with preload, or smaller checkpoints — an operator decision. Also records that the double load is settled by inspection: pipeline_utils has no mo…f51840e -
fix: load the model once; derive the second workflow from it
Unverified signatureapp.py built both workflows and each ran from_pretrained into its own cache slot, so we materialised two complete independent copies of the model — roughly 67 GiB each. On ZeroGPU that is ~134 GiB packed to disk against a 150 GB cap, with packing running after both builds, so it blows the ceiling even when the download fits comfortably. On a 96 GiB card it is an outright OOM on the second build. At the pinned revision the two pipeline classes take identical weight-bearing components; the only difference is audio_scheduler, which carries no weights and defaults to None. There is no configuration in which two copies are wanted, which is why this landed before the measurement confirming its magnitude — the fix is correct in every branch, unlike a knob chosen on a diagnosis that might be wrong. The base is always the superset class. Deriving superset to subset can only drop audio_scheduler, which the subset does not accept; the reverse would drop it from the pipeline that uses it, degrad…43f432f -
fix: no advertised lever may be attached to nothing
Unverified signatureAudited all six env vars the orchestrator was told were no-push escape hatches. FOUR were inert, not one: LTX25_TRANSFORMER_PRECISION inert -> DELETED LTX25_TEXT_ENCODER_PRECISION inert -> DELETED LTX25_GPU_DURATION_SECONDS inert -> WIRED LTX25_LORA_MAX_GB inert -> WIRED LTX25_ZEROGPU_SIZE worked LTX25_OUTPUT_RETENTION_GB worked LTX25_OUTPUT_DIR worked The precision pair is worse than the eight defects before it. Those failed silently; this one spoke — expected_weights_gb applied the precision factor while the loader ignored it, so the startup report stated 53.20 GB for a load that was still 72.19 GB, wrong in the direction that misdirects the next decision. Deleted rather than half-wired: wiring needs bitsandbytes or torchao, neither pinned, neither verifiable without a GPU, both using tensor subclasses whose interaction with ZeroGPU's packer is unknown. estimate_duration read the module constant instead of the config, so …9829d39 -
fix: download weights at runtime; a private Space cannot preload
Unverified signatureThe build succeeded and the preload init step failed with 401 on a gated file. 401 is unauthenticated, not unauthorised: the preload runs as a separate platform-managed step that does not receive Space secrets, so the request arrives with no credentials however valid the token is. Verified both ways — the public 2.3 Space preloads a gated=manual repo fine, this private one 401s on gated=auto, and the same URL fetches 200 by hand. preflight.ensure_weights() fetches the 37 files inside the app, where the token exists, from the same download_allow_list() so the single-source property is unchanged. It logs every file, because a 72 GB cold start that goes quiet for forty minutes is indistinguishable from one that has stopped, and it fails naming the file, its position, the underlying error class and its URL. This is the better design even if that diagnosis is wrong, which is why it beat making the Space public: it removes the ownership bug class outright. Preloaded files belong to the bui…f355817 -
docs: warn the next reader not to trust the gate the way we did
Unverified signaturePuts the load-bearing fact where someone actually lands rather than in section 9: this suite was green at 1102 passing tests while five blockers were live, with the app unable to reach a GPU, unable to apply a LoRA, and reporting successful runs as internal errors. Nothing was failing. Anyone opening this repo in six months sees ~1800 passing tests and a careful-looking plan. The single most useful thing we can tell them is that this same suite, minus its seam and reachability layers, was green while the app did not work. Also corrects the status line, which still claimed implementation had not started.86ce43f -
docs: name the common cause behind the five blockers
Unverified signatureThey were one mistake with five faces. Everyone verified the thing they had just built, with a check they designed, against the mental model they already held — unit specs, hand-written probes, AST scans, sys.modules checks. All of them tested a belief rather than the system. Records the number that makes it concrete: the gate was green at 1102 tests with all five blockers in the tree, and the ~700 added since are almost entirely seams and reachability rather than more unit coverage. More tests of the same shape would have found none of them.d99f134 -
docs: record what this build actually taught
Unverified signatureNew §9.1.1: a module can be complete, documented, specced and green and still be connected to nothing — and it can be imported and still be connected to nothing that matters. The corollary took five blockers, all found after the gate first went green, and the table names each one. #5 is the one to remember: nine validation stages built to stop silent garbage, silently doing nothing, with 19 passing specs saying otherwise. Records why the usual checks miss the class — a unit test proves a module correct in isolation, which is exactly what a disconnected module is, and a probe tests the mental model you already have — and the one check that does not depend on the checker already suspecting the answer: import app, evict what it loaded, run a real generation, diff again. §9.1.2 records the misdescribing-history incident as a process finding: git log for outputs.py returns a commit about other files, because a directory-wide git add swept up another agent's uncommitted work. Three agents …118d513 -
docs: batched corrections pass
Unverified signatureThe official Lightricks adapter is the exception for carrying model_version, not the template — a validator written from it as the reference would have been written from the one file in the population unrepresentative of the rest. Flipping precision is a rebuild, not a runtime switch: preloading is all-or-nothing, so a quantised variant needing different files cannot be fetched on demand into a build-user-owned cache. Setting the variable alone only changes how the existing weights are loaded. Records the startup report verbatim, with the cache count called out as the load-bearing line — 36/37 with the first missing file named is what makes 'preload worked' and 'preload silently did nothing' two different sentences. Retention now documents both call sites and why they are non-fatal. A policy that exists but is never called prevents nothing, which was literally true here until the call sites were added. scheduler gets its own row, noting it needs scheduler_config.json rather than co…3f740e9 -
feat: preflight cache-completeness check; retract HF_HOME advice
Unverified signatureMoving HF_HOME to dodge the read-only preload problem would have caused the failure it was meant to prevent: preload_from_hub writes to the default cache and does not follow a moved HF_HOME, so the app would look somewhere empty and re-download 72 GB on top of the preloaded copy. Plan corrected, with the upstream warning quoted. The real fix is an invariant rather than a mitigation. If every file we load is in the preload set, nothing is fetched lazily and the permission error cannot arise — so preflight now verifies every allow-list path is present in the cache and fails naming the file and its URL, instead of letting a lazy download surface as Permission denied halfway through a generation. Also records the preload revision field and the 1h startup timeout.b17d455 -
feat: env-driven config; 9-stage LoRA validator design
Unverified signatureThe validator gains two stages, because seven were not sufficient. An LTX-Video-2B adapter passes stage 5 (diffusion_model. prefix), stage 6 (clean lora_A/lora_B pairs), stage 4 (empty metadata, which our own correct absent-is-not-fatal rule permits), diffusers' only built-in check, and can pass stage 7 as well because its module names match ours. The single thing wrong with it is a 2048 hidden dim where this model is 4096. Stage 6.5 compares per module, not against one expected hidden dim. The transformer carries two streams — video at 4096 and audio at 2048, each with a 4x feed-forward expansion — so a naive 4096 rule would reject the most-downloaded LTX LoRA on the Hub. 'Contains 2048' is not a signal; '2048 at a module that expects 4096' is. Stage 5.5 detects IC-LoRAs from reference_downscale_factor, which otherwise neither error nor work on the plain pipeline. config.py resolves precision, retention, duration and LoRA size bounds from environment variables with loud validation, …5516933 -
feat: implement errors, constraints and model_files
Unverified signatureInnermost three modules against the test teammate's specs. 416 of 417 of their assertions for these modules pass; the one failure is a data disagreement reported separately, not a defect here. errors.py: single root plus four grouping bases, eighteen concrete errors that each take keyword arguments naming the offending thing and render a sentence containing them. The predecessor's week-long outage came from four layers each turning a failure into a falsy value, so an error that does not name the file, the value, the URL or the prefixes it found is treated as a bug. constraints.py: validate_* raises, snap_* fixes, never both. The temporal compression ratio is a parameter with a documented fallback of 8, not a literal — upstream silently floors a bad num_frames rather than raising, and at ratio 8 every value from 113 to 120 collapses to the same latent count with no warning. model_files.py: byte-exact sizes and live-shard file lists read from the repo's own index files, so the transfo…706048e -
docs: fold in gated-repo evidence; add GPU test plan
Unverified signatureThe LTX-2.5-Diffusers gate is open, so several things the plan was reasoning about are now settled by the real configs. model_index.json confirms _diffusers_version 0.40.0.dev0 (the SHA pin is not a guess), text_encoder is Gemma4UnifiedForConditionalGeneration (so CRF provably takes the gemma4 branch, 18), vocoder is LTX2VocoderWithBWE rather than plain LTX2Vocoder, and prompt_enhancer and processor ship as [None, None] — the repo itself declares the enhancer absent, confirming the no-dead-toggle decision. Trap A is now measured rather than inferred: the transformer's 4-shard set and the connectors' 2-shard set are the live ones, so the single 6.344 GB connectors file is the dead one — the opposite of the naive guess. Shipped subset is 33 files, 72.186 GB, 67.23 GiB. latent_upsampler and temporal_latent_upsampler are not in model_index.json and load by subfolder, so model_files.py must model two loading mechanisms; a resolver assuming one silently misses the two components the stage…ca6e2de -
docs: fold in ZeroGPU recon; add pinned requirements
Unverified signaturePLAN.md rewritten against corrected recon. The substantive reversals: - ZeroGPU is RTX PRO 6000 Blackwell via MIG, not an A10G and not an H200 — large is 48 GB, xlarge is 96 GB, and the API's 'zero-a10g' string is a stale identifier. VRAM was never the constraint. - enable_model_cpu_offload() must NOT be used. It is counterproductive on ZeroGPU: anything accelerate parks on CPU is never registered for packing, so it stays in main-process RAM, is fork-inherited by every worker, and is re-copied host to device on every hook fire. The platform's own O_DIRECT disk packer replaces it. Section 8 rewritten around import order, module-scope .to('cuda'), static @spaces.GPU, and the never-return-CUDA-tensors rule. - CPU RAM (~68 GB) and transient disk (~144 GB peak against a 150 GB cap) are the real ceilings. bf16 ships as the default with a loud preflight; quantization is a one-line fallback switch, not the default, because ZeroGPU's packer handling of quantised tensor …bfaee84 -
docs: add LTX-2.5 Studio design plan
Unverified signatureLocks the six design decisions: diffusers integration path, 72.2 GB disk subset of the 164.19 GB Diffusers pack, the 11-mode registry, the seven-stage LoRA validator, the resilience doctrine, and the private ZeroGPU Space deploy shape. Records the gated-repo blocker and six open questions, including a blocking conflict between pinning diffusers to a release and shipping duration_head / diffusion_decoder, which only exist on main.1324740