Spaces:
Running on Zero
Running on Zero
Commit History
2 days ago
-
fix: the decoder choices are named by what they do: "Auto", "Steadier (diffusion decoder)" and "Sharper (standard decoder)", in the choice, its line, the Upscale lock line, the Done card and the backend's refusals; the values sent stay auto, diffusion and conv (#27)β
Unverified signatureOn the Space the same seed and prompt through both decoders gave the same scene: the diffusion decoder a little softer and steadier from frame to frame (fine detail 10.3 against 11.6, shimmer 1.47 against 1.88), the standard decoder sharper frame by frame with about 28% more shimmer (evidence/space/matrix/ab_decoder). The page called the softer one "Best quality" and the sharper one "Fast". Now the choices read "Auto" (still the vendor's default), "Steadier (diffusion decoder)" ("A little softer, steadier from frame to frame; takes a few seconds longer.") and "Sharper (standard decoder)" ("Sharper frame by frame, with a little more shimmer; faster."). Auto's line is true on every machine: "Diffusion decoder on an NVIDIA GPU that can run it, standard decoder elsewhere." Off CUDA, Auto always takes the standard decoder (pipelines.StageRunner._single_stage_decoder), so the old "where this machine can run it" was untrue on a Mac, and round 4 asked for the NVIDIA wording. The Upscale line β¦1c8a58d -
fix: the GPU reservation follows each run's size and length, from the Space's measurements with a 1.3 margin, so a default run needs 120 s of quota to start instead of 360, and on the Space a run past the measured envelope is refused before any GPU time (#21)β
Unverified signatureEvery run reserved 180 s (LTX25_GPU_DURATION_SECONDS) x1.5 with Upscale, +60 s with an adapter and +180 s with Enhance, whatever it rendered; ZeroGPU wants twice that free on xlarge, so a visitor with less than 360 s of quota could not start a 25-second default run, and nothing stopped a run too large to finish inside its reservation. zerogpu.estimate_duration now reads the request as the run will use it (a sound-only mode's pinned 256 x 256, no Upscale where the mode skips it, the duration head's longest pick, the length a recording or a clip set at the gate and the longest the app allows until it has, the tail of the schedule a clip variation runs) and models its window from the Space's measurements (six deploy-2 runs up to 1920 x 1088 at 10 s and the 24 deploy-1 calls): 25 s before the first step, steps x t(N1) with t(N) = 0.46 + 1.81e-4 N + 1.94e-9 N^2, then 1.0 s + 1.3e-7 s per returned pixel-frame; Upscale adds 3 s + 3 x t(N2) and its larger post; Enhance 60 s; any adapters 30 sβ¦ac68de5 -
fix: no entry can sample on an MPS PyTorch that fails the numerics self-test: build_local_runner applies the app's gate on mps, so scripts/local_generate.py --device mps exits refused (2) before loading anything, and the remedy names the README's floor, PyTorch 2.14 (#11)β
Unverified signaturetorch 2.11 on MPS reuses the first scale scaled-dot-product attention saw, which gives wrong numbers without any error. The app's pipeline factory refused such a build through device.require_mps_numerics, but local_runtime.build_local_runner never called it, and scripts/local_generate.py builds the runner directly: on an affected torch it would have produced a wrong video. build_local_runner now calls require_mps_numerics on mps, after the watermark and availability checks and before any file is looked at, so every entry is covered by one check; LTX25_MPS_SELF_TEST=warn still lets an operator run unverified. The script counts MpsNumericsError among its refusals (exit 2). MPS_CLEAN_TORCH said 2.15 while the README states PyTorch 2.14 or newer (2.14.1 measured the same as 2.15.0 in the numerics checks), so the refusal and the unverified note now say 2.14, as the README does. Tests: the script test begun on wip/deploy2-w14 (ca91f07), unchanged; a runner-level test under enforce and warnβ¦444bc2b
3 days ago
-
fix: the Space harness judges what a mode promises over the files a call returns, so a video-with-sound run passes with its mp4 carrying both tracks and the wav beside it judged as the sound on its ownβ
Unverified signatureSince 20155a1 (F30) a mode that exports video and audio returns two files: the mp4 with both tracks, and output.wav, the same sound on its own. The harness still judged every returned file against the mode's whole export list (verify_streams.verify(mode, file) for each file), as when the mp4 came back in both outputs, so the wav "lacked" a video stream. On 2026-11-16 the Space returned m05_text_to_audio_video correctly (h264 704x480, 121 frames, AAC stereo 48 kHz, plus the wav), the harness called it a MISMATCH, and the plan stopped. Every call that returns video with sound would have done the same: modes 05, 06, 07 and 11, both Upscale calls, the enhancer, the model-chosen length and the operator's audio. What each output holds now comes from the registry (returned_files, from the mode's exports). The video output must carry every track the mode exports and is judged as before, strictly, with the geometry. Beside it, a mode that also exports audio returns its sound on its own. That fβ¦a40a311 -
fix: the local script leaves the change level to the mode and writes the run record into its JSON, and the record names the device that ranβ
Unverified signature- scripts/local_generate.py: --noise-scale defaults to None, so a run without it varies at the mode's own level. It was 0.5, which snapped video to video to "a little" (0.42) instead of its 0.91, as the lead measured. --strength's help says no mode reads it. The run record (result.record as a dict) is written into the --json file beside the artifacts. - generate.RunRecord carries the device the run ran on (record.device): the runner's own device, else this process's resolution. The UI already reads it to label the run's time. - zerogpu.assert_returnable no longer takes a plain string device for a CUDA tensor: a torch object's device is a torch.device, never a str. Without this the record's "cuda" made every result through the Space entrypoint refuse itself (the suite caught it through app.py's run_generation). tests/test_local_generate.py drives the script through its parser and main() with a fake runner that writes real frames: no flag gives 0.909375, the flag stillβ¦419e0a5 -
fix: the Space harness measures an audio track's onsets and band levels, so noise at a normal level no longer reads as soundβ
Unverified signatureEach audio track was measured by its level, peak, clipping, silence, spectral flatness, centroid and band shares. Noise at -19 dBFS has a perfectly ordinary level, so the one figure people read first said nothing was wrong. Two more measurements sit beside it now, numpy only: an onset count (rises of the 10 ms level by 8 dB or more over the lowest level of the preceding 50 ms, at least 100 ms apart; steady noise has none, speech, hits and chimes many) and each band's energy as a level in dBFS. The line printed per call shows flatness, onsets and band shares with the level. They are measurements, never verdicts: only digital silence fails a call. The test puts white noise and a steady tone at the same -19 dBFS beside twenty 120 ms tone bursts: the noise has flatness above 0.8, no onsets and three quarters of its energy above 6 kHz; the tone has flatness below 0.05 and no onsets; the bursts have exactly twenty.fee4645 -
fix: the Space harness gates a chained input when its file exists, runs the app's gate as the Space does, and can carry the lead's matrix over several daysβ
Unverified signatureTwo harness tests were red. The dry run handed generate.prepare_request a made-up path for an input an earlier call produces, and since the app opens every video before dispatch that path was refused: the harness's validation was wrong. A call with a chained input is now marked "checked when it exists" in the dry run, and the app's gate runs on the real file, before anything is sent, once the producer has run; every call whose inputs exist still goes through prepare_request, and a bad real file is still refused offline. The example plan test wrote 0.34 s of audio for every wav, so any set length was refused: its fixture now has the lengths, rates and channels of the lead's files, and the plan was stale too. The gate runs as the Space runs it. Stage 2 and the diffusion decoder are refused off CUDA, so on a machine with LTX25_DEVICE=cpu or mps the gate would have applied that machine's limits to a Space plan; device.current() now answers with the Space's own resolution (ZeroGPU, CUDA) wβ¦6ae3c5a -
chore: the hinode scripts go with the host: nothing in the repo runs, imports or globs themβ
Unverified signaturehinode, the remote GPU box these were written for, no longer exists. Each of the twelve files names `./scripts/hinode_run.sh` or "Run on hinode" as its only invocation and defaults to `/home/wakeuser` paths. A `git grep` of every stem at 2f356b5 finds them in no test, no kept script, no CI (there is none), not in pyproject.toml or scripts/gate.sh; the remaining mentions are the historical docs, which keep their bodies, and two prose mentions in files other owners hold (generate.py:1425, tests/verify_streams.py:10). host_evidence.py and host_audio_levels.py need only av, numpy and PIL and would run anywhere, but nothing uses them either; `git show 2f356b5:scripts/<name>` restores any of these. The two ruff F401s once reported in host_lora.py and host_mel_roundtrip.py were already gone at b438b1f.9f61032 -
fix: an explicit diffusion decode, or stage 2, that the decoder cannot take is refused before dispatchβ
Unverified signatureBoth facts are known before a GPU is reserved: the decoder's minimum size (read from its config file, the same one the run will load) and, off CUDA, the size its CPU path can hold. prepare_request now checks them, so the user is told "refused before running, nothing was billed" (gpu_used False) and the UI's readiness check can disable Generate with the reason. auto is never refused; it falls back to conv with a note as before. The runner keeps the same checks against the loaded decoder; the two share pipelines.check_decoder_minimum and pipelines.check_cpu_decoder_size. The tiny stack used by the real-pipeline tests now has its own composite snapshot, and its runs point LTX25_MODEL_SNAPSHOT_DIR at it, so the gate reads the tiny decoder's config rather than whatever real checkpoint the HF cache holds.18a60f4 -
fix: the Space harness records the app's new picture check as it returns it, and sends the real decoder field by nameβ
Unverified signatureThe harness's one picture-check wrapper now calls outputs.assess_picture and records its PictureCheck verbatim in verify.json: the measurements, the conditions that fired (flat, veiled, static) and the notes. It still holds none of the check's thresholds; a fired condition is shown in the summary and never fails a call. decoder is now a real appended field, so the appended-field test uses a hypothetical key for the pass-through and sends decoder="conv" by name beside it. The picture tests compare verify.json with assess_picture's own result and pin that the script calls it from one place only.a63770b -
feat: the decoder is a per-run choice, and auto decodes with the model's diffusion decoder wherever it can runβ
Unverified signatureSingle-stage runs returned the pipeline's own convolutional VAE decode; the diffusion decoder, which the vendor calls the native default from LTX-2.5 on, only ever ran after stage 2. The decode policy is auto | diffusion | conv, from a new optional request field (GenerationRequest.decoder, appended last), else LTX25_DECODER, default auto: - auto is the diffusion decoder on CUDA when its kernel processor builds and the video is at least the decoder's minimum (224x224x17 for the pinned config); otherwise conv, and the run notes say why. - an explicit diffusion that cannot run raises DecoderUnavailableError (cannot load, or off CUDA beyond the CPU decoder's size limit) or DecoderTooSmallError naming the smallest request; it is never downgraded quietly. - conv stays reachable for A/B; stage 2 always finishes with the diffusion decoder; the modes that discard their video always use conv. A single-stage diffusion decode asks for latents, decodes them with the existing LTX2VideoDiβ¦174bb90 -
feat: the Space harness runs a plan of calls by field name inside a hard daily GPU budget, and keeps the evidence of every callβ
Unverified signaturescripts/space_matrix.py could only loop the eleven modes at 256x256x9 with one prompt and six settable fields, could not apply a LoRA, fed each conditioned mode the harness's own earlier output (so "follows its input" proved nothing), priced an enhanced call at half what the app reserves, kept its ledger inside whichever checkout ran it, and never saw the queue's progress events. A plan (--plan FILE) is a list of calls: a label, a mode, any /on_generate field by its key (a field appended to ui/fields.py is passed through once the spec has it), media as files under --assets-dir, optional LoRAs and an expectation. An input comes from an earlier call only when the plan says {"from_call": ...}, and the record says "chained". --plan default is the old loop with its chains written out, --mode builds one call from flags, --only picks labels. Every call is checked offline by the app's own generate.prepare_request (limits, required inputs, the length an audio mode takes from its audio) and agaβ¦d043b52 -
fix: stage 2 runs for every mode that offers it, and a video too small for the decoder is refused before stage 1β
Unverified signatureThe condition pipeline builds its conditioning mask from height/width (512x768 by default) even when it is handed latents, so every image- and video-conditioned mode failed stage 2 with "Provided latents tensor has shape ...". Stage 2 is now given the upsampled latents' own size (latent H, W x the VAE's spatial ratio) and a frame count read off the latents, which also covers the duration head, where the request has none. text2video keeps upstream's recipe of no size. audio_to_video failed stage 2 with "got multiple values for keyword argument 'noise_scale'": the request already carried noise_scale and audio_latents and the stage-2 kwargs added both. Each call is now built by merging dicts, stage 2's own values winning. The diffusion decoder's neighbourhood windows need a minimum grid (224 px per side and 17 frames for the pinned config). The minimum is computed from the loaded decoder's kernels and checked before stage 1, raising DecoderTooSmallError with the closest request that worβ¦d2490e1 -
fix: refuse an MPS watermark pair that makes every empty_cache() fail, before anything loadsβ
Unverified signatureWith PYTORCH_MPS_HIGH_WATERMARK_RATIO below 1.4 and the low ratio unset, torch keeps its 1.4 low default and torch.mps.empty_cache() raises 'invalid low watermark ratio 1.4' β inside from_single_file and in the runner's own staging. memory.mps_watermarks() reads both ratios and names the problem and the fix; a high ratio without a low one at or below it, or a value that is not a number, is refused with a typed MpsWatermarkError. build_local_runner checks it on mps; scripts/local_generate.py prints both values and torch's recommended MPS maximum in the plan (dry run too) and exits 2 before touching anything. Nothing sets the variables.adad9b6 -
fix: the local runner gives model-file pages back β the committed gauge no longer counts weights twiceβ
Unverified signatureMeasured by the lead on real weights: after a component moved to MPS, its memory-mapped safetensors pages stayed on the active queue for minutes, and active + wired + compressor (the gauge we keep) counted them, so G0 peaked ~22 GB over budget. Evicting them with mmap + msync(MS_INVALIDATE) dropped committed by the file's size in a fraction of a second. - memory.release_file_cache(paths): darwin msync(MS_INVALIDATE) per file, linux posix_fadvise(DONTNEED); resolves the HF cache's symlinks, walks directories, counts each file once, never raises. Data on disk untouched. - The runner moves every safetensors component (the small set, the text encoder, the transformer) tensor by tensor and releases the files' pages every ~2 GB, instead of one .to() that holds file pages plus the device copy; one release at the end, one line logged per release. The GGUF file is released after load_gguf_checkpoint (everything is copied by then) and after the move. Releasing the text encoder or β¦75d0c53 -
test: the real-pipeline tier β every mode through the app's own generate() on a tiny random checkpoint, on the CPU and on MPSβ
Unverified signatureThe offline suite only ever exercised fakes. It was green at 1102 tests while five blockers were live. `scripts/local_smoke.py` runs the pinned diffusers LTX-2.5 classes on optimum-intel-internal-testing/tiny-random-ltx2.5 (~95 MB, pinned revision, fetched only with --fetch). It goes through `import app`, `generate()`, the GPU entrypoint and `get_pipeline` on the resolved device: - every mode, with its required media made on the fly, at <=128 px and <=17 frames; - the duration head; - prompt enhancement with the tiny enhancer through the app's own `enhance()`; - a rank-4 LoRA on the tiny transformer's real attention Linears (12 modules attached); - stage 2 with tiny random upsampler and decoder models built from the real configs; - the typed refusal of a too-large stage 2 off CUDA; - on MPS, the typed self-test refusal under `enforce`. Each case reports PASS/FAIL, the exception, the artifact and verify_streams' verdict. The script prints a table, writes JSON, and exits non-zero on any β¦45cef14 -
feat: scripts/local_generate.py β one local generation through the app's own generate(), with its memory storyβ
Unverified signatureThe lead's entry point for real-weights runs on the Mac. The local runner is injected through generate()'s pipeline-factory seam, so it works before pipelines.get_pipeline knows about local devices. It prints the resolved plan, every preflight line, per-phase wall time, the peak committed memory (vm_stat, sampled every 0.5 s) and MPS driver memory, the files written and verify_streams' verdict, and writes the same with --json. --dry-run prints the plan and the projected preflight arithmetic and loads nothing. Nothing is downloaded: a missing file is reported with the exact hf download command. Exit codes: 0 ok, 2 refused (memory, validation, missing files), 3 error. The runner now attaches the connectors after the pipeline load (passed in, diffusers logs the whole module), and the frame bound used for an auto-duration estimate is public for the dry run.af26f09
4 days ago
-
test: `scripts/gate.sh` β the gate says which environment its verdict belongs toβ
Unverified signatureThe suite was green for its whole recorded history in torch-less venvs. In the environment README prescribes it was 34 failed, 20 errors (fixed in 1c404fd). Nothing in a result told the two apart. The gate now does: PYTHON=.venv/bin/python scripts/gate.sh * It prints the interpreter and Python version, and the torch version or "NOT importable", before it runs anything. * It runs `pytest -q -p no:cacheprovider`, then `ruff check --no-cache .` (the whole tree, unscoped) if ruff imports in that env. If ruff is missing it says lint was NOT checked; it never passes silently. * Exit 1 on any failure and 2 if `PYTHON` cannot run. The default is `python3` because a bare `python` is not on every machine. * It sets PYTHONDONTWRITEBYTECODE and disables both caches, so a run leaves no untracked files. README's local-run section now says the tests are meant for the venv with requirements.txt installed. A torch-less venv also passes, but it skips the checks against real torch, torchβ¦c51fc56 -
chore: drop the two unused imports that were the whole of `ruff check .`'s outputβ
Unverified signature`ruff check .` reported exactly two errors, both F401, and nothing else in the repo. The next commit makes ruff part of the gate, so they go first: * `scripts/host_lora.py:17`: `import os`. The name is used nowhere in the file. * `scripts/host_mel_roundtrip.py:191`: a function-local `import torch` in `ab_transpose`, which never uses `torch`. The file's module-level `import torch` (:33) is untouched and still serves `main`. One line each, no other edit, no behaviour change.b438b1f
-
feat: prompt enhancement ships β a real toggle, in the web process, at zero GPU costβ
Unverified signatureThe build refused this twice, and both reasons were wrong. β "The pack ships it as absent, so `enable_prompt_enhancement=True` fails on our subset." `model_index.json` does carry `prompt_enhancer: [None, None]`, and that is not what it means. Upstream types the component `Gemma4ForConditionalGeneration | None` (pipeline_ltx2.py:249) and lists it in `_optional_components` (:235) β a plain transformers class. The empty slot is there because the component is CONSTRUCTED, not resolved from the snapshot. So it is an ordinary runtime `from_pretrained` into the HF cache, and nothing in model_files.py / preflight.py / preload_from_hub needs to know it exists. β "ZeroGPU charges it twice at the packing peak, ~20.6 GB." A placement artefact, not a property of the model. Packing runs once per container, at `demo.launch()`, over tensors registered while the module scope ran. A model loaded lazily inside a request handler, on the CPU, is in neither window. It is charged once, as a download. β Anβ¦2f36d8a
-
fix: motion is reported, never refused β the temporal gate demoted after it destroyed a real renderβ
Unverified signatureMIN_MEAN_FRAME_DELTA = 0.5 refused image_to_audio_video at 512x960x121: structure 0.9974, high-freq 0.0559, block std 81.6, luma band 238.0 β a coherent photographic scene, a red balloon over a lake at dawn β thrown away for mean |Ξ| 0.4071. It is near-static because it is conditioned on a still image of that scene, which is the mode working as designed. The gate never had the margin it appeared to have. Measured with the runner's own decoder on the post-int8-fix m2 runs, the only banked artifacts whose frames have been looked at and are photographs: 0.3859, 0.4071, 0.4168 β against the operator's colour static at 0.403. A legitimate render sits BELOW the failure the threshold was calibrated against. The two populations overlap, and no threshold on this axis separates them, because a temporal measure cannot tell a still scene from a dead one: both measure near zero, and what distinguishes them lives inside the frames rather than between them. So mean |Ξ| WARNS. It is still measured, β¦8c042d4 -
feat: measure whether there is a picture in the frames β structure, not distinctnessβ
Unverified signatureEvery content check this project had could be passed by structureless garbage, and one artifact proved it: 121 of 121 distinct frames, the geometry the request asked for, non-zero bytes, a full luma range β and it was colour static. space_matrix scored artifacts like it as PASS, and the app inspected the frames not at all (generate.py:597 checks only frames-non-empty; generate.py:690 and outputs.py check only that a path exists and is not 0 bytes). The reason no arrangement of those numbers could ever have worked is that distinctness, byte count, geometry and brightness range are properties of NOISE. Noise has all of them in abundance. What noise does not have is structure at a scale above one pixel. Two ratios say it, both divided by the frame's own luma std so they describe shape rather than contrast β a dim shot and a bright shot of the same scene land in the same place: SR = std(8x8 box means) / std(pixels) real 0.918..0.994, failure 0.530 HF = std(laplacian4) / std(pβ¦e54477c
-
docs: 11 of 11 on the Space, and what that still does not claimβ
Unverified signatureThe two video-INPUT modes were re-run at Space sha 287ba9c carrying d2eb6f2 and both pass, so follow-up #1 closes: 19.95s / 63,438 B / -19.5 dBFS for video_to_audio, 13.67s / 33,812 B / 9 distinct frames / mean |d| 17.277 for video_to_video. Both land inside the 13.67-21.59s band every successful generation occupies, against the 3.53-9.13s the failures took. The uint8 choice is recorded as the part that was load-bearing on the metered side rather than locally: upstream re-compresses a single-frame condition at the model CRF and rejects a non-uint8 array while doing it, so a float payload would have passed the offline gate and failed only on the Space. Fixture provenance is recorded with the result, because without it the numbers prove much less. The runner chains fixtures forward, and by the end of the matrix video.mp4 had drifted to audio_to_video's near-static 7,619 B output at mean |d| 0.905. Conditioning on that would have been unreadable in either direction. The fixture was rebuβ¦17b589c -
docs: the fork-boundary fix is a pair, and half of it is a regressionβ
Unverified signatureThree corrections to SPACE_MODE_MATRIX.md, plus four stale claims elsewhere. The fix for the fork boundary was recorded as one method: a `__reduce__` on LTXStudioError. Reading `spaces` rather than trying it shows that is wrong in both directions, and neither half works alone. `__reduce__` alone does nothing. ExceptionResult (types.py:28-31) carries only `traceback`, `error_cls` and `gradio_error`, and `exception_result` populates the third only when `isinstance(exc, gr.Error)`. Our exception object is therefore never transmitted at all, nothing is ever pickled, and `__reduce__` is never consulted. That also settles the open question of whether restoring picklability would restore the message: it would not, and no live run is needed to know it. A `gr.Error` subclass alone is worse than the status quo. It makes the object eligible for `gradio_error`, which must then cross a multiprocessing.Queue that our classes cannot pickle onto. `spaces` anticipates this at wrappers.py:272-275 andβ¦f927c4e -
fix: the quota ceiling only ever saw its own invocationβ
Unverified signatureThe cumulative counter lived entirely in one process's memory, so every invocation started at zero and every invocation independently announced that it fit. Five real invocations reported 43.1, 88.8, 97.0, 16.2 and 100.9 GPU-s and none of them knew the others had happened; five at 1900 each would each have printed FITS while spending 9500 of a 2400/day budget. That is the failure class this project treats as the worse one. An inert LEVER merely does nothing; an inert REPORT tells the operator they are safe. PersistentQuota is a JSON ledger keyed to the UTC day, at a FIXED path under outputs/ (already gitignored). Fixed deliberately: putting it under --fixture-dir or --download-dir would reintroduce the defect, because two invocations pointed at different directories would each keep their own total and each announce FITS. --quota-ledger moves it on purpose instead of by accident. It is not keyed on the Space, because the allowance is per ACCOUNT and two Spaces on one token share one bβ¦62d612f -
scripts: a Space matrix runner that bounds its spend and refuses to cross itβ
Unverified signatureTen of the eleven modes have never run on the Space. This runs them, and it is built around the three ways that exercise can lie: A cost figure that reads as billing. There is no readable ZeroGPU quota endpoint, so billed seconds can only be bounded. Every figure printed is wall clock times the xlarge multiplier and carries "<=" and "UPPER BOUND" at every site -- a bound that loses its label becomes a measurement the moment it is quoted, which this build has already spent a day undoing once. A run that produced nothing and reads as success. `handlers.on_generate` converts every failure into (None, None, markdown), which over the wire is success-SHAPED. A missing artifact for a track the REGISTRY says the mode exports now raises, naming the mode and quoting the Space's own status text verbatim. An artifact that is present, well-formed and empty. H9 passed every structural check 50 dB below its siblings. So artifacts are decoded, not opened: video must have its frame count, distinct fβ¦f7e7379
-
scripts: fail the sync in 20s with a diagnosis, not 75s with an ssh errorβ
Unverified signatureAn unreachable box took 75 seconds to fail -- ssh's system default, since no ConnectTimeout was set here -- and reported only "Operation timed out". The failure was already SAFE: extraction is staged and swapped on the remote side, so a box that never accepted the connection is left entirely untouched. But a slow, unnamed failure invites someone to assume it hung and start a second one. Now it probes reachability first, exits 4 in ~20s, and says both what failed and that nothing was sent -- because "did this leave the box half-synced?" is the first question a reader has, and the answer should not require reading the script. Found by running it against the box while it was down, which is the only condition that exercises this path -- the same way the staleness guard's own defect surfaced.8c40da2 -
scripts: an unreachable box means staleness is UNKNOWN, not fineβ
Unverified signatureThe guard I added an hour ago had the defect it was written to prevent. With the box offline the SHA probe returned empty, the condition `[ -n "$BOX_SHA" ]` was false, and the guard SKIPPED β then the run died 20s later on a raw ssh timeout. So an unreachable box read as "no staleness problem, proceed", and the failure presented as a network blip rather than as "your staleness check did not run". That is the same shape as startup_report's missing_files=None once meaning "nothing is missing": a check that cannot fail is not a check, and "could not determine" is not "fine". Found by running the guard against the box while it was down, which is the only condition that exercises this path. Now it distinguishes three states and exits on two of them: unreachable (exit 4, naming connectivity), no .synced_sha at all (exit 4, naming the sync), and a SHA that differs (exit 3, naming the missing commits). 15.6s and a diagnosis, against 35.7s and two ssh timeouts.487b597
-
scripts: refuse to run against a stale box, and name the deltaβ
Unverified signatureA measurement taken on a box that lags local HEAD is a measurement of the wrong code, and it looks exactly like a measurement of the right code. That is how "seeding is incomplete" came to be reported: the control ran against 4804da1, which predates the seeding commit, and produced different md5s and 8.7% variation. Both numbers were real and neither was about the fix. The box's SHA was a thing assumed constant and never checked β the same shape as a fixture never varied, which produced the audio_to_audio retraction an hour earlier. Four wrong attributions today share it, so this fixes the class: hinode_run.sh now compares .synced_sha against local HEAD, prints the missing commits by name, and EXITS rather than running. LTX25_ALLOW_STALE=1 makes proceeding a deliberate act. Exiting rather than warning is the point. A warning on stderr scrolls past above a long log, and the failure it prevents is silent by construction β the run succeeds and the conclusion is wrong.419f635 -
scripts: noise_scale=1.0 as a mechanism check, labelled as not a passβ
Unverified signatureAt 1.0 the input is discarded entirely -- noised = 1.0*noise + 0.0*latents -- so this asks only whether the audio branch produces anything when audio_latents is supplied at all. It is also the single setting where our noise_scale already equals the recipe's sigmas[0]=1.0, so it REMOVES the suspected mismatch rather than holding it constant in both arms. Labelled in the table itself because a healthy result here is not evidence that a user's upload works: at 1.0 no upload survives. Same treatment as MEL_FORCE_DEVICE -- an experiment that stands in for something must say what it does not cover, in the place someone reads it.4804da1 -
scripts: change the fixture, keep the criterion strict, stop the message misleadingβ
Unverified signatureThree corrections, and the first is the one that matters: the criterion goes back to requiring ALL tones, and the FIXTURE changed instead. The swap discriminator already established a pairing this path recovers 4/4 -- ch0 (220, 2640) and ch1 (440, 1320) -- where the original recovers 3/4. The difference is the spread WITHIN a channel: 440 + 2640 is 6x and shares 64 log-spaced mel bins differently from 440 + 1320. A fixture that demands something the instrument physically cannot deliver is testing the fixture, not the code. So the strict gate stays and the fixture stops asking for the impossible β rather than the threshold moving until the result is green. The verdict message was actively misleading and had two chances to mislead. It pointed at A.5.1's convention list on any failure, including one where the length tracks the input and the latents validate β which would send the next person to change `power` or `mel_scale` against a working path. The convention list is now offered onlyβ¦1a033a0 -
scripts: gate H17 on placement and length, not on harmonic retentionβ
Unverified signatureThis is a loosening from the previous criterion and it is labelled as one. The justification is a measurement, not the fact that it makes the test green. Both earlier versions were wrong in opposite directions. ">=1 tone per channel" passed with 3 of 4 tones missing BY DESIGN. "Every tone" then failed on a tone measured at 47.5% retention once the pairing was swapped β so that loss is this fixture's tone combination meeting a log-spaced 64-bin mel, not a convention error. v3 gates on what the test is for. A wrong mel_scale, f_min/f_max, norm or center DISPLACES frequencies; it does not attenuate one harmonic and leave the rest in place. So: the reconstruction length must scale with the input length, each channel's fundamental must return at its own frequency, and no tone's energy may appear where it did not come from. Harmonic retention is reported as fidelity data. The length check is the strongest of the three and it is new. The broken path returned exactly 29,280 samples for a 4.β¦d413797 -
scripts: A/B the transpose in one process, to prove the cause not just the effectβ
Unverified signatureComparing two commits proves a difference exists somewhere between them. Running both variants side by side over one input, in one process, with one line as the only difference, proves which line β the difference between evidence of a regression and evidence of its cause. Every step uses the app's own helper β match_channel_count, build_mel_front_end, _matched_to_module, unpacked_latents, decode_latents_to_waveform β so this is not a reimplementation of the path. It is the path, forked at one point. It also reports whether validate_audio_latents_shape accepts each variant, because both are 4D and the guard passing on the wrong one is exactly why this needed a reconstruction to catch.636f96c -
scripts: an opt-in device move, to answer the frequency question while blockedβ
Unverified signatureThe app does not move the waveform to the VAE's device, which is the open CPU/CUDA defect, so H17 deliberately does not either β that is what stopped this harness hiding it, and the flag is off by default. But device placement does not change arithmetic: the mel, the VAE and the vocoder compute the same values wherever the tensor lives. So an opt-in move can answer a question purely about frequency content β which tones survive the round trip, and whether a loss is frequency-dependent or channel-dependent β while the app path stays blocked. The distinction is written into the flag's own logging: it answers nothing about whether a user's upload works, and a run using it must not be reported as an app-path pass. Separating "what does the arithmetic do" from "does the product work" is the point; conflating them is what made the earlier pass mean less than it appeared to.2e85b86 -
scripts: measure band energy per tone, and require all of themβ
Unverified signatureThe pass criterion was ">=1 tone per channel", so H17 reported PASS with 3 of 4 tones recovered β by design, not by accident. A threshold that greens while a quarter of the signal is missing is the shape of result this build keeps producing, and it was mine. All expected tones must now be recovered, and the criterion is printed beside the verdict so nobody has to read the source to know what passed. Top-N peak picking was also the wrong instrument. It reported the four largest peaks, so a real tone sitting fifth was indistinguishable from a tone that was not there β which is why ch1's 2640 Hz could not be called either way. Band energy at each expected frequency answers "is there energy here" without depending on what else is louder, and it is measured relative to the input's own concentration at that tone so a harmonic carried at 0.3 amplitude is not held to the fundamental's bar. MEL_SWAP=1 puts the high tone on the other channel. If the high tone is lost whichever channel carries β¦62975e1 -
scripts: read the H17 fixture through load_waveform, as the app doesβ
Unverified signatureThis harness moved the waveform to the VAE's device and dtype before encoding, and that is how it concealed a real defect: the app does not move it, so encode_waveform_to_latents hands a CPU mel to CUDA weights and raises β while H17, having moved it, passed and reported the mel validated. A harness that performs a step the application omits is the same failure as a float32 fixture over a bfloat16 bug, and this one was mine. The fixture is now written to a wav and read back through load_waveform, so the round trip starts exactly where a user's upload starts: same decoder, same dtype, same device. That also makes load_waveform part of what H17 covers rather than something it routes around, and it means the device mismatch will show up here rather than only in the mode runs.dfacddd -
scripts: test the shape load_waveform actually returnsβ
Unverified signatureTwo changes now that 5aa5d7b casts inside encode_waveform_to_latents. The harness wrapper is gone β leaving it in would test the wrapper rather than the app. And the waveform is unbatched (channels, samples), which is what load_waveform returns and what match_channel_count expects. That is the shape a user's upload takes, so it is the one worth testing; batching it instead characterises a shape no caller produces, and my earlier batched run tripped match_channel_count into reading the batch dim as a channel count.50e60f9 -
scripts: batch the waveform, or the latents come back the wrong rankβ
Unverified signatureAn unbatched (channels, samples) waveform produces an unbatched (channels, mel, frames) mel. conv2d accepts that as a single sample without complaint, so the encode succeeds and returns 3D latents β which validate_audio_latents_shape then correctly refuses, because AUDIO_LATENT_DIMS is 4D [batch, channels, frames, mel]. Worth noting the guard did its job: a 3D tensor reaching the pipeline is the case its docstring describes as "not reliably an error upstream -- it can be silently misinterpreted". It caught a rank error that the VAE itself was happy to accept.81a3e24 -
scripts: prove the mel cast is sufficient, not just necessaryβ
Unverified signaturebuild_mel_front_end computes the STFT in float32 β correct, bf16 would lose precision where the log floor matters β and returns float32 into a bf16 VAE, so audio_vae.encode raises on dtype. The fix is one cast. Rather than wait on it, the harness wraps the constructed transform through the documented to_mel= seam, which is exactly what the module's own error message tells a caller to do. That establishes the cast is SUFFICIENT and not merely necessary, and it lets H17 validate the chosen mel conventions now β those are the last unvalidated claim before the PR, and nothing offline can check them. No change to audio.py; the fix stays the implementer's to place.482bb84 -
scripts: H17, validate the constructed mel by reconstructionβ
Unverified signatureThe forward mel does not exist in the checkpoint, so build_mel_front_end constructs it: five parameters read across two components, and power, f_min/f_max, mel_scale/norm, center and the log floor chosen because nobody declares them. Nothing offline can validate a convention, and a wrong one is audible as degraded audio and reported as nothing β H10/H11 will produce well-formed audio through a wrong analysis window, which is what that phrase means. The fixture is two tones an octave apart plus an envelope, per channel. A wrong window smears narrow spectral peaks, so tones are what make the damage measurable; broadband noise has no peaks to smear and silence has no content, and both reconstruct to something that looks fine. Same trap as a float32 fixture over a bfloat16 defect. Comparison is spectral, not sample-wise. The vocoder does not preserve phase and the round trip is lossy by design, so a sample-wise difference would fail on a correct implementation β spectral shape and where β¦db817a4 -
scripts: the docstring still said "inaudible" after I retired the wordβ
Unverified signatureThe flag was relabelled VERY-QUIET but the module docstring above it still called -55 dBFS inaudible, which is the overstatement the relabel existed to remove. A file that argues one thing in its docstring and another in its output teaches the reader to trust neither. "50 dB below its siblings" is also the more useful framing: it is the comparison that made the reading actionable, where an absolute adjective was what made it arguable.3ce9bcf -
docs+scripts: "very quiet" rather than "near-silent", and the H9 clip's provenanceβ
Unverified signature-55 dBFS is faint, not inaudible: turned up it is plainly audible. Calling it near-silent overstated the measurement and invited a real disagreement with someone who had actually listened β the meter and the ear were never in conflict, my label was. The flag now says VERY-QUIET and names its threshold. The measurement itself survived every check: per-channel so not a summing artifact, both H9 files, correct int16 normalisation demonstrated by H8 peaking at exactly 1.0, an independent tool agreeing at int16 RMS 58.3, and a tensor read upstream of the file. What it establishes is narrower than "broken" and still decisive: the level is not a property of the input, because the same mode with the same clip and seed gives -5.1 dBFS at strength 0.5. The condition clip's provenance is recorded because it was briefly unrecoverable and because it changes the test: it is H1's own generated output, silent, and only its frames ever reach the model. Judging audio-to-video correspondence against a β¦8d73795 -
scripts: never overwrite a run log, and record each run's inputsβ
Unverified signatureEvery run wrote to <script>.log, so the next run destroyed the previous one's record. That cost real provenance: the fixture path behind a recorded H9 result could no longer be read back from the box, and the inputs of a published measurement should not depend on nobody having run anything since. Logs are now timestamped, and each launch appends the SHA, the script and the forwarded environment to an append-only RUNS.manifest. A symlink keeps the stable name so polling is unchanged. The manifest carries the inputs rather than the output because that is the half that was irrecoverable β a result whose inputs are unknown is not a measurement.1079f3c -
scripts: sweep the video condition strength, and report level with every runβ
Unverified signaturevideo_to_audio lands ~50 dB below every other audio mode, and the levels sort by condition type rather than by mode: no condition -5 to -6 dBFS, an image condition -8 to -15, a video condition -55. modes._conditions applies a video condition at MAX_VIDEO_CONDITION_STRENGTH = 0.99, and an image condition anchors one frame while a video condition anchors all 121. The sweep tests that mechanism directly with one lever instead of probing an input property: if a condition strong enough to leave the shared denoise schedule with nothing to do is what silences the audio stream, the level climbs as strength falls. Four points rather than two, so the shape is visible and not just the endpoints. Every run now reports peak/rms/dBFS for any artifact carrying audio. H9 passed every structural check we own β correct codec, correct rate, two channels, verify_streams exit 0 β while being inaudible, so a level belongs beside the stream verdict rather than in a separate pass someone has to remember to β¦2bdb935 -
scripts: measure the audio tensor before encode, to localise two findingsβ
Unverified signatureThe encode step sits between the model and every artifact, so two findings cannot be attributed from the files alone. video_to_audio lands ~50 dB below text_to_audio. If the raw tensor is already that quiet the model produced near-silence and our export is faithful; if it is healthy, something in our path attenuates it. Those are different defects with different owners and the .wav cannot tell them apart. And the PCM wavs peak at exactly 1.0000 β clipping at the int16 boundary rather than a signal that merely reaches full scale β while the aac containers decode to 1.78. Either the vocoder emits hot and we clip, or we add gain. Uses generate(encoder=...), a documented seam, so the probe sits precisely where the real encoder would and the generation is unchanged. It measures and then delegates to the real writer, so the run still produces a genuine artifact and the never-empty contract still holds β a probe that returned a fake path would be testing a different code path than the one β¦15b76b3 -
scripts: measure audio level, because a silent track passes every stream checkβ
Unverified signatureverify_streams proves a mode declared audio and the container carries an audio stream with the right codec, rate and channel count. It cannot tell whether there is any sound in it, and a track at -55 dBFS satisfies every structural check while being inaudible β the same family as an empty output: correctly shaped and useless. Flags NEAR-SILENT below -40 dBFS, clipping above full scale, and near-mono when |L-R| is a negligible fraction of the signal's own level. The stereo test is relative rather than absolute because a quiet track's channel difference can be a rounding error next to its amplitude and still clear a fixed threshold.9eaae1e -
scripts: judge stereo difference relative to level, not absolutelyβ
Unverified signatureThe |L-R| check used an absolute 1e-5 threshold and pronounced "real stereo" on a track whose own rms was 0.0018 β a channel difference of 3.6e-5 there is a rounding error next to the signal, not stereo content. Meanwhile a healthy track showed a difference 30% of its level. A near-mono quiet track would therefore have passed, which is the exact shape of failure this check exists to catch. Comparing the difference to the mean level makes the two cases separable regardless of how loud the track is.8724a1d -
scripts: make the artifacts judgeable, and measure the same-seed pairsβ
Unverified signatureEvery stream check in the matrix passes on a file whose pixels are garbage. "Output looks right?" is the column no tool can fill and where every confidently-wrong artifact on this build came from, so this produces what a person needs to close it. Frames are sampled ACROSS the clip rather than from the start: a pipeline that degrades after frame 40 looks perfect in frame 1. Per-frame mean/std/min/max go in the log too, because a flat-colour frame is a red flag the stream check cannot see and a number catches it without anyone opening the PNG. Spectrograms because silence and noise have identical stream layout β correct codec, rate and channel count say nothing about whether there is signal. It also reports |L-R|, since a channel count of 2 does not mean two different channels and a mono downmix is otherwise invisible. Same-seed pairs are measured as mean absolute pixel difference over co-located frames, not compared by file size: two clips can differ in bytes for container reasons whβ¦863e0e3 -
scripts: two stage-2 runs in one process, to settle the NATTEN questionβ
Unverified signatureNATTEN's version->ref lookup goes over the network even when the kernel is fully cached. Whether that fires once per process or once per generation decides whether a private Space needs outbound network at every press of Generate or only at first stage-2 use β a real product risk either way, but a very different one. One stage-2 run cannot tell the two apart. H14b adds a second in the same process, with a different seed so it is a separate generation rather than something a cache could satisfy.ed718e6 -
scripts: add H14 stage 2 and the two same-seed comparison pairsβ
Unverified signatureH14 runs enable_stage_2 against H1's prompt and seed, so the only difference is the refinement chain and the artifact must differ from H1's. H4b and H10b are pairs that must differ from EACH OTHER, and both halves carry the same pinned seed β seed=-1 draws fresh per call, so a pair left at the default differs for that reason alone and passes vacuously. H10b is the acceptance test for the audio-input path rather than a nicety. noise_scale=1.0 discards the audio input by design, so a bug that drops the input entirely is indistinguishable from correct operation: the mode produces audio either way and every stream check passes. Two noise scales producing identical audio is the only visible signature of the input never arriving.f584611