Spaces:
Running on Zero
Running on Zero
Commit History
3 days ago
-
fix: the bar names what is happening: a staged local run's loads read "loading the model" at the bar's current position until the denoise really starts, and a single pass that decodes inside the pipeline call (the conv decoder, "Fast") reads "decoding the frames" after its last step (#12)
Unverified signatureMeasured on real weights: "sampling - 12%" for the first 60 s of a staged local run while the text encoder and the transformer loaded, and "sampling - 82%" for the last 34 s while the frames decoded; on the Space a Fast run went from "sampling" straight to "writing the file". The sampling callback, and with it the "sampling" announcement, was built while the stage graph put the call's kwargs together, before the runner's seam had made anything resident. StageRunner._call now builds it inside the seam, right before the pipeline call, so the local runner's text-encoder and transformer loads come first, and the local runner names them on the run's reporter as "loading the model" (ProgressReporter.loading): at the bar's current position, since that band is behind it by then, once per denoise and only when something actually loads. A repeated prompt with the transformer resident goes straight to "sampling". A single pass whose pipeline decodes inside its own call (the conv decoder) now sa…aeab028 -
fix: no entry can sample on an MPS PyTorch that fails the numerics self-test: build_local_runner applies the app's gate on mps, so scripts/local_generate.py --device mps exits refused (2) before loading anything, and the remedy names the README's floor, PyTorch 2.14 (#11)
Unverified signaturetorch 2.11 on MPS reuses the first scale scaled-dot-product attention saw, which gives wrong numbers without any error. The app's pipeline factory refused such a build through device.require_mps_numerics, but local_runtime.build_local_runner never called it, and scripts/local_generate.py builds the runner directly: on an affected torch it would have produced a wrong video. build_local_runner now calls require_mps_numerics on mps, after the watermark and availability checks and before any file is looked at, so every entry is covered by one check; LTX25_MPS_SELF_TEST=warn still lets an operator run unverified. The script counts MpsNumericsError among its refusals (exit 2). MPS_CLEAN_TORCH said 2.15 while the README states PyTorch 2.14 or newer (2.14.1 measured the same as 2.15.0 in the numerics checks), so the refusal and the unverified note now say 2.14, as the README does. Tests: the script test begun on wip/deploy2-w14 (ca91f07), unchanged; a runner-level test under enforce and warn…444bc2b
4 days ago
-
fix: a run never inherits another run's adapter: before anything is attached, every run removes any adapter left on the model (on its components, or kept for a released staged transformer), and a run whose leftover cannot be removed is refused as an adapter failure and never sampled
Unverified signatured9da392 -
fix: after the local runner frees memory, the memory check judges the gauge once it has stopped falling, and a check that fails reads it again before refusing
Unverified signatureThe session's first run (mode 05, 704x480, 5 s, staged, Q8 GGUF) was refused at "load transformer": committed 67.3 + need 25.1 = 92.4 GB > 90 GB, read 10-400 ms after the text encoder was released; eight seconds later the gauge read 38.8 GB. Freed MPS memory is wired and the kernel un-wires it over 0.1-0.35 s (the lead's release probe: 62.3, 59.3, 46.3, 38.3 GB at 0.1 s steps, then flat). File pages given back by release_file_cache leave the gauge at once (measured with a 2 GB file, three passes: the first reading after the release was already at the base level), so only device frees count as a release. - memory.preflight, on the live gauge with a limit: released=True first reads every 50 ms until two consecutive readings are within 0.25 GB and judges that reading; a FAIL is read again every 0.1 s and passes as soon as a reading fits, else it is refused on the last reading. Both waits share LTX25_MEMORY_SETTLE_SECONDS (default 2.0, 0 to 60; 0 is the single reading of before;…ba2d582 -
fix: the local script leaves the change level to the mode and writes the run record into its JSON, and the record names the device that ran
Unverified signature- scripts/local_generate.py: --noise-scale defaults to None, so a run without it varies at the mode's own level. It was 0.5, which snapped video to video to "a little" (0.42) instead of its 0.91, as the lead measured. --strength's help says no mode reads it. The run record (result.record as a dict) is written into the --json file beside the artifacts. - generate.RunRecord carries the device the run ran on (record.device): the runner's own device, else this process's resolution. The UI already reads it to label the run's time. - zerogpu.assert_returnable no longer takes a plain string device for a CUDA tensor: a torch object's device is a torch.device, never a str. Without this the record's "cuda" made every result through the Space entrypoint refuse itself (the suite caught it through app.py's run_generation). tests/test_local_generate.py drives the script through its parser and main() with a fake runner that writes real frames: no flag gives 0.909375, the flag still…419e0a5 -
fix: everything a user reads after Generate is in the UI's words, and audio too short for the set length is refused in them
Unverified signatureThe notes above Diagnostics now state facts that change what the user got, in plain words. A silent mode says the sound the model made was left out and names the mode that keeps it (video to video names none, since it has none). Audio to audio says the sound is the model's, made from the user's, and how much it was changed; audio to video says the sound is the user's, unchanged. Lengths and the trim are said in seconds. A requested level is mentioned only when it was snapped to a different level, never as "you asked for 0.42 ... 0.42 is the nearest". The upscaled size, the decoder and the prompt sent are the run record's and are no longer repeated as notes. The decoder policy, evictions, and a track an encoder offered that the mode does not declare go to the diagnostics. The upscale phase reads "upscaling ×2", and Upscale off CUDA is refused as "Upscale ×2 needs a CUDA GPU; this machine runs on cpu." A set length longer than the audio is refused before dispatch with the longest lengt…8a7fa0c -
feat: one stage graph for the Space and the local runner: the local runner is a StageRunner that supplies residency through one seam
Unverified signatureIssue F60, brief W2P item L. LocalStageRunner re-implemented the single-stage call, so every stage-graph change existed only on the Space path: locally, stage 2 was refused outright, an explicit diffusion decode was refused at any size, and the decode policy, audio conditioning, clip variation and run facts were copies. - StageRunner routes every pipeline call through one seam, _around_call(pipeline, call, stage=...): each denoise ("single", "stage 1", "stage 2") and each stage-2 part ("upsample", "decode"). It yields the kwargs the call is made with; the Space runner's is a no-op. The audio-conditioned pipeline is derived after the seam, so it shares the modules the seam made resident. - LocalStageRunner is now a StageRunner subclass. Its seam sets the CRF on every condition, swaps the prompt for embeddings (a cache hit for stage 2, so the text encoder is not reloaded), makes the transformer resident when staged, preflights the denoise, and runs it under inference_mod…c712e7e -
fix: the local runner preflights adapters and frees memory after a failure, keeps CPU weights' pages, sizes auto-duration as the head does; the worker logs a small GPU slice and the override fetches nothing
Unverified signatureReview minors local-runtime-7, -8, -9, -10, space-path-2 and space-path-3. - local-runtime-7: attaching a LoRA on the local runner is preflighted like any load (file x spike, scaled for a converting dtype), from generate before lora.apply, which would report a memory refusal as an unreadable file. A transformer reload that puts captured adapters back promises their bytes too. - local-runtime-8: every load, the clip encode, the denoise and a failed build drop what the failed step held (its frames' locals) and empty the device cache before re-raising; a failed build releases everything it loaded. - local-runtime-9: on the CPU at the checkpoint's own dtype the weights are the file's mapped pages; the move no longer releases them (which evicted the live model the denoise then faulted back in uncounted). A converting load and every unload still release. - local-runtime-10: the auto-duration denoise estimate is the longest clip the duration head can return (constraints.dur…f851454 -
fix: the local runner never restores a failed run's adapter, preflights a converting load at its real cost, and a real-sized snapshot override goes through it
Unverified signatureReview findings local-runtime-2 (= lora-1), -3, -4 and -6 (= lora-2). - A staged release captures the transformer's adapters to restore them after a reload. When the run then failed before the reload (a text-encoder preflight refusal, a load or encode error), generate()'s detach found the transformer released and deleted nothing, so the next run - with no LoRA asked for - generated through the old adapter. The staged pipeline's delete_adapters now also prunes the captured copy (as load_lora_weights is already guarded). - The first LoRA run on a fresh staged runner attaches (generate makes the transformer resident first since the run-record commit); pinned with a test through the real runner. - A converting load (fp32 or fp16 from the bf16 checkpoint) copies every tensor at the target size while the pages it read stay active until the move: the load preflights and the dry-run plan now promise file x spike x (target/stored) + the file pages, with the stored dtype rea…12a7ca3 -
feat: video to video is a variation of the user's clip, noised to a level and denoised along the schedule's tail
Unverified signatureMeasured on real weights (F66): the video condition's strength was a switch - the clip came back unchanged at 0.3-0.8 and the prompt alone acted at 0.0 - because the condition is blended into the predicted clean sample at every step. Mode 04 now does what stage 2 does to its upsampled latents: the clip is encoded with the video VAE to raw latents, noised to a level s, and only the tail of the distilled schedule from s is denoised, with the prompt and no video condition. - Registry: video_to_video has video_conditioning="variation" and builds no condition from its clip; the request's noise_scale is its level, snapped to 0.421875 / 0.725 / 0.909375 (default 0.909375, the only level at which the prompt acts). The clip travels as one modes.VariationClip kwarg. noise_scale now defaults to None (the mode's default: 0.42 for audio_to_audio, unchanged in effect; 0.91 for a clip); strength is read by no mode now and is scoped to its default. - Length: the frame count comes from the…85405e9 -
fix: a derived pipeline shares the loaded modules without casting them, so the audio modes run with a quantized transformer
Unverified signatureOn real weights every audio-input run on the local runner died with "Casting a quantized model to a new dtype is unsupported": the audio-conditioned pipeline was derived with from_pipe(base, dtype=...), which calls .to(dtype) on every module, and a GGUF-quantized transformer refuses that (bitsandbytes on the Space is skipped with a warning; without a dtype from_pipe casts the shared modules to float32 in place). derive_pipeline now constructs the target class directly from base.components, filtered to the arguments it accepts, so nothing is moved or cast. It is the one derivation used by the text2video pipeline, the audio-conditioned pipeline and the tiny stage; stage 2's parts are separate models and are not derived. Tests: every mode once, and audio_to_audio at each variation level, through generate() on the local runner with the tiny GGUF transformer (needs the gguf package); deriving with a fake quantized module whose dtype cast raises; the audio pipeline on the real tiny classes…b12415f -
fix: audio conditioning has its own noise level and schedule, so audio_to_video and audio_to_audio sample the video like every other mode
Unverified signatureBoth modes passed the UI's "audio noise scale" (default 0.5) as the condition pipeline's single noise_scale. Upstream applies that one number to the audio latents AND to the video's initial noise, while the schedule still starts at sigma 1.0: the denoiser is told "pure noise" and handed half-variance video latents, which is the grey veil seen in audio_to_video. On the tiny pipeline the video started at std 0.46 in audio_to_video and 0.72 in audio_to_audio. New ltx25_studio/audio_conditioning.py: AudioConditionedPipeline (derived from the loaded condition pipeline, sharing every module) noises supplied audio to its own level, and AudioScheduler (in the pipeline's audio_scheduler slot) runs min(video sigma, level) per step, indexing by its own counter because upstream calls it with the video's timestep. A step whose sigma does not move returns the sample untouched, so upstream's 0/0 velocity at sigma 0 never reaches the latents. noise_scale is no longer sent, so the video starts at its …be9c1db -
fix: the local runner keeps the checkpoint's float64 RoPE on MPS by computing it on the CPU, after every load
Unverified signaturedevice.compute_rope_on_cpu is now the one RoPE mechanism for both runners: on MPS, which has no float64, each RoPE module's forward runs on the CPU at the checkpoint's double precision and only its tables move to the device. local_runtime.apply_rope_precision is the single place the runner applies it: to the connectors once, and at the end of every load_transformer, the safetensors and the GGUF route alike, so each staged reload of the transformer is hooked again (the hooks live on module instances). The rope_double_precision=False load kwarg is gone; cpu is untouched. Tests: after a staged reload the transformer is a new instance whose RoPE modules keep double precision and carry the CPU hooks, and an fp32 forward on MPS matches the CPU within 1e-3; the resident-MPS, MPS-smoke and GGUF-on-MPS tests now pin float64 plus the hooks instead of single precision. Also adds the test 5c70a8f lacked: weight files are found on disk whatever they are called, and a component with none is reporte…5e4829a -
fix: refuse an MPS watermark pair that makes every empty_cache() fail, before anything loads
Unverified signatureWith PYTORCH_MPS_HIGH_WATERMARK_RATIO below 1.4 and the low ratio unset, torch keeps its 1.4 low default and torch.mps.empty_cache() raises 'invalid low watermark ratio 1.4' — inside from_single_file and in the runner's own staging. memory.mps_watermarks() reads both ratios and names the problem and the fix; a high ratio without a low one at or below it, or a value that is not a number, is refused with a typed MpsWatermarkError. build_local_runner checks it on mps; scripts/local_generate.py prints both values and torch's recommended MPS maximum in the plan (dry run too) and exits 2 before touching anything. Nothing sets the variables.adad9b6 -
fix: the local runner gives model-file pages back — the committed gauge no longer counts weights twice
Unverified signatureMeasured by the lead on real weights: after a component moved to MPS, its memory-mapped safetensors pages stayed on the active queue for minutes, and active + wired + compressor (the gauge we keep) counted them, so G0 peaked ~22 GB over budget. Evicting them with mmap + msync(MS_INVALIDATE) dropped committed by the file's size in a fraction of a second. - memory.release_file_cache(paths): darwin msync(MS_INVALIDATE) per file, linux posix_fadvise(DONTNEED); resolves the HF cache's symlinks, walks directories, counts each file once, never raises. Data on disk untouched. - The runner moves every safetensors component (the small set, the text encoder, the transformer) tensor by tensor and releases the files' pages every ~2 GB, instead of one .to() that holds file pages plus the device copy; one release at the end, one line logged per release. The GGUF file is released after load_gguf_checkpoint (everything is copied by then) and after the move. Releasing the text encoder or …75d0c53 -
feat: scripts/local_generate.py — one local generation through the app's own generate(), with its memory story
Unverified signatureThe lead's entry point for real-weights runs on the Mac. The local runner is injected through generate()'s pipeline-factory seam, so it works before pipelines.get_pipeline knows about local devices. It prints the resolved plan, every preflight line, per-phase wall time, the peak committed memory (vm_stat, sampled every 0.5 s) and MPS driver memory, the files written and verify_streams' verdict, and writes the same with --json. --dry-run prints the plan and the projected preflight arithmetic and loads nothing. Nothing is downloaded: a missing file is reported with the exact hf download command. Exit codes: 0 ok, 2 refused (memory, validation, missing files), 3 error. The runner now attaches the connectors after the pipeline load (passed in, diffusers logs the whole module), and the frame bound used for an auto-duration estimate is public for the dry run.af26f09
5 days ago
-
feat: a local mps/cpu runner — the text encoder and the transformer are never resident together
Unverified signaturelocal_runtime.build_local_runner returns a LocalStageRunner that is call-compatible with pipelines.StageRunner (runner(enable_stage_2=False, progress=None, **kwargs), .pipeline, .kind), so generate() drives it through the same pipeline-factory seam. - Staged residency: per call, release the transformer, load the text encoder, encode exactly what the pipeline would (negatives only when guidance needs them), release it, load the transformer, and call the pipeline with the embeddings. A bounded cache skips the text encoder for a repeated prompt and says so. 'resident' loads everything once and lets the pipeline encode, which is the reference the staged path is proven against: identical frames and audio on the tiny checkpoint for text, image, video, audio-latent and audio-producing modes, with the duration head, and with the same number of progress steps. - The transformer comes from the snapshot or from a GGUF file through diffusers' own loader, after renaming the 12 pr…2155bf9