Spaces:
Running on Zero
Running on Zero
Commit History
4 days ago
-
feat: video to video is a variation of the user's clip, noised to a level and denoised along the schedule's tail
Unverified signatureMeasured on real weights (F66): the video condition's strength was a switch - the clip came back unchanged at 0.3-0.8 and the prompt alone acted at 0.0 - because the condition is blended into the predicted clean sample at every step. Mode 04 now does what stage 2 does to its upsampled latents: the clip is encoded with the video VAE to raw latents, noised to a level s, and only the tail of the distilled schedule from s is denoised, with the prompt and no video condition. - Registry: video_to_video has video_conditioning="variation" and builds no condition from its clip; the request's noise_scale is its level, snapped to 0.421875 / 0.725 / 0.909375 (default 0.909375, the only level at which the prompt acts). The clip travels as one modes.VariationClip kwarg. noise_scale now defaults to None (the mode's default: 0.42 for audio_to_audio, unchanged in effect; 0.91 for a clip); strength is read by no mode now and is scoped to its default. - Length: the frame count comes from the…85405e9 -
fix: audio conditioning has its own noise level and schedule, so audio_to_video and audio_to_audio sample the video like every other mode
Unverified signatureBoth modes passed the UI's "audio noise scale" (default 0.5) as the condition pipeline's single noise_scale. Upstream applies that one number to the audio latents AND to the video's initial noise, while the schedule still starts at sigma 1.0: the denoiser is told "pure noise" and handed half-variance video latents, which is the grey veil seen in audio_to_video. On the tiny pipeline the video started at std 0.46 in audio_to_video and 0.72 in audio_to_audio. New ltx25_studio/audio_conditioning.py: AudioConditionedPipeline (derived from the loaded condition pipeline, sharing every module) noises supplied audio to its own level, and AudioScheduler (in the pipeline's audio_scheduler slot) runs min(video sigma, level) per step, indexing by its own counter because upstream calls it with the video's timestep. A step whose sigma does not move returns the sample untouched, so upstream's 0/0 velocity at sigma 0 never reaches the latents. noise_scale is no longer sent, so the video starts at its …be9c1db -
fix: a run consumes only what its mode declares, and is validated before a GPU is reserved
Unverified signatureThe UI keeps the value of every field it hides and /on_generate forwards them all, and nothing downstream scoped them. A stale audio upload was encoded into every mode and muxed into 05/06/07 as "the audio YOU supplied"; a stale image or video became a condition in every condition mode; a hidden prompt drove 04/09/10/11; a hidden enhance flag added 180 s to the reservation and ran Gemma. Nothing checked that a mode's required inputs were present, so image_to_video with no image was an unconditioned, billed run reported as a success. And every check ran after the ZeroGPU dispatch, so each refusal cost a GPU slot. generate() now prepares the request before the dispatch branch, so the reservation estimate and the worker both see the prepared request: scope it to the mode (scope_request_to_mode), require the mode's inputs (require_inputs -> MissingInputError naming the mode, each missing input and a mode that does not need it), validate geometry and length with the new upper bounds, open …6e07c8e
-
fix: `audio_to_video` hands back YOUR audio — the model's track is silence when audio is supplied
Unverified signatureA controlled discriminator on the live Space, same pipeline, same 512x960x121 geometry, same seed, one variable — whether an audio input was supplied: audio_to_video (audio IN) video+aac ✓ -88.3 dBFS digital silence text_to_audio_video (no audio) video+aac ✓ -24.2 dBFS real content 64 dB of separation. **Supplying audio suppresses the model's own audio generation.** That is a property of the model, not a defect in our plumbing: `4683a57`'s mux is correct and both files carry a proper 48 kHz stereo AAC track. But `4683a57` chose to mux the model's track, so a user who hands us audio has been getting back a file whose AAC stream is empty — the same emptiness the original mode-11 defect had, now wearing a stream header, which is arguably worse because it looks like a track until you play it. RULED, for `audio_to_video` only: mux the audio the user SUPPLIED. He provided it, he expects it back, and the model's is empty anyway. The six reasons at the mode-11 sp…e2f538a
-
fix: audio-conditioned video came back silent — mode 11 now muxes the model's own track
Unverified signature`audio_to_video` took an audio upload and returned an mp4 with ZERO audio streams. Confirmed twice on the live Space: the operator hit it, and a controlled run reproduced it as `audio: False` in the response tuple. A user who conditions on audio expects audio in the file. The registry declared exports=("video",), so `outputs._write_video` took its `include_audio` branch, discarded the generated track, and reported it through `dropped_audio_warning` — which advised "use the matching audio+video mode to keep it". Mode 11's `audio_counterpart` is None. THERE IS NO SUCH MODE. The one mitigation the drop had pointed at nothing, and sent the user hunting for a mode nobody built. RULED: THE MUXED TRACK IS THE AUDIO THE MODEL GENERATED, NOT THE AUDIO THE USER SUPPLIED. Both were defensible. The generated one is the one that cannot silently mislead. Six reasons, recorded in full at the mode-11 spec: 1. `audio_to_audio` already answers this. Mode 10 takes the same input, through the sa…4683a57
-
docs: keep the record that the condition ceiling never binds
Unverified signatureThe revert of 0734d2e took two things with it that were right independently of the fix: the documentation that MAX_VIDEO_CONDITION_STRENGTH never binds, and the test that fails if that stops being true. Restored, and strengthened with why it matters: this constant has now been proposed as the fix for the video_to_audio silence twice, and condition strength was not the cause either time. min(request.strength, 0.99) with a request default of 0.8 never reaches 0.99, so lowering it would have changed nothing. The sweep that appeared to implicate strength measured a real interaction with the broken SFT guidance -- a genuine effect inside a pipeline that no longer exists. With the distilled recipe corrected, H9 runs healthy at the unchanged default of 0.8. video_to_audio keeps the default. Halving the condition strength would have weakened the one thing that mode is for -- audio that matches its conditioning video -- to fix a problem that no longer exists.7d6980a -
Revert "fix: per-mode video condition strength; video_to_audio at 0.5"
Unverified signatureThis reverts 0734d2e. The evidence it rested on has been superseded by the corrected matrix (809b87f). After d513de5, H9 runs at the UNCHANGED default strength 0.8 and comes out at -17.8 dBFS against H8's -15.3 -- a 2.5 dB gap where there was 50 dB. The strength sweep measured an interaction between condition strength and the broken SFT guidance, not a root cause. There was never a strength defect to fix. Keeping the cap would be worse than doing nothing: 0.5 was measured only under the broken recipe, so it would apply an unmeasured value to a mode that is now healthy at its default, on the authority of a sweep whose conclusion has been withdrawn. Worth recording that the source citation held throughout. cond.py:2017 applies conditioning_mask to noise_pred_video only and never touches noise_pred_audio, so the video condition cannot suppress audio directly. That refutation was correct, the clean single-lever correlation did not contradict it, and the correlation is what got believed.…18d387a -
fix: per-mode video condition strength; video_to_audio at 0.5
Unverified signaturevideo_to_audio produced -55.0 dBFS -- inaudible -- against text_to_audio's -5.2 at identical settings. The cause is the strength=0.8 request default reaching the video condition, measured on the card at seed 1234: 0.99 -> -38.6 dBFS near-silent 0.80 -> -55.0 dBFS near-silent (the default that shipped) 0.50 -> -5.1 dBFS HEALTHY 0.20 -> -11.5 dBFS healthy 0.05 -> -13.2 dBFS healthy 0.5 is used because it was MEASURED, not because it is middling. The sweep is non-monotonic -- 0.99 is louder than 0.80, and 0.50 is louder than 0.20 -- so it is a cliff in how many denoise steps survive, not an attenuation curve. Interpolating a value between measured points would be a guess across a discontinuity, and the table and that warning are in the source beside the constant because the next person will want to widen it. The mechanism, which the source alone could not have found: strength sets where in the sigma schedule the run starts, so at 0.8 only the last ~20% of steps e…0734d2e -
test: spec output retention; align mode workflows with the upstream map
PLAN §2.7 resolved OQ-6 with a number, so retention is now specifiable: 20 GB, bounded-bytes, oldest-first, whole run directories, and never the newest run — generating a video and evicting it in the same sweep is not a retention policy. Mode pipeline kinds now use upstream's own _workflow_map vocabulary (text2video / condition) per the amended §3.1-3.3, rather than names this suite invented. Modes 10 and 11 move to the condition workflow with the table.2941f37 -
test: spec the mode registry, the never-empty-output invariant, app build and layering
The generate.py specs are the ones that matter: failure is injected at four depths (model load, decode, missing output file, zero-byte output file) and each must surface a distinct typed error rather than a falsy path. Also asserts there is no newest-file-in-output-dir fallback, statically and behaviourally — the 2.3 version had no run correlation and would show a previous run's video for a run that failed. Mode identifiers are fixed here as the contract: PLAN §3.2 numbers and labels the modes but never names them.1e0df90