techfreakworm commited on
Commit
bfaee84
·
unverified ·
1 Parent(s): ca5c091

docs: fold in ZeroGPU recon; add pinned requirements

Browse files

PLAN.md rewritten against corrected recon. The substantive reversals:

- ZeroGPU is RTX PRO 6000 Blackwell via MIG, not an A10G and not an
H200 — large is 48 GB, xlarge is 96 GB, and the API's 'zero-a10g'
string is a stale identifier. VRAM was never the constraint.
- enable_model_cpu_offload() must NOT be used. It is counterproductive
on ZeroGPU: anything accelerate parks on CPU is never registered for
packing, so it stays in main-process RAM, is fork-inherited by every
worker, and is re-copied host to device on every hook fire. The
platform's own O_DIRECT disk packer replaces it. Section 8 rewritten
around import order, module-scope .to('cuda'), static @spaces.GPU,
and the never-return-CUDA-tensors rule.
- CPU RAM (~68 GB) and transient disk (~144 GB peak against a 150 GB
cap) are the real ceilings. bf16 ships as the default with a loud
preflight; quantization is a one-line fallback switch, not the
default, because ZeroGPU's packer handling of quantised tensor
subclasses is unverified. Recorded as technical risk #1.
- The classic pipelines decode with the conv VAE, not the diffusion
decoder, which is 2.5's native default — so a naive call silently
discards the headline quality feature. Stage graph now decodes
explicitly, with denormalize=False mandatory against double
denormalization.
- Multishot downgraded to honest: no API, and the enhancer system
prompts actively discourage scene cuts.
- The 11 UI modes are documented as a presentation layer over four
real workflows rather than eleven code paths.

Adds twelve API gotchas and seven operational lessons as explicit test
cases, resolves all six open questions, and adds a five-entry risk
register.

requirements.txt carries the full pin set: torch trio pinned together
against the torchvision 0.28 trap, diffusers at an exact SHA with a
startup version assertion, gradio 6 since 5.x is EOL, numpy 2. No
spaces line — the platform pins its own version and a conflicting pin
breaks build-time resolution.

Adds preflight and zerogpu module stubs, a dev extra for test tooling,
and a 20 GB output retention cap.

README.md CHANGED
@@ -7,4 +7,4 @@ video + audio generation model — covering every mode the model supports, with
7
  Runs locally on Apple Silicon (MPS) / NVIDIA (CUDA) and deploys to a private
8
  Hugging Face Space on ZeroGPU.
9
 
10
- Status: greenfield. See `docs/PLAN.md` once the design lands.
 
7
  Runs locally on Apple Silicon (MPS) / NVIDIA (CUDA) and deploys to a private
8
  Hugging Face Space on ZeroGPU.
9
 
10
+ Status: design locked, implementation starting. See [`docs/PLAN.md`](docs/PLAN.md).
docs/PLAN.md CHANGED
@@ -4,19 +4,23 @@
4
  **Branch:** `soma/ltx25-studio` · **Remote:** `git@github.com:techfreakworm/ltx2.5-studio.git`
5
  **Target:** private Hugging Face Space `techfreakworm/LTX2.5-Studio` (gradio SDK, ZeroGPU).
6
 
7
- This document is the contract for the build. Every section below is either a **decision**
8
- (settled, build to it) or an **open question** (§10, blocks work, needs an answer). Where a
9
- claim was verified against real source or real repo metadata, the evidence is cited inline so a
10
- future reader can re-check it instead of trusting it.
11
 
12
  **Prime directive:** this app must not repeat the failure modes of its predecessor
13
- (`ltx2.3-AIO-generator`). Those failures were not model failures. They were *packaging,
14
- pinning, and silent-failure* failures. §5 is therefore not a nice-to-have section; it is the
15
- section the build will be judged on.
 
 
 
 
16
 
17
  ---
18
 
19
- ## 1. Decision 1 — Integration path: **Diffusers**
20
 
21
  We build on `diffusers`, loading `Lightricks/LTX-2.5-Diffusers`.
22
 
@@ -30,102 +34,118 @@ We build on `diffusers`, loading `Lightricks/LTX-2.5-Diffusers`.
30
 
31
  ### 1.2 Why C
32
 
33
- **The `ltx2` module is real and complete enough.** `diffusers` ships a first-class
34
- `pipelines/ltx2` package. Verified present in the installed source tree:
35
- `pipeline_ltx2.py`, `pipeline_ltx2_condition.py`, `pipeline_ltx2_image2video.py`,
36
- `pipeline_ltx2_latent_upsample.py`, plus `connectors.py`, `latent_upsampler.py`,
37
- `vocoder.py`, `export_utils.py`, `utils.py`. Models `AutoencoderKLLTX2Video`,
38
- `AutoencoderKLLTX2Audio`, `LTX2VideoTransformer3DModel` are exported from the top-level
39
- `diffusers` namespace.
40
-
41
- **Path A is a heavy, fragile install for a Space.** It requires `uv sync` of a git workspace,
42
- CUDA >= 12.7 and torch ~= 2.7. A Space build that depends on resolving a git workspace at
43
- image-build time is a build we cannot reproduce or pin.
44
 
45
  **Path B is the documented cause of the fragility we are told to avoid.** The 2.3 project's
46
- `requirements.txt` vendors an entire ComfyUI runtime — roughly 50 dependencies, nearly all
47
- unpinned: `spandrel`, `torchsde`, `kornia`, `PyOpenGL`, `glfw`, `SQLAlchemy`, `alembic`,
48
- `Mako`, `blake3`, `comfyui-frontend-package`, `comfyui-workflow-templates`,
49
- `comfyui-embedded-docs`, plus a hand-maintained block of *custom-node deps we discovered are
50
- required but aren't auto-installed by their respective nodes* (`gguf`, `imageio_ffmpeg`,
51
- `opencv-python`, `matplotlib`, `scikit-image`, `onnxruntime`, `yt-dlp`). That comment block is
52
- itself the evidence: the dependency surface was discovered by breakage, not by design.
53
- Diffusers replaces all of it with ~15 pinned direct deps and zero custom-node surface.
54
 
55
  **Decisive coupling to §2.** The `*-comfy-int8-convrot` quantised weights are **ComfyUI-only**
56
- — the model card says so explicitly. Choosing A or C means those 36.9 GB of int8 weights are
57
- unusable to us. Path choice and disk budget are therefore one decision, not two, and §2 is
58
- computed on the bf16 Diffusers pack.
 
 
 
 
 
 
 
 
 
 
 
59
 
60
- ### 1.3 The single-file escape hatch: investigated and REJECTED
 
 
 
61
 
62
- It is tempting to skip the Diffusers pack and load the 200.85 GB single-file release through
63
- `from_single_file`. That does not work, and the reason is specific.
 
 
 
 
 
 
 
 
 
 
 
64
 
65
  `diffusers/loaders/single_file_utils.py` does contain an LTX-2 route:
66
  `CHECKPOINT_KEY_NAMES["ltx2"]` → `model_type = "ltx2-dev"` → config repo
67
- `{"pretrained_model_name_or_path": "Lightricks/LTX-2"}`. But note the repo it points at:
68
- `LTX-2`, i.e. **2.0, not 2.5**. And the converters it ships are only:
69
-
70
- - `convert_ltx2_transformer_to_diffusers` (`LTX_2_0_TRANSFORMER_KEYS_RENAME_DICT`)
71
- - `convert_ltx2_vae_to_diffusers` (`LTX_2_0_VIDEO_VAE_RENAME_DICT`)
72
- - `convert_ltx2_audio_vae_to_diffusers` (`LTX_2_0_AUDIO_VAE_RENAME_DICT`)
73
 
74
  There is **no** single-file converter for 2.5's `connectors`, `diffusion_decoder`, `vocoder`,
75
- `duration_head` or `latent_upsampler`. Even a successful single-file load would give us a
76
- pipeline missing the components that make 2.5 what it is. We use the Diffusers pack. This is
77
- settled; do not revisit it without new upstream converters.
78
 
79
  ---
80
 
81
- ## 2. Decision 2 — Disk budget: **~72.2 GB of a 150 GB cap**
82
 
83
- ZeroGPU gives us a 150 GB persistent-storage cap. The `Lightricks/LTX-2.5-Diffusers` repo
84
- totals **164.19 GB across 56 files** — which would blow the cap. That total is *not* what we
85
- download, and understanding why is a correctness requirement, not an optimisation.
 
 
 
 
 
 
86
 
87
  ### 2.1 Trap A — duplicate shard sets
88
 
89
- Two folders in the repo ship the *same weights twice*, in two different shardings:
90
 
91
  | folder | set 1 | set 2 | note |
92
  |---|---|---|---|
93
  | `transformer/` | 4 shards: 9.932 + 9.919 + 9.986 + 8.138 = **37.975 GB** | 8 shards: 4.991 + 4.944 + 4.994 + 4.928 + 4.977 + 4.995 + 4.994 + 3.163 = **37.986 GB** | only **one** `diffusion_pytorch_model.safetensors.index.json` |
94
  | `connectors/` | single file: **6.344 GB** | 2 shards: 5.000 + 1.344 = **6.344 GB** | same |
95
 
96
- A naive `snapshot_download` of those folders pulls **both** sets and roughly doubles the
97
- footprint (+44.33 GB of pure duplication). `from_pretrained` does the right thing: it reads
98
- the index JSON and fetches only the shards the index references.
99
-
100
- **Two consequences that are binding on the build:**
101
 
102
- 1. We rely on the index. We do **not** hand-roll a `snapshot_download` of `transformer/`.
103
- 2. Any `preload_from_hub:` list in the Space README must name **only the live shards** — the
104
- ones the index references. A wrong list here silently doubles disk and fails the deploy at
105
- 72 GB in, which is the worst possible time to find out.
106
 
107
  ### 2.2 Trap B — `transformer_full/` is the DEV transformer
108
 
109
- `transformer_full/` is a further **37.975 GB** and is the *dev* (non-distilled) transformer. We
110
- ship the distilled 8-step path by default, so we do not ship it. It is documented as an opt-in
111
- (§2.5), not deleted from our mental model.
112
 
113
- ### 2.3 The subset we ship
114
 
115
  | component | GB | why |
116
  |---|---:|---|
117
  | `transformer/` (distilled, one shard set) | 37.975 | the model |
118
- | `text_encoder/` (5 shards: 4.961 + 4.967 + 4.884 + 4.986 + 4.121) | 23.919 | Gemma-3 12B (see §10 OQ-4) |
119
  | `connectors/` | 6.344 | text → transformer conditioning; also a LoRA target |
120
- | `vae/` | 1.452 | video VAE |
121
  | `audio_vae/` | 0.107 | audio VAE (mel ↔ latent) |
122
  | `vocoder/` | 0.258 | mel → waveform |
123
  | `latent_upsampler/` | 0.996 | stage-2 spatial upscale |
124
  | `temporal_latent_upsampler/` | 0.262 | stage-2 temporal upscale |
125
- | `diffusion_decoder/` | 0.834 | high-quality decode (see §10 OQ-1) |
126
- | `duration_head/` | 0.004 | duration prediction (see §10 OQ-1) |
127
  | `tokenizer/` | 0.032 | |
128
- | **TOTAL** | **72.185** | **≈ 72.19 GB** |
129
 
130
  ### 2.4 What we exclude, and why
131
 
@@ -133,137 +153,197 @@ ship the distilled 8-step path by default, so we do not ship it. It is documente
133
  |---|---:|---|
134
  | `transformer_full/` | 37.975 | the DEV transformer; not needed for the distilled 8-step path |
135
  | duplicate shard sets (transformer 8-shard + connectors 2-shard) | 44.330 | Trap A — same weights, different sharding |
136
- | `ltx-2.5-22b-distilled-lora-450-bf16.safetensors` (repo root) | 9.699 | the distilled LoRA for **DEV** workflows; useless when we already ship the distilled transformer |
137
 
138
- **Arithmetic check:** 72.185 + 37.975 + 44.330 + 9.699 = **164.189 GB** — which closes exactly
139
- against the repo's stated 164.19 GB total. The accounting above is complete; there is no
140
  unexplained remainder.
141
 
142
- ### 2.5 Headroom
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
143
 
144
- `150 − 72.19 ≈ **77.8 GB**` left for the pip environment, user-uploaded LoRAs, and generated
145
- outputs. The dev transformer (`transformer_full/`, 37.98 GB) fits inside that headroom, which is
146
- why it is an *opt-in* rather than an impossibility — but enabling it leaves ~40 GB, and outputs
147
- must then be pruned aggressively.
148
 
149
- **Implied requirement:** outputs are not immortal. `outputs/` needs a retention policy (bounded
150
- count or bounded bytes, oldest-first eviction) or the Space will fill up in normal use.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
 
152
  ---
153
 
154
- ## 3. Decision 3 — Mode coverage: all 11, honestly
 
 
 
 
 
 
 
 
155
 
156
- All 11 requested modes are reachable. Several are reachable only in a way that costs more than
157
- the name suggests, and the plan states that plainly rather than shipping a fast-looking button
158
- that is secretly slow.
 
 
 
159
 
160
- ### 3.1 Three facts that shape the whole registry
 
 
 
 
161
 
162
  **(a) One conditioning mechanism covers both image and video.**
163
- `LTX2VideoCondition` is a dataclass whose `frames` field is typed
164
  `PIL.Image.Image | list[PIL.Image.Image] | np.ndarray | torch.Tensor` — "the image (**or
165
- video**) to condition the video on". It carries `index` (the frame position at which the
166
- conditioning takes effect, default `0`) and `strength` (default `1.0`). So image-conditioning
167
- and video-conditioning are *the same code path* with a different payload. The UI should expose
168
- it that way: a list of conditions, each with a source, an index and a strength.
169
 
170
  **(b) The model is jointly audio-visual. There is no video-off switch.**
171
- The return type is `LTX2PipelineOutput(frames=..., audio=...)` — both fields are non-optional
172
- `torch.Tensor`. Every call spends video compute and returns a video. This is the single most
173
- important cost fact in the app, and it is why the "audio" modes below carry a warning.
174
 
175
  **(c) Audio input is not a first-class kwarg.**
176
- There is no `audio=` argument. Audio goes in as `audio_latents` (waveform → mel →
177
- `audio_vae` → latents), combined with `noise_scale`. `prepare_audio_latents` routes through
178
- `_create_noised_state(latents, noise_scale, generator)`, whose body is
179
- `noised_latents = noise_scale * noise + (1 - noise_scale) * latents` — i.e. classic
180
- img2img-style strength, where `noise_scale=1.0` discards the input entirely and `0.0` keeps it.
181
- Audio frame count and video frame count are computed independently, so **they can desync**;
182
- §5.5 makes validating that a hard requirement.
183
 
184
- ### 3.2 The mode registry
185
 
186
- | # | Mode | How it is reached | Cost note |
187
- |---|---|---|---|
188
- | 1 | text → video | `LTX2Pipeline(prompt=…)` | first-class |
189
- | 2 | image → video | `LTX2ConditionPipeline` with one `LTX2VideoCondition(frames=PIL, index=0)`, empty prompt | first-class |
190
- | 3 | image + text → video | as #2 with a prompt | first-class |
191
- | 4 | video → video | `LTX2ConditionPipeline` with `LTX2VideoCondition(frames=<video>, strength<1.0)` | first-class |
192
- | 5 | text → audio+video | #1 — the audio track is always produced | first-class; **this is the default shape** |
193
- | 6 | image → audio+video | #2 | first-class |
194
- | 7 | image + text → audio+video | #3 | first-class |
195
- | 8 | text → audio | #1, export only the audio track | ⚠️ **still pays full video compute** |
196
- | 9 | video → audio | #4, export only the audio track | ⚠️ same |
197
- | 10 | audio → audio | `audio_latents` + `noise_scale`, export only audio | ⚠️ same |
198
- | 11 | audio → video | `audio_latents` + `noise_scale`, export only video | ⚠️ pays audio encode on top |
199
-
200
- **Mitigation for modes 8–11** (mandatory, not optional):
201
- run at the **minimum legal resolution**, **skip the stage-2 spatial upscale**, and export only
202
- the requested track. The UI must say, in words, that audio-only modes still render video
203
- internally — a user who does not know that will read the runtime as a bug.
204
-
205
- ### 3.3 Multishot is a UI affordance, not a 12th mode
206
-
207
- 2.5's native multishot is driven by (i) prompt structure — per the LTX prompting guide — and
208
- (ii) multiple `LTX2VideoCondition`s at different `index` values. It is not a separate pipeline.
209
- We therefore build it as a **first-class UI affordance** on top of the condition list: add a
210
- shot, give it a prompt segment and an anchor frame index. Building it as a separate "mode"
211
- would duplicate the condition-list code for no gain.
212
-
213
- ### 3.4 Prompt enhancement: **deferred, and we ship no dead toggle**
214
-
215
- `enable_prompt_enhancement=True` requires a separate `prompt_enhancer` component
216
- (`google/gemma-4-E2B-it`). It is optional in the pipeline constructor and **is not present in
217
- the Diffusers pack**. Shipping a toggle that cannot work is exactly the kind of silent lie §5
218
- exists to prevent. Documented as deferred/opt-in with that reason; no UI control until the
219
- component is actually fetchable and wired.
220
-
221
- ### 3.5 Duration prediction: ship it (subject to OQ-1)
222
-
223
- `duration_head` is an optional constructor component and **is** in the pack (0.004 GB). Usage:
224
- omit `num_frames` and pass `min_seconds` / `max_seconds`; the head predicts the duration.
225
- This is a real differentiator and we ship it — **conditional on OQ-1 (§10), which is currently
226
- unresolved.**
 
 
 
 
 
 
 
 
227
 
228
  ---
229
 
230
  ## 4. Decision 4 — LoRA: the headline feature
231
 
232
- User-uploadable LoRA at inference time is the reason this app exists. It must be the most
233
- robust thing in it.
234
 
235
  ### 4.1 Verified mechanics
236
 
237
- - All three generation pipelines (`LTX2Pipeline`, `LTX2ConditionPipeline`,
238
- `LTX2ImageToVideoPipeline`) are declared as
239
  `class LTX2…Pipeline(DiffusionPipeline, FromSingleFileMixin, LTX2LoraLoaderMixin)`.
240
- - `LTX2LoraLoaderMixin` sets `_lora_loadable_modules = ["transformer", "connectors"]`. Note
241
- **`connectors`** — a LoRA may legitimately touch the text-conditioning path, not just the
242
- transformer.
243
  - **Both key conventions load.** `lora_state_dict()` detects
244
  `is_non_diffusers_format = any(k.startswith("diffusion_model.") for k in state_dict)` and runs
245
  `_convert_non_diffusers_ltx2_lora_to_diffusers(state_dict)`; separately it detects
246
  `has_connector = any(k.startswith("text_embedding_projection.") for k in state_dict)` and
247
- converts those into the connectors sub-dict. So ComfyUI-style and diffusers-style checkpoints
248
- both work.
249
  - `lora_state_dict(..., return_lora_metadata=True)` returns the file's `__metadata__`.
250
 
251
- ### 4.2 The danger: validation is almost non-existent upstream
252
 
253
- The **only** built-in check is a substring test, which raises
254
- `ValueError("Invalid LoRA checkpoint. Make sure all LoRA param names contain `'lora'` substring.")`.
255
 
256
- A LoRA built for a *completely different architecture* passes that check — it has `lora` in its
257
- key names — and then attaches to **zero** modules. Generation proceeds. Output looks like the
258
- base model. **No error is raised.** That is silent garbage, and it is precisely the class of
259
- failure the operator has told us to eliminate.
260
 
261
- We also have a real compatibility signal available. Reading the official LoRA's safetensors
262
- header directly: 3320 tensors, all under prefix `diffusion_model.`, peft-style
263
- `lora_A.weight` / `lora_B.weight`, targeting `to_q` / `to_k` / `to_v` / `to_gate_logits`,
264
- `adaln_single`, `audio_adaln_single`; and
265
  `__metadata__ = {'lora_rank': '450', 'lora_alpha': '450', 'model_version': '2.5.0', 'license': …}`.
266
- **`model_version` is machine-readable** and we should use it.
267
 
268
  ### 4.3 The validator (7 stages — each stage is a test unit)
269
 
@@ -271,94 +351,202 @@ Accepts either a **Hub repo-id** or a **direct file upload**.
271
 
272
  | # | Stage | Rule | On failure |
273
  |---|---|---|---|
274
- | 1 | **Size bound** | reject above a configured max | refuse before any download/parse |
275
- | 2 | **Header-only parse** | read the safetensors header, **never** load the tensor payload to validate | — |
276
- | 3 | **Format** | must be safetensors with a parseable header | specific message: not-safetensors vs unparseable-header |
277
- | 4 | **Version** | read `__metadata__.model_version`; compare to 2.5 | **warn** on absent, **reject** on explicit mismatch |
278
- | 5 | **Convention** | classify keys: `diffusion_model.` / `transformer.` / `text_embedding_projection.` / unknown | on unknown, the error **names the prefixes actually found** |
279
- | 6 | **Pairing** | require `lora_A`/`lora_B` (or `lora_down`/`lora_up`) to pair up | name the unpaired keys |
280
- | 7 | **Attach assertion** | **after** `load_lora_weights`, assert the adapter attached to **> 0** modules | raise a specific error if zero — *this is the anti-silent-garbage check* |
 
 
 
 
281
 
282
- Stage 2 is a hard rule, not a preference: a multi-GB upload must never be fully materialised
283
- just to find out it is the wrong architecture.
284
 
285
- Stage 7 is the one that upstream does not do for us and the one that actually protects the
286
- user. It must run on every load, including the built-in LoRAs.
 
 
 
 
287
 
288
  ---
289
 
290
  ## 5. Decision 5 — Resilience doctrine
291
 
292
- Five rules. These are binding on all code in the repo.
293
 
294
- ### 5.1 Pin every dependency, with upper bounds
295
 
296
- Evidence from the predecessor's `requirements.txt`: `spaces>=0.30.0` and `torch>=2.4.0` — floors
297
- with **no ceiling** — alongside exactly one hard pin, `torchaudio==2.10.0`, carrying this
298
- comment:
299
 
300
  > *"HF Spaces ships torch==2.10.0; bare 'torchaudio' lets pip pick an older wheel whose C++ ABI
301
  > doesn't match → import crashes the whole audio code path… Pin the matching version."*
302
 
303
- That comment is a post-mortem. It documents a break that already happened, and the fix was
304
- applied to exactly one line while ~50 other unbounded lines were left. Every direct dependency
305
- here gets a floor **and** a ceiling. `requirements.txt` is pending exact pins from the lead and
306
- is deliberately not written yet.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
307
 
308
  ### 5.2 One central model-file indirection
309
 
310
- Every Hub repo-id, revision, subfolder and filename lives in **one** module —
311
- `ltx25_studio/model_files.py` — as data, never inlined at a call site. An upstream rename must
312
- be a one-line change in one file.
313
 
314
- A missing file must raise an error naming **both the file and its URL**. This is not
315
- hypothetical: Lightricks deleted a 2.3 checkpoint from the Hub with no warning and broke a live
316
- Space. When it happens again, the error message should tell the operator exactly which URL to
317
- go look at.
318
 
319
- Pinning a **revision** (commit SHA) per repo is part of this module's job — a repo whose content
320
- can change under us is not pinned just because its name is stable.
321
 
322
- ### 5.3 Never return an empty output
323
 
324
  Any generation that completes without producing a file **must** raise a specific, actionable
325
  error that the UI renders visibly.
326
 
327
- The 2.3 outage hid for a **week** because a mid-graph load failure returned an empty video path,
328
- and the UI rendered an empty path as a silent form reset. The user saw "nothing happened". No
329
- error, no log, no signal. Structurally: a success path that can return a falsy artifact is a
330
- bug, regardless of whether it is currently reachable.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
331
 
332
- ### 5.4 Read audio parameters from config, never hardcode
333
 
334
- Mel/audio parameters come from `audio_vae.config` at runtime:
335
- `audio_sampling_rate`, `audio_hop_length`, `audio_mel_bins`, `audio_vae_mel_compression_ratio`,
336
  `audio_vae_temporal_compression_ratio`, `audio_latent_channels`.
337
 
338
- This mirrors what the pipeline itself does — its constructor reads
339
- `self.vae.spatial_compression_ratio`, `self.vae.temporal_compression_ratio`,
340
- `self.audio_vae.mel_compression_ratio`, `self.audio_vae.temporal_compression_ratio` and
341
- `self.transformer.config.patch_size` from the loaded components, using literals only as
342
- fallbacks when a component is absent. We follow the same discipline.
343
 
344
  ### 5.5 Enforce and surface the hard constraints
345
 
346
  | constraint | source | handling |
347
  |---|---|---|
348
- | `height % 32 == 0` and `width % 32 == 0` | enforced upstream: `if height % 32 != 0 or width % 32 != 0: raise ValueError(...)` in all three pipelines *and* in the latent-upsample pipeline | validate **before** the call, snap in the UI, show the snapped value |
349
- | `num_frames % 8 == 1` | **not** hard-raised upstream; it falls out of `latent_num_frames = (num_frames - 1) // self.vae_temporal_compression_ratio + 1`, which silently floors a bad value | validate ourselves. **Derive the modulus from `vae.temporal_compression_ratio`, do not hardcode 8** — same principle as §5.4 |
350
- | image conditioning CRF | `LTX2_5_IMAGE_CRF = 18` for 2.5, vs the generic `DEFAULT_IMAGE_CRF = 33` | 2.5 image conditioning uses **18**; the default is wrong for us |
351
- | audio/video length agreement | audio and video frame counts are computed independently (§3.1c) | validate they agree; raise rather than emit a desynced file |
 
 
 
 
 
 
 
 
 
352
 
353
  **Verified constants** (`diffusers/pipelines/ltx2/utils.py`):
354
 
355
- ```python
356
- DISTILLED_SIGMA_VALUES = [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875] # 8 steps, CFG=1
357
- STAGE_2_DISTILLED_SIGMA_VALUES = [0.909375, 0.725, 0.421875] # 3 steps
358
  ```
 
 
 
 
 
 
 
 
359
 
360
- These live in `constraints.py` as re-exports/asserts against the installed diffusers, so a
361
- schedule change upstream fails a test instead of silently changing output quality.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
362
 
363
  ---
364
 
@@ -367,19 +555,33 @@ schedule change upstream fails a test instead of silently changing output qualit
367
  | | |
368
  |---|---|
369
  | **Space** | `techfreakworm/LTX2.5-Studio`, **private** |
370
- | **SDK** | gradio |
371
- | **Hardware** | ZeroGPU — `suggested_hardware: zero-a10g` (the identifier HF uses); account is PRO |
372
- | **Weights** | README frontmatter `preload_from_hub:` (the mechanism the 2.3 Space uses) |
373
- | **Secret** | `HF_TOKEN` — required, because `Lightricks/LTX-2.5-Diffusers` is gated |
 
 
374
  | **Git** | GitHub `git@github.com:techfreakworm/ltx2.5-studio.git`; Space and GitHub stay in sync |
375
 
376
- Naming precedent in the operator's account: `techfreakworm/LTX2.3-Studio`, `wan-studio`
377
- (private), `SAM3-cutout-studio` (private).
 
 
378
 
379
- **`preload_from_hub` must list only the live shards** — see §2.1 Trap A. Getting this list wrong
380
- does not fail fast; it fails after tens of GB of download.
381
 
382
- ### 6.1 🚨 BLOCKER — repo gate not accepted
 
 
 
 
 
 
 
 
 
 
 
383
 
384
  `Lightricks/LTX-2.5-Diffusers` is gated (`gated: auto`) and the operator's token has **not**
385
  accepted that gate. Verified response:
@@ -391,227 +593,317 @@ click resolves it:
391
 
392
  **https://huggingface.co/Lightricks/LTX-2.5-Diffusers**
393
 
394
- Until it is accepted, nothing that touches real weights can be built or tested — local or Space.
395
- Escalated to the orchestrator.
396
 
397
  ---
398
 
399
  ## 7. Architecture
400
 
401
- ### 7.1 Module layout (proposed)
402
 
403
  ```
404
- app.py # entrypoint: local + Space. Thin — builds UI, launches.
405
  ltx25_studio/
406
- __init__.py
407
- config.py # runtime config: device/dtype selection, paths, env, retention policy
408
- model_files.py # §5.2 — THE single source of truth for repo ids, revisions,
409
- # subfolders, filenames, expected sizes. Data only, no logic.
410
- errors.py # typed exception hierarchy; every error carries an actionable message
411
- constraints.py # §5.5 — dimension/frame validation & snapping, CRF, sigma schedules
412
- pipelines.py # pipeline construction, component wiring, caching, offload policy
413
- conditioning.py # UI inputs -> list[LTX2VideoCondition]; multishot anchoring (§3.3)
414
- audio.py # waveform <-> mel <-> audio_vae latents; all params from config (§5.4)
415
- modes.py # §3.2 registry: 11 modes as data + per-mode kwarg builders
416
- generate.py # orchestration: mode -> stage 1 -> optional stage 2 -> outputs.
417
- # Owns the never-empty-output guarantee (§5.3).
418
- outputs.py # encode/mux video+audio, audio-only export, retention/eviction
 
 
 
 
 
 
 
419
  lora/
420
- __init__.py
421
- inspect.py # header-only safetensors parse; metadata + key-convention detection
422
- validate.py # the 7-stage validator (§4.3 stages 1-6)
423
- apply.py # load_lora_weights + the post-attach assertion (§4.3 stage 7)
424
  ui/
425
- __init__.py
426
  blocks.py # Gradio Blocks assembly
427
  components.py # reusable widgets (condition list, LoRA picker, constraint hints)
428
- handlers.py # thin adapters: UI event -> generate.py -> UI update. No logic.
429
  ```
430
 
431
  ### 7.2 Layering rules
432
 
433
  - **Dependencies point inward.** `ui/` may import anything; `generate.py` must not import `ui/`.
434
  `model_files.py`, `errors.py` and `constraints.py` import nothing from the package.
435
- - **`modes.py` is data, not branching.** Adding a mode should mean adding a registry entry, not
436
- adding an `if` to `generate.py`.
437
- - **`ui/handlers.py` contains no business logic.** Every handler is: marshal inputs → call one
438
- function in `generate.py` or `lora/` → marshal outputs. This is what makes the app testable
439
- without a GPU and without Gradio.
440
- - **Errors cross layers as typed exceptions**, and the UI is the only place that turns an
441
- exception into a string for a human. `generate.py` never returns an error-shaped success.
442
-
443
- ### 7.3 Rationale for the shape
444
-
445
- The predecessor put backend, models, modes and workflow in four flat top-level modules with
446
- `ui.py` importing all of them. That works until the failure modes get interesting: the empty-
447
- output bug (§5.3) lived precisely in the seam between "workflow returned something" and "UI
448
- rendered it", and there was no layer boundary at which to assert. Here, `generate.py` owns that
449
- invariant and can be tested for it directly.
450
 
451
- ---
452
 
453
- ## 8. VRAM and offload
454
-
455
- **The weights do not fit in VRAM.** ~72 GB of components against a ZeroGPU allocation an order
456
- of magnitude smaller. Therefore:
457
-
458
- - **`enable_model_cpu_offload()` is mandatory**, not a tuning option. It is part of pipeline
459
- construction in `pipelines.py`, not a user-facing toggle.
460
- - **Peak GPU residency ≈ transformer (37.98 GB) + activations.** The transformer is the
461
- irreducible resident component during denoising; everything else (text encoder, VAEs, vocoder,
462
- upsamplers, decoder) can be paged in and out around it.
463
- - **The text encoder is the second-largest component (23.9 GB) and is needed only once**, before
464
- denoising. It should be evicted after prompt encoding rather than held.
465
- - **Stage 2 (latent upsample) is a separate pipeline** — `LTX2LatentUpsamplePipeline` — which
466
- means it is a separate residency window and can be skipped entirely for the audio-only modes
467
- (§3.2 mitigation).
468
- - **Exact ZeroGPU numbers pending** from the lead; the offload strategy above does not depend on
469
- them, but the maximum supported resolution × frame count does.
470
-
471
- **Local (Apple Silicon / MPS) is a different regime.** MPS has no `enable_model_cpu_offload()`
472
- equivalent with the same semantics and unified memory changes the arithmetic entirely. Device
473
- strategy is selected in `config.py` and the local path is expected to be capability-limited
474
- rather than identical. Do not assume a code path that works on ZeroGPU works on MPS.
475
 
476
- ---
477
 
478
- ## 9. Testing strategy
 
479
 
480
- We are running an **H3 TDD harness**. The gate is:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
481
 
482
  ```
483
- .venv/bin/python -m pytest -q
 
 
 
 
 
 
 
 
 
 
 
 
 
 
484
  ```
485
 
486
- **Ownership:** a separate teammate owns `tests/**`. The implementer does not create or edit
487
- anything under `tests/`. This plan's job is to make the units *testable*; writing the tests is
488
- someone else's.
489
 
490
- ### 9.1 What must be testable without a GPU and without weights
 
 
 
 
 
491
 
492
- This is the design constraint that shapes §7. The following must be exercisable with no CUDA,
493
- no downloaded weights, and no network:
 
494
 
495
- | unit | how it is tested without weights |
496
- |---|---|
497
- | `constraints.py` | pure functions over ints — dimension snapping, frame-count validation, sigma schedule shape |
498
- | `model_files.py` | data-only assertions: every entry has a repo, revision, subfolder, filename; no duplicates; the §2.3 subset sums to the stated total |
499
- | `lora/inspect.py` | synthesised safetensors **headers** — small fixtures, no real LoRA needed (this is why stage 2 is header-only) |
500
- | `lora/validate.py` | the 7 stages against crafted good/bad headers, including the wrong-architecture case |
501
- | `modes.py` | registry completeness (all 11 present), and each mode's kwarg builder produces the expected kwargs for given inputs |
502
- | `errors.py` | every error type produces a message containing the actionable detail (filename, URL, found-prefixes) |
503
- | `outputs.py` retention | eviction logic over a temp dir |
504
- | `generate.py` never-empty invariant | inject a stub pipeline that returns nothing → assert a specific exception, **not** an empty path |
505
 
506
- ### 9.2 GPU-gated tests
507
 
508
- Anything that needs real weights is marked and skipped by default (the predecessor used a
509
- `gpu` marker plus a `--gpu` flag; same pattern is appropriate here). CI and the default local
510
- run must be green with zero weights on disk.
511
 
512
- ### 9.3 The invariant tests that matter most
 
 
513
 
514
- Three tests earn their keep more than all the others:
 
 
 
515
 
516
- 1. **A LoRA for the wrong architecture is rejected** — and specifically, that it is rejected at
517
- stage 7 if it survives stages 1–6 (§4.3).
518
- 2. **A generation that produces no file raises** — never returns an empty path (§5.3).
519
- 3. **The `preload_from_hub` list contains only live shards** — assertable against the index JSON
520
- without downloading anything (§2.1).
521
 
522
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
523
 
524
- ## 10. Open questions and implementer push-back
525
 
526
- Recorded rather than silently resolved. **OQ-1 is blocking.**
 
 
527
 
528
- ### 🚨 OQ-1 (BLOCKING) — "pin everything" and "ship duration_head" are currently in conflict
 
 
529
 
530
- **Evidence.** Inspecting an installed **released** `diffusers==0.37.1`:
 
531
 
532
- - `diffusers/pipelines/__init__.py` exports exactly four LTX2 pipelines:
533
- `LTX2Pipeline`, `LTX2ConditionPipeline`, `LTX2ImageToVideoPipeline`,
534
- `LTX2LatentUpsamplePipeline`. Plus models `AutoencoderKLLTX2Video`,
535
- `AutoencoderKLLTX2Audio`, `LTX2VideoTransformer3DModel`.
536
- - **Not present at any version on disk:** `LTX2InContextPipeline`, `LTX2ReferenceCondition`,
537
- `LTX2HDRPipeline`, `LTX2HDRReferenceCondition`, `LTX2VideoDiffusionDecodePipeline`,
538
- `LTX2DurationHead`, `LTX2VocoderWithBWE`, `LTX2VideoHDRProcessor`.
539
- - `LTX2ConditionPipeline.__init__` in 0.37.1 takes exactly:
540
- `scheduler, vae, audio_vae, text_encoder, tokenizer, connectors, transformer, vocoder`.
541
- There is **no `duration_head` parameter and no `prompt_enhancer` parameter**. Grepping the
542
- whole `pipelines/ltx2/` tree for `duration_head`, `prompt_enhancer`, `min_seconds`,
543
- `max_seconds`, `enable_prompt_enhancement` returns **zero hits**.
544
- - `LTX2_5_IMAGE_CRF` also does not exist in 0.37.1 — only `DISTILLED_SIGMA_VALUES` and
545
- `STAGE_2_DISTILLED_SIGMA_VALUES` are in `utils.py` (both matching §5.5 exactly).
546
 
547
- **Conclusion.** The export list in the brief is from `diffusers` **main (unreleased)**, not from
548
- any published release. That is not a problem in itself — but it collides head-on with §5.1
549
- "pin every dependency with upper bounds". As written, we cannot both pin diffusers to a release
550
- *and* ship `duration_head` (§3.5), the diffusion decoder (§2.3, 0.834 GB) or `LTX2_5_IMAGE_CRF`
551
- (§5.5).
552
 
553
- **Recommendation — pin to a git commit SHA, not to `main`:**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
554
 
555
  ```
556
- diffusers @ git+https://github.com/huggingface/diffusers@<exact-sha>
557
  ```
558
 
559
- A SHA is *more* pinned than a version range, not less: it is byte-for-byte reproducible, it
560
- cannot drift, and it gets us the full 2.5 feature set. It costs a git clone at build time and it
561
- means we own upgrades manually — both acceptable. The alternative, `diffusers>=0.37,<0.38`, is a
562
- smaller build but silently drops duration prediction, the diffusion decoder, HDR and in-context.
563
 
564
- **This needs the lead's ruling before `requirements.txt` is written or `pipelines.py` is built,**
565
- because it decides which components `model_files.py` lists and which modes `modes.py` can offer.
566
 
567
- ### OQ-2 — the `diffusion_decoder` in the disk budget has no consumer at 0.37.1
 
 
 
 
 
 
 
 
 
 
 
 
568
 
569
- The 0.834 GB `diffusion_decoder/` is used via `LTX2VideoDiffusionDecodePipeline`, which does not
570
- exist in the released package (see OQ-1). If OQ-1 resolves to a release pin, `diffusion_decoder`
571
- and `duration_head` should come out of the §2.3 subset (−0.838 GB) and the plan should say so
572
- rather than shipping weights nothing loads.
573
 
574
- ### OQ-3 — `num_frames % 8 == 1` is a derived rule, not an upstream check
 
575
 
576
- Upstream computes `latent_num_frames = (num_frames - 1) // self.vae_temporal_compression_ratio + 1`
577
- and **silently floors** a non-conforming value — there is no raise. Two implications:
578
 
579
- 1. We must validate it ourselves; nothing upstream will catch it.
580
- 2. The `8` is `vae.temporal_compression_ratio`, read from the loaded VAE with `8` only as a
581
- fallback default. Hardcoding `8` in our validator would violate §5.4's own principle. §5.5 is
582
- written to derive it; flagging it because the brief stated it as a literal.
 
 
 
 
 
 
 
 
 
 
 
 
583
 
584
- ### OQ-4 — text encoder is Gemma-**3**, not Gemma-4
585
 
586
- The pipeline constructors type the text encoder as `Gemma3ForConditionalGeneration` with a
587
- `GemmaTokenizer | GemmaTokenizerFast`. The brief says "Gemma4-12B". Cosmetic for the disk budget
588
- (the 23.92 GB figure is unaffected), but `model_files.py` and any user-facing label should not
589
- claim a model family the loader does not use. Note this is *separate* from the prompt enhancer,
590
- which genuinely is `google/gemma-4-E2B-it` (§3.4) — that one is a different model and is not in
591
- the pack anyway.
592
 
593
- ### OQ-5 — audio/video length desync: which direction?
 
594
 
595
- §3.1(c) states audio frame count is inferred from supplied latents' shape. In 0.37.1's
596
- `LTX2Pipeline` the derivation reads `audio_num_frames = round(duration_s * audio_latents_per_second)`
597
- — derived from *duration*, not from the latent shape. The desync risk is real either way, so
598
- §5.5's "validate that they agree" stands regardless; but the exact mechanism should be re-read on
599
- whichever diffusers revision OQ-1 selects before `audio.py` is written.
 
 
 
 
 
 
 
 
 
 
 
 
 
600
 
601
- ### OQ-6 — output retention is unspecified
602
 
603
- §2.5 leaves ~78 GB of headroom, shared between the pip environment, user LoRA uploads and
604
- outputs. Video outputs are large and a Space that never evicts will fill. Proposing bounded-bytes
605
- retention with oldest-first eviction in `outputs.py`, configured in `config.py`. Needs a number.
606
 
607
  ---
608
 
609
  ## 11. Build order
610
 
611
  1. ✅ **This document**, committed.
612
- 2. **Scaffold** — venv, `pyproject.toml` (metadata + ruff + pytest with `pythonpath = ["."]`),
613
- package with importable stubs matching §7.1, `.gitignore`. **No app logic.**
614
- 3. **Blocked on OQ-1** — `requirements.txt` (pins from the lead) and `model_files.py`.
615
- 4. **Blocked on §6.1** — anything that touches real weights.
616
- 5. Then, TDD, innermost-out: `errors` → `constraints` → `model_files` → `lora/` → `modes` →
617
- `generate` → `outputs` → `ui`.
 
4
  **Branch:** `soma/ltx25-studio` · **Remote:** `git@github.com:techfreakworm/ltx2.5-studio.git`
5
  **Target:** private Hugging Face Space `techfreakworm/LTX2.5-Studio` (gradio SDK, ZeroGPU).
6
 
7
+ This document is the contract for the build. Every section is either a **decision** (settled,
8
+ build to it) or a **risk** (§10, needs measurement not argument). Where a claim was verified
9
+ against real source or real repo metadata, the evidence is cited inline so a future reader can
10
+ re-check it instead of trusting it.
11
 
12
  **Prime directive:** this app must not repeat the failure modes of its predecessor
13
+ (`ltx2.3-AIO-generator`). Those were not model failures. They were *packaging, pinning, and
14
+ silent-failure* failures. §5 and §8 are the sections the build will be judged on.
15
+
16
+ **A note on how to read this document.** Several statements here contradict things that are
17
+ true of *other* diffusion apps, or true of LTX-2.5 on a normal GPU box but false on ZeroGPU. The
18
+ biggest is §8: **`enable_model_cpu_offload()` must not be used.** If a future reader's instinct
19
+ says otherwise, §8.4 explains why the instinct is wrong here.
20
 
21
  ---
22
 
23
+ ## 1. Decision 1 — Integration path: **Diffusers**, pinned to an exact SHA
24
 
25
  We build on `diffusers`, loading `Lightricks/LTX-2.5-Diffusers`.
26
 
 
34
 
35
  ### 1.2 Why C
36
 
37
+ **Path A is a heavy, fragile install for a Space.** It needs `uv sync` of a git workspace,
38
+ CUDA >= 12.7, torch ~= 2.7. A Space build that resolves a git workspace at image-build time is
39
+ a build we cannot reproduce or pin.
 
 
 
 
 
 
 
 
40
 
41
  **Path B is the documented cause of the fragility we are told to avoid.** The 2.3 project's
42
+ `requirements.txt` vendors an entire ComfyUI runtime — ~50 dependencies, nearly all unpinned:
43
+ `spandrel`, `torchsde`, `kornia`, `PyOpenGL`, `glfw`, `SQLAlchemy`, `alembic`, `Mako`, `blake3`,
44
+ `comfyui-frontend-package`, `comfyui-workflow-templates`, `comfyui-embedded-docs`, plus a
45
+ hand-maintained block of *custom-node deps we discovered are required but aren't auto-installed
46
+ by their respective nodes* (`gguf`, `imageio_ffmpeg`, `opencv-python`, `matplotlib`,
47
+ `scikit-image`, `onnxruntime`, `yt-dlp`). That comment block is the evidence: the dependency
48
+ surface was discovered by breakage, not by design. Diffusers replaces all of it with ~16 pinned
49
+ direct deps and zero custom-node surface.
50
 
51
  **Decisive coupling to §2.** The `*-comfy-int8-convrot` quantised weights are **ComfyUI-only**
52
+ — the model card says so. Choosing A or C means those 36.9 GB of int8 weights are unusable to
53
+ us. Path choice and disk budget are one decision, not two.
54
+
55
+ ### 1.3 The pinned revision — and why a release tag will not do
56
+
57
+ ```
58
+ diffusers @ git+https://github.com/huggingface/diffusers@ac56fa2f25c1c15b4cbaa1e0b3255e9bc38c06dc
59
+ ```
60
+
61
+ HEAD at 2026-08-19T12:11:04Z. Contains the LTX-2.5 merge (PR #14447, 2026-08-11) plus #14453
62
+ and #14462. PEP 508 direct reference — **not** the legacy `#egg=` form.
63
+
64
+ **A SHA is more pinned than a version range, not less.** It is byte-for-byte reproducible and
65
+ cannot drift. It costs a git clone at build time and manual upgrades; both are acceptable.
66
 
67
+ **The latest release (0.39.0) does not have 2.5, and it fails LATE.** `LTX2Pipeline` imports
68
+ fine on 0.39.0 — so the app starts, the UI renders, weights begin downloading — and only then
69
+ does it fail on a missing `temporal_latent_upsampler`. A failure that happens after 70 GB of
70
+ download is the worst kind.
71
 
72
+ **Therefore: a startup fail-fast assertion.**
73
+
74
+ ```python
75
+ assert diffusers.__version__ == "0.40.0.dev0", (
76
+ f"LTX-2.5 needs the pinned diffusers SHA (reports 0.40.0.dev0); got {diffusers.__version__}. "
77
+ "See docs/PLAN.md §1.3."
78
+ )
79
+ ```
80
+
81
+ This converts a late, confusing, expensive failure into an immediate, obvious one. It lives in
82
+ `preflight.py` (§7.1) and it is a test case (§9.3).
83
+
84
+ ### 1.4 The single-file escape hatch: investigated and REJECTED
85
 
86
  `diffusers/loaders/single_file_utils.py` does contain an LTX-2 route:
87
  `CHECKPOINT_KEY_NAMES["ltx2"]` → `model_type = "ltx2-dev"` → config repo
88
+ `{"pretrained_model_name_or_path": "Lightricks/LTX-2"}`. Note the repo: **`LTX-2`, i.e. 2.0,
89
+ not 2.5**. The converters it ships are only `convert_ltx2_transformer_to_diffusers`,
90
+ `convert_ltx2_vae_to_diffusers` and `convert_ltx2_audio_vae_to_diffusers` — all keyed on
91
+ `LTX_2_0_*` rename dicts.
 
 
92
 
93
  There is **no** single-file converter for 2.5's `connectors`, `diffusion_decoder`, `vocoder`,
94
+ `duration_head` or `latent_upsampler`. Even a successful load would give us a pipeline missing
95
+ the components that make 2.5 what it is. Settled; do not revisit without new upstream
96
+ converters.
97
 
98
  ---
99
 
100
+ ## 2. Decision 2 — Disk, RAM and dtype budget
101
 
102
+ Three separate resource ceilings, and the tightest one is not the obvious one.
103
+
104
+ | resource | ceiling | our load | margin |
105
+ |---|---|---|---|
106
+ | **VRAM** (`xlarge`) | ~95 GiB | 67.2 GiB weights | ~28 GiB for activations — **comfortable** |
107
+ | **CPU RAM** | **~68 GB** (community cgroup observation, *not* an official figure — HF publishes ZeroGPU RAM as `"-"` / "Dynamic resources") | ~72 GB | **marginal — risk #1** |
108
+ | **Ephemeral disk** | 150 GB, **wiped on every restart/rebuild** | ~144 GB transient peak | **marginal — risk #1** |
109
+
110
+ **VRAM was never the problem.** The problems are RAM and transient disk.
111
 
112
  ### 2.1 Trap A — duplicate shard sets
113
 
114
+ Two folders ship the *same weights twice*, in two different shardings:
115
 
116
  | folder | set 1 | set 2 | note |
117
  |---|---|---|---|
118
  | `transformer/` | 4 shards: 9.932 + 9.919 + 9.986 + 8.138 = **37.975 GB** | 8 shards: 4.991 + 4.944 + 4.994 + 4.928 + 4.977 + 4.995 + 4.994 + 3.163 = **37.986 GB** | only **one** `diffusion_pytorch_model.safetensors.index.json` |
119
  | `connectors/` | single file: **6.344 GB** | 2 shards: 5.000 + 1.344 = **6.344 GB** | same |
120
 
121
+ A naive `snapshot_download` of those folders pulls **both** sets (+44.33 GB of pure
122
+ duplication). `from_pretrained` reads the index JSON and fetches only the referenced shards.
 
 
 
123
 
124
+ **Binding consequences:** (1) we rely on the index, never hand-rolling a `snapshot_download` of
125
+ `transformer/`; (2) any `preload_from_hub:` entry must name **only the live shards**. A wrong
126
+ list here does not fail fast — it fails after tens of GB, and on a 150 GB cap it evicts us.
 
127
 
128
  ### 2.2 Trap B — `transformer_full/` is the DEV transformer
129
 
130
+ A further **37.975 GB**, the *dev* (non-distilled) transformer. We ship the distilled 8-step
131
+ path, so we do not ship it. Documented as opt-in (§2.5), not deleted from our mental model.
 
132
 
133
+ ### 2.3 The bf16 subset we ship
134
 
135
  | component | GB | why |
136
  |---|---:|---|
137
  | `transformer/` (distilled, one shard set) | 37.975 | the model |
138
+ | `text_encoder/` (5 shards: 4.961 + 4.967 + 4.884 + 4.986 + 4.121) | 23.919 | Gemma-4 unified (§10 R-4) |
139
  | `connectors/` | 6.344 | text → transformer conditioning; also a LoRA target |
140
+ | `vae/` | 1.452 | video VAE (fallback decoder — §7.4) |
141
  | `audio_vae/` | 0.107 | audio VAE (mel ↔ latent) |
142
  | `vocoder/` | 0.258 | mel → waveform |
143
  | `latent_upsampler/` | 0.996 | stage-2 spatial upscale |
144
  | `temporal_latent_upsampler/` | 0.262 | stage-2 temporal upscale |
145
+ | `diffusion_decoder/` | 0.834 | **the native 2.5 decoder** — §7.4 |
146
+ | `duration_head/` | 0.004 | duration prediction |
147
  | `tokenizer/` | 0.032 | |
148
+ | **TOTAL** | **72.185** | **≈ 72.19 GB = 67.2 GiB** |
149
 
150
  ### 2.4 What we exclude, and why
151
 
 
153
  |---|---:|---|
154
  | `transformer_full/` | 37.975 | the DEV transformer; not needed for the distilled 8-step path |
155
  | duplicate shard sets (transformer 8-shard + connectors 2-shard) | 44.330 | Trap A — same weights, different sharding |
156
+ | `ltx-2.5-22b-distilled-lora-450-bf16.safetensors` (repo root) | 9.699 | the distilled LoRA for **DEV** workflows; useless when we ship the distilled transformer |
157
 
158
+ **Arithmetic check:** 72.185 + 37.975 + 44.330 + 9.699 = **164.189 GB**, closing exactly against
159
+ the repo's stated 164.19 GB across 56 files. The accounting is complete; there is no
160
  unexplained remainder.
161
 
162
+ ### 2.5 The transient disk peak
163
+
164
+ Startup holds **two** copies: the HF cache (~72 GB) *and* the packed copy ZeroGPU writes to
165
+ `~/.zerogpu/tensors` (~72 GB) ≈ **144 GB against a 150 GB cap**. The 2.3 project was evicted
166
+ with exactly this: `"Workload evicted, storage limit exceeded (150G)"`.
167
+
168
+ **Mitigating fact:** the packer auto-prunes the HF cache blobs after packing —
169
+ `ZEROGPU_MMAP_AUTOPRUNE_PATTERN` = `~/.cache/huggingface/hub/models--*/blobs/*` — logging
170
+ `"Cleaned NGB of tensor files ... after packing"`. So the 144 GB peak is **transient**. It is
171
+ still a peak, and 6 GB of margin is not much. This is why §5.6's preflight measures free disk
172
+ and says a number rather than letting the platform evict us anonymously.
173
+
174
+ Remaining headroom after packing: `150 − 72.19 ≈ 77.8 GB` for the pip environment, user LoRA
175
+ uploads and outputs. `transformer_full/` (37.98 GB) fits in that — which is why it is opt-in
176
+ rather than impossible — but enabling it leaves ~40 GB and makes output eviction urgent.
177
+
178
+ ### 2.6 dtype and quantization — **default bf16**, quantization is a documented fallback
179
+
180
+ Packed weight size decides which ZeroGPU tier we land on, and the tier decides quota cost. The
181
+ threshold is `xlarge_threshold = 42.00 GiB`: **exceed it and ZeroGPU silently auto-promotes to
182
+ `xlarge` and doubles quota cost, with no code change.**
183
+
184
+ | config | approx total | packed vs 42.00 GiB | tier | quota | verdict |
185
+ |---|---:|---|---|---|---|
186
+ | **bf16 everything** | 72.19 GB (67.2 GiB) | over | `xlarge` | **2×** | **DEFAULT** — documented by the model card, matches the official Lightricks Space pattern; RAM/disk marginal |
187
+ | fp8 transformer (~19) + bf16 TE (~24) + rest (~10) | ~53 GB (~49.3 GiB) | over | `xlarge` | 2× | comfortable RAM/disk, no quota win |
188
+ | fp8 transformer + int8 TE | ~41 GB (~38.2 GiB) | **under** | `large` | **1×** | best quota economics, highest accuracy risk |
189
 
190
+ **Why bf16 is the default and not the quantised config.** Shipping quantised by default would
191
+ trade a *known* uncertainty for an *unverified* one:
 
 
192
 
193
+ - The RAM ceiling (~68 GB) is a community cgroup observation, not an official number. It is
194
+ uncertain, but it is uncertainty about a configuration that upstream documents and that a
195
+ production Space already runs.
196
+ - Quantization carries its own **unquantified** risk on this specific platform. bitsandbytes
197
+ `Int8Params` / `Params4bit` and torchao both use **custom tensor subclasses**, and ZeroGPU's
198
+ packer tracks aliased tensors in a `WeakTensorKeyDictionary`. Whether it packs subclassed
199
+ tensors correctly is **unverified**. A packer that mishandles them fails in exactly the way
200
+ we cannot tolerate: quietly, as a quality regression.
201
+
202
+ So bf16 ships, `size="xlarge"` is **declared explicitly** rather than being silently promoted
203
+ (§8.5), and dtype/quantization is **one switch in `model_files.py`** so moving to fp8 or fp8+int8
204
+ is a one-line change. This is **technical risk #1** (§10), and it is to be settled by
205
+ measurement on the Space, not by argument.
206
+
207
+ ### 2.7 Output retention — 20 GB, evicted oldest-first
208
+
209
+ `outputs/` is capped at **20 GB**, bounded-bytes, oldest-first eviction, as a single constant in
210
+ `config.py` overridable by env. Swept **on startup as well as after each run**, logging what was
211
+ evicted.
212
+
213
+ Rationale: ~78 GB headroom minus the pip environment and a user LoRA or two. 20 GB is many
214
+ generations and leaves the margin the 2.3 Space did not have when it filled its disk and went
215
+ unhealthy.
216
 
217
  ---
218
 
219
+ ## 3. Decision 3 — Mode coverage: 11 UI modes over 4 real workflows
220
+
221
+ All 11 requested modes are reachable. **They are not 11 code paths**, and the plan says so
222
+ plainly — describing them as independent implementations would be dishonest and would produce a
223
+ much worse registry.
224
+
225
+ ### 3.1 The four real workflows
226
+
227
+ From `_workflow_map` in `diffusers/modular_pipelines/ltx2/modular_blocks_ltx25.py`:
228
 
229
+ ```
230
+ text2video {prompt}
231
+ image2video {image, prompt}
232
+ condition {conditions, prompt}
233
+ in_context {reference_conditions, num_frames, prompt}
234
+ ```
235
 
236
+ Our **11 UI modes are a presentation layer over those four**, plus the audio-latent input route
237
+ and the audio-only export. That is the honest description, and it is also the design that keeps
238
+ the registry small: `modes.py` is data over four builders, not eleven.
239
+
240
+ ### 3.2 Three facts that shape everything
241
 
242
  **(a) One conditioning mechanism covers both image and video.**
243
+ `LTX2VideoCondition.frames` is typed
244
  `PIL.Image.Image | list[PIL.Image.Image] | np.ndarray | torch.Tensor` — "the image (**or
245
+ video**) to condition the video on" — and carries `index` and `strength`. Image- and
246
+ video-conditioning are *the same code path* with a different payload. See §5.7 for the
247
+ `index` trap, which is severe.
 
248
 
249
  **(b) The model is jointly audio-visual. There is no video-off switch.**
250
+ The return type is `LTX2PipelineOutput(frames=..., audio=...)`, both non-optional tensors. Every
251
+ call spends video compute and returns a video. This is the most important cost fact in the app.
 
252
 
253
  **(c) Audio input is not a first-class kwarg.**
254
+ There is no `audio=` argument. Audio enters as `audio_latents` — upstream documents it as
255
+ *"Optional pre-encoded audio latents; random noise is used when not provided"* — combined with
256
+ `noise_scale`. `prepare_audio_latents` routes through `_create_noised_state`, whose body is
257
+ `noise_scale * noise + (1 - noise_scale) * latents`: img2img-style strength, where `1.0`
258
+ discards the input entirely.
 
 
259
 
260
+ ### 3.3 The mode registry
261
 
262
+ | # | Mode | Workflow | How it is reached | Cost note |
263
+ |---|---|---|---|---|
264
+ | 1 | text → video | text2video | `prompt` | first-class |
265
+ | 2 | image → video | condition | one `LTX2VideoCondition(frames=PIL, index=0)`, empty prompt | first-class |
266
+ | 3 | image + text → video | condition | as #2 with a prompt | first-class |
267
+ | 4 | video → video | condition | `LTX2VideoCondition(frames=<video>, strength<1.0)` | first-class |
268
+ | 5 | text → audio+video | text2video | #1 — the audio track is always produced | **the default shape** |
269
+ | 6 | image → audio+video | condition | #2 | first-class |
270
+ | 7 | image + text → audio+video | condition | #3 | first-class |
271
+ | 8 | text → audio | text2video | #1, export only the audio track | ⚠️ **still pays full video compute** |
272
+ | 9 | video → audio | condition | #4, export only the audio track | ⚠️ same |
273
+ | 10 | audio → audio | condition | `audio_latents` + `noise_scale`, export only audio | ⚠️ same |
274
+ | 11 | audio → video | condition | `audio_latents` + `noise_scale`, export only video | ⚠️ pays audio encode on top |
275
+
276
+ **Mandatory mitigation for modes 8–11:** run at the **minimum legal resolution**, **skip the
277
+ stage-2 spatial upscale**, export only the requested track. The UI must say in words that
278
+ audio-only modes still render video internally — a user who does not know that will read the
279
+ runtime as a bug.
280
+
281
+ ### 3.4 Multishot — **downgraded to honest**
282
+
283
+ There is **no multishot or scene API**. Multishot is prompt-structure driven only.
284
+
285
+ Worse: the 2.5 enhancer system prompts **explicitly instruct the model not to describe scene
286
+ cuts**, so prompt-level multishot actively fights the enhancer. We present it as *prompt-structure
287
+ driven, unverified, no API support* — a documented technique in the UI copy, not a feature with a
288
+ button that implies support we cannot back.
289
+
290
+ (Superseding an earlier draft of this plan that called it a first-class affordance. It is not.)
291
+
292
+ ### 3.5 Prompt enhancement: **unavailable by default; we ship no dead toggle**
293
+
294
+ `LTX25AutoBlocks` declares a `prompt_enhancer` component, and it appears in
295
+ `_optional_components`. But **the Diffusers pack has no `prompt_enhancer/` folder** — all 56
296
+ files enumerated. It would come from `google/gemma-4-E2B-it`, a separate model. So
297
+ `enable_prompt_enhancement=True` fails on our subset.
298
+
299
+ Shipping a toggle that cannot work is exactly the silent lie §5 exists to prevent. No UI
300
+ control until the component is fetchable and wired.
301
+
302
+ Recorded for whoever enables it later: `prompt_max_new_tokens=600`, greedy decoding,
303
+ `prompt_enhancement_seed` is **inert under LTX-2.5's greedy decoding**, and the system prompt is
304
+ auto-selected between `LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT` and `LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT`.
305
+
306
+ ### 3.6 Duration prediction: ships
307
+
308
+ `duration_head` is in the pack (0.004 GB) and in `_optional_components`. Omit `num_frames`, pass
309
+ `min_seconds` / `max_seconds`. See §5.7 for the `num_frames` default trap and the >1-prompt
310
+ restriction.
311
 
312
  ---
313
 
314
  ## 4. Decision 4 — LoRA: the headline feature
315
 
316
+ User-uploadable LoRA at inference time is why this app exists. It must be the most robust thing
317
+ in it.
318
 
319
  ### 4.1 Verified mechanics
320
 
321
+ - All generation pipelines are declared
 
322
  `class LTX2…Pipeline(DiffusionPipeline, FromSingleFileMixin, LTX2LoraLoaderMixin)`.
323
+ - `LTX2LoraLoaderMixin._lora_loadable_modules = ["transformer", "connectors"]`. Note
324
+ **`connectors`** — a LoRA may legitimately touch the text-conditioning path.
 
325
  - **Both key conventions load.** `lora_state_dict()` detects
326
  `is_non_diffusers_format = any(k.startswith("diffusion_model.") for k in state_dict)` and runs
327
  `_convert_non_diffusers_ltx2_lora_to_diffusers(state_dict)`; separately it detects
328
  `has_connector = any(k.startswith("text_embedding_projection.") for k in state_dict)` and
329
+ converts those into the connectors sub-dict.
 
330
  - `lora_state_dict(..., return_lora_metadata=True)` returns the file's `__metadata__`.
331
 
332
+ ### 4.2 The danger: upstream validation is almost non-existent
333
 
334
+ The **only** built-in check is a substring test:
335
+ `ValueError("Invalid LoRA checkpoint. Make sure all LoRA param names contain `'lora'` substring.")`
336
 
337
+ A LoRA for a *completely different architecture* passes that check — it has `lora` in its key
338
+ names — then attaches to **zero** modules. Generation proceeds. Output looks like the base
339
+ model. **No error is raised.** That is silent garbage, and it is precisely the class of failure
340
+ the operator has told us to eliminate.
341
 
342
+ We do have a real compatibility signal. Reading the official LoRA's safetensors header directly:
343
+ 3320 tensors, all under prefix `diffusion_model.`, peft-style `lora_A.weight`/`lora_B.weight`,
344
+ targeting `to_q`/`to_k`/`to_v`/`to_gate_logits`, `adaln_single`, `audio_adaln_single`; and
 
345
  `__metadata__ = {'lora_rank': '450', 'lora_alpha': '450', 'model_version': '2.5.0', 'license': …}`.
346
+ **`model_version` is machine-readable.**
347
 
348
  ### 4.3 The validator (7 stages — each stage is a test unit)
349
 
 
351
 
352
  | # | Stage | Rule | On failure |
353
  |---|---|---|---|
354
+ | 1 | **Size bound** | reject above a configured max | refuse before any download or parse |
355
+ | 2 | **Header-only parse** | read the safetensors header, **never** materialise the payload | — |
356
+ | 3 | **Format** | safetensors with a parseable header | distinct messages: not-safetensors vs unparseable-header |
357
+ | 4 | **Version** | read `__metadata__.model_version` | **warn** if absent, **reject** on explicit mismatch |
358
+ | 5 | **Convention** | classify: `diffusion_model.` / `transformer.` / `text_embedding_projection.` / unknown | on unknown, the error **names the prefixes actually found** |
359
+ | 6 | **Pairing** | `lora_A`/`lora_B` (or `lora_down`/`lora_up`) must pair | name the unpaired keys |
360
+ | 7 | **Attach assertion** | **after** `load_lora_weights`, assert the adapter attached to **> 0** modules | specific error if zero — *the anti-silent-garbage check* |
361
+
362
+ Stage 2 is a hard rule: a multi-GB upload must never be fully materialised just to discover it
363
+ is the wrong architecture. Stage 7 is the one upstream does not do for us and the one that
364
+ actually protects the user; it runs on **every** load, built-in LoRAs included.
365
 
366
+ ### 4.4 Fusing — the connectors trap
 
367
 
368
+ `fuse_lora` / `unfuse_lora` default to `components=["transformer"]` **only**.
369
+
370
+ **Connector LoRA layers are silently not fused** unless we pass
371
+ `components=["transformer", "connectors"]`. A LoRA that trains the text-conditioning path would
372
+ partially apply — worse than not applying, because it looks like it worked. We always pass both
373
+ components explicitly, and this is a test case.
374
 
375
  ---
376
 
377
  ## 5. Decision 5 — Resilience doctrine
378
 
379
+ Binding on all code in the repo.
380
 
381
+ ### 5.1 Pin every dependency, with upper bounds — and know what NOT to pin
382
 
383
+ Evidence from the predecessor: `spaces>=0.30.0` and `torch>=2.4.0` — floors with **no ceiling**
384
+ — alongside exactly one hard pin, `torchaudio==2.10.0`, carrying this comment:
 
385
 
386
  > *"HF Spaces ships torch==2.10.0; bare 'torchaudio' lets pip pick an older wheel whose C++ ABI
387
  > doesn't match → import crashes the whole audio code path… Pin the matching version."*
388
 
389
+ That comment is a post-mortem, applied to one line while ~50 stayed unbounded.
390
+
391
+ **But the inverse mistake is also real, and the 2.3 repo made it.** Its own `AGENTS.md:19` says
392
+ **do not pin `spaces`** — the platform pins its own version (live Spaces report 0.49.3 / 0.50.0 /
393
+ 0.50.4 while PyPI is 0.51.1) and a conflicting pin breaks pip resolution at build time. Its
394
+ `requirements.txt:2` then pins `spaces>=0.30.0` anyway. We follow the rule the repo wrote and
395
+ did not keep: **no `spaces` line in `requirements.txt`**, and the import is guarded in code
396
+ (`try: import spaces / except ImportError: spaces = None`) so local dev works.
397
+
398
+ The full pin set:
399
+
400
+ | package | pin | why |
401
+ |---|---|---|
402
+ | `torch` | `==2.11.0` | ZeroGPU supports exactly 2.8.0 / 2.9.1 / 2.10.0 / **2.11.0**; ceiling is 2.11.0. CUDA 12.8, sm_120 needs cu128+ |
403
+ | `torchvision` | `==0.26.0` | **trap:** `torchvision 0.28.0` hard-declares `torch==2.13.0`, so a floating torchvision silently upgrades torch past the ceiling |
404
+ | `torchaudio` | `==2.11.0` | must match torch exactly (the ABI break above) |
405
+ | `gradio` | `>=6.24.0,<7` | **5.x is END OF LIFE** (last stable 5.50.0, 2025-11-21) |
406
+ | `diffusers` | `@ git+…@ac56fa2f…` | §1.3 |
407
+ | `transformers` | `>=5.15.1,<6` | |
408
+ | `accelerate` | `>=1.14.0,<2` | |
409
+ | `peft` | `>=0.20.0,<0.21` | LoRA backend |
410
+ | `safetensors` | `>=0.8.0,<0.9` | |
411
+ | `huggingface_hub` | `>=1.28.0,<2` | |
412
+ | `hf_xet` | `>=1.6.0,<2` | |
413
+ | `imageio` | `>=2.37.4,<3` | |
414
+ | `imageio-ffmpeg` | `>=0.6.0,<0.7` | |
415
+ | `av` | `>=18.1.0,<19` | PyAV — required by CRF re-compression (§5.7) |
416
+ | `numpy` | `>=2.5.2,<3` | **numpy 2.** Do NOT pin `<2`: 1.26.4 is 2 years EOL and nothing in the stack caps it |
417
+ | `pillow` | `>=12.3.0,<13` | |
418
+ | `spaces` | **absent, deliberately** | platform-pinned; see above |
419
+
420
+ **gradio 6 breaking changes that hit us:** `gr.Video` subtitle-tuple return removed;
421
+ `show_download_button` / `show_share_button` → `buttons=[...]`; `gr.Audio` `show_controls` →
422
+ `show_recording_waveform`; `min_length` / `max_length` → event-listener validators.
423
+
424
+ **Python 3.12 is required, not preferred:** numpy 2.5.2 needs ≥3.12 and `av` 18 needs ≥3.11. The
425
+ default 3.10.13 image silently resolves both *downward* rather than failing.
426
 
427
  ### 5.2 One central model-file indirection
428
 
429
+ Every Hub repo-id, revision, subfolder, filename **and dtype/quantization choice** (§2.6) lives
430
+ in **one** module — `ltx25_studio/model_files.py` — as data, never inlined at a call site.
 
431
 
432
+ A missing file must raise an error naming **both the file and its URL**. Not hypothetical:
433
+ Lightricks deleted a 2.3 checkpoint from the Hub with no warning and broke a live Space. When it
434
+ happens again, the message should tell the operator which URL to go look at.
 
435
 
436
+ Pinning a **revision** per repo is part of this module's job — a repo whose content can change
437
+ under us is not pinned just because its name is stable.
438
 
439
+ ### 5.3 Never return an empty output — and the four layers that let it happen
440
 
441
  Any generation that completes without producing a file **must** raise a specific, actionable
442
  error that the UI renders visibly.
443
 
444
+ **The full anatomy of the 2.3 silent reset**, because each layer was individually "reasonable"
445
+ and that is the lesson:
446
+
447
+ 1. `models.ensure_models` hit a missing file and did `logger.warning(...) + continue`.
448
+ 2. ComfyUI's executor caught every node exception and returned a FAILURE **tuple**, without
449
+ raising.
450
+ 3. An `OutputEvent(video_path="")` was pushed **unconditionally**.
451
+ 4. The UI did `event.video_path if event.video_path else gr.update()` — rendering an empty path
452
+ as a silent form reset.
453
+
454
+ The user saw "nothing happened". No error, no log, no signal. It hid for a **week**.
455
+
456
+ **Our rules, both halves mandatory:**
457
+ - **An empty path becomes an error AT THE SOURCE**, and every layer propagates it. A success
458
+ path that can return a falsy artifact is a bug regardless of current reachability.
459
+ - **No "newest file in the output directory" fallback.** The 2.3 one had no run correlation and
460
+ would show the *previous* run's video for a failed run — a worse failure than showing nothing.
461
+ **Each run is stamped into its own output subdirectory**, and a run's artifact is only ever
462
+ looked up inside that run's directory.
463
 
464
+ ### 5.4 Read parameters from config, never hardcode
465
 
466
+ Mel/audio parameters come from `audio_vae.config` at runtime: `audio_sampling_rate`,
467
+ `audio_hop_length`, `audio_mel_bins`, `audio_vae_mel_compression_ratio`,
468
  `audio_vae_temporal_compression_ratio`, `audio_latent_channels`.
469
 
470
+ This mirrors the pipeline itself, whose constructor reads `self.vae.spatial_compression_ratio`,
471
+ `self.vae.temporal_compression_ratio`, `self.audio_vae.mel_compression_ratio`,
472
+ `self.audio_vae.temporal_compression_ratio` and `self.transformer.config.patch_size` from the
473
+ loaded components, using literals only as fallbacks. Same discipline. §5.7 extends this to CRF.
 
474
 
475
  ### 5.5 Enforce and surface the hard constraints
476
 
477
  | constraint | source | handling |
478
  |---|---|---|
479
+ | `height % 32 == 0`, `width % 32 == 0` | enforced upstream (raises) in all generation pipelines *and* the latent-upsample pipeline | validate before the call, snap in the UI, **show** the snapped value |
480
+ | `num_frames % 8 == 1` | **not** hard-raised upstream. Falls out of `latent_num_frames = (num_frames - 1) // vae.temporal_compression_ratio + 1`, which **silently floors** a bad value | we raise. **The `8` is `vae.temporal_compression_ratio`** — see below |
481
+ | audio/video length agreement | audio and video lengths are computed independently, and a supplied `audio_latents` **overrides** the duration-derived audio length (§5.7) | validate they agree; raise rather than emit a desynced file |
482
+
483
+ **The `num_frames` modulus is a parameter, not a literal.** Hardcoding `8` would violate §5.4.
484
+ Resolution:
485
+
486
+ - `constraints.py` takes the ratio as a **parameter with a documented fallback of `8`**, because
487
+ the UI must validate input *before* any VAE exists.
488
+ - `pipelines.py` **asserts the loaded `vae.temporal_compression_ratio` equals that fallback** at
489
+ load time, raising loudly if upstream ever changes it.
490
+
491
+ That gives a working pre-load validator *and* a tripwire. Both are test cases.
492
 
493
  **Verified constants** (`diffusers/pipelines/ltx2/utils.py`):
494
 
 
 
 
495
  ```
496
+ DISTILLED_SIGMA_VALUES = [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875] # 8 steps, CFG=1
497
+ STAGE_2_DISTILLED_SIGMA_VALUES = [0.909375, 0.725, 0.421875] # 3 steps
498
+ ```
499
+
500
+ Re-exported and asserted in `constraints.py` against the installed diffusers, so a schedule
501
+ change upstream fails a test instead of silently changing output quality.
502
+
503
+ ### 5.6 Preflight — fail loudly on resources, before the platform kills us
504
 
505
+ At startup, `preflight.py` measures **free RAM and free disk** and compares them against the
506
+ requirement computed from the `model_files.py` table. If short, it raises a specific error
507
+ naming the numbers:
508
+
509
+ > `need ~67.2 GiB RAM for the bf16 load, saw 41.3 GiB` (see §2.6 for the quantised fallbacks)
510
+
511
+ This is the fail-loudly principle applied to the resource budget. It turns an anonymous OOM or a
512
+ `"Workload evicted, storage limit exceeded (150G)"` into a diagnosable message that names the
513
+ fix. It also runs the §1.3 diffusers-version assertion. Test case.
514
+
515
+ ### 5.7 API gotchas that cause silent wrongness
516
+
517
+ Every row here is a **test case**, not a note. These are the ways this API is wrong-by-default.
518
+
519
+ | # | Gotcha | Consequence | Our handling |
520
+ |---|---|---|---|
521
+ | 1 | **`LTX2VideoCondition.index` is a LATENT index, not a pixel frame index.** `start_idx = max((latent_start_idx - 1) * vae_temporal_compression_ratio + 1, 0)` | **off-by-8 guaranteed** if a pixel frame number is passed | convert at the boundary in `conditioning.py`; UI speaks pixel frames, the pipeline gets latent indices |
522
+ | 2 | **Out-of-range condition indices are silently SKIPPED** with only a `logger.warning` | a whole conditioning image does nothing, silently | validate range ourselves and **raise**; this is a silent-failure path we convert into a loud one |
523
+ | 3 | **`noise_scale` default differs by class:** `0.0` on `LTX2Pipeline` / `LTX2ImageToVideoPipeline`, `None` on `LTX2ConditionPipeline` (resolving to `sigmas[0]` or 1.0) | copying a call between classes changes behaviour silently | always pass `noise_scale` explicitly; never rely on the default |
524
+ | 4 | **`num_frames` default differs:** `None` (auto-duration) on `LTX2Pipeline`/`ImageToVideo`/`Condition`; hard `121` on `LTX2InContextPipeline` and `LTX2HDRPipeline`, which have **no `duration_head` at all** | auto-duration silently unavailable on two pipelines | registry records per-workflow duration support; auto-duration also **raises** with >1 prompt and `num_frames=None` — validate before the call |
525
+ | 5 | `latents` must be **UNPACKED 5D** `[B,C,F,H,W]`; `audio_latents` **UNPACKED 4D** `[B,C,L,M]` | shape errors, or worse, silent misinterpretation | assert shapes at our boundary |
526
+ | 6 | **`output_type="latent"` returns ALREADY-DENORMALIZED latents** | re-feeding them re-normalizes → corrupted output | drives the mandatory `denormalize=False` in §7.4 |
527
+ | 7 | **Image-conditioning CRF re-compression is applied BY DEFAULT**, needs PyAV + uint8/PIL input; a float tensor **raises** unless `crf=0` | crashes on float input | normalise to uint8/PIL in `conditioning.py`; `av` is a pinned dep for this reason |
528
+ | 8 | **CRF is 18 when `text_encoder.config.model_type in ("gemma4_unified", "gemma4")`, else 33** | hardcoding either number is wrong for the other | **derive from `text_encoder.config.model_type` at runtime** (§5.4) |
529
+ | 9 | `from diffusers.pipelines.ltx2.export_utils import encode_video` is **DEPRECATED**, removal in 0.40.0 — and our pin *is* 0.40.0.dev0 | breaks on the very revision we pin | use `from diffusers.utils import encode_video` |
530
+ | 10 | `spatio_temporal_guidance_blocks` defaults to `[28]` for 2.3/2.5 (`[29]` was 2.0) and is a **mutable shared default** | in-place mutation corrupts every later call in the process | never mutate in place; copy before modifying |
531
+ | 11 | A supplied `audio_latents` **overrides** the duration-derived `audio_num_frames`: upstream logs `"Got audio_latents of shape […], audio_num_frames will be inferred"` then does `_, _, audio_num_frames, _ = audio_latents.shape` | supplying audio **silently redefines audio length** while `num_frames` still governs video → desync | §5.5 validation, with a concrete desync test case |
532
+ | 12 | `fuse_lora`/`unfuse_lora` default `components=["transformer"]` | connector LoRA layers silently unfused | §4.4 — always pass both |
533
+
534
+ Reference: `_optional_components = ["processor", "prompt_enhancer", "duration_head"]` (plus
535
+ `audio_scheduler` on Condition), and
536
+ `model_cpu_offload_seq = "prompt_enhancer->text_encoder->connectors->duration_head->transformer->vae->audio_vae->vocoder"`.
537
+ Recorded for completeness — **but see §8.4: we do not use accelerate's offload at all.**
538
+
539
+ ### 5.8 Operational lessons from the 2.3 post-mortem
540
+
541
+ | # | Lesson | Rule |
542
+ |---|---|---|
543
+ | 1 | **ZeroGPU quota attribution rides a contextvar** (`gradio.context.LocalContext.request`). It does **not** cross a plain `threading.Thread` — without it every user is bucketed as Space-anonymous and you get *"Space app has reached its GPU limit"* at **0 minutes used** | any thread we spawn must carry `contextvars.copy_context()`; better, do not spawn plain threads across the GPU boundary |
544
+ | 2 | The `duration=` callable must accept **the exact same positional signature** as the decorated function, trailing kwargs included | otherwise TypeError at schedule time; test the signature match |
545
+ | 3 | **Hidden 1.5× duration multiplier on `large`** (`duration_factor`): `duration=60` is scheduled as 90 s. On `xlarge` the factor is skipped but the request is **doubled** instead | never reason about wall-clock quota from the `duration=` number alone |
546
+ | 4 | Default duration is 60 s. The **max is a server-side, tier-dependent, unpublished cap** | **discover it, never hardcode it.** Empirically ~120 s is the safe ceiling; the official Lightricks LTX Space ships `@spaces.GPU(duration=75, size="xlarge")` |
547
+ | 5 | **Do NOT auto-retry a ZeroGPU timeout** using the captured Gradio request: the `X-IP-Token` ages past TTL during attempt 1, so attempt 2 always 401s with *"Expired ZeroGPU proxy token"* — **masquerading as the root cause** | surface the original timeout; never retry behind the user's back |
548
+ | 6 | `preload_from_hub` is **capped at 10 entries**, and preloaded files are owned by the *build* user and **read-only to runtime uid 1000** — a later lazy `hf_hub_download` fails with `Permission denied (os error 13)` unless the cache is mirrored | §6.2 |
549
+ | 7 | Accumulated generations filled the 2.3 Space's disk and it went unhealthy | §2.7 — sweep on startup and after each run |
550
 
551
  ---
552
 
 
555
  | | |
556
  |---|---|
557
  | **Space** | `techfreakworm/LTX2.5-Studio`, **private** |
558
+ | **SDK** | `sdk: gradio`, `sdk_version: 6.24.0` — **a custom Dockerfile forfeits ZeroGPU entirely** |
559
+ | **Python** | `python_version: "3.12"` — required, not optional (§5.1) |
560
+ | **Hardware** | ZeroGPU. `suggested_hardware: zero-a10g` is the **legacy id string the Spaces API still reports — it is NOT an A10G** (§8.1) |
561
+ | **Size** | `size="xlarge"` declared explicitly (§8.5) |
562
+ | **Weights** | README frontmatter `preload_from_hub:` |
563
+ | **Secret** | `HF_TOKEN` — required; `Lightricks/LTX-2.5-Diffusers` is gated |
564
  | **Git** | GitHub `git@github.com:techfreakworm/ltx2.5-studio.git`; Space and GitHub stay in sync |
565
 
566
+ Naming precedent: `techfreakworm/LTX2.3-Studio`, `wan-studio` (private), `SAM3-cutout-studio`
567
+ (private).
568
+
569
+ ### 6.2 `preload_from_hub` hazards
570
 
571
+ Two, and they interact:
 
572
 
573
+ 1. **Only live shards** (§2.1 Trap A). A wrong list fails after tens of GB, not fast.
574
+ 2. **Capped at 10 entries.** Our subset spans 11 component folders and ~20 individual shard
575
+ files, so a naive one-file-per-entry list **cannot fit**. Entries take the form
576
+ `repo_id file1,file2,…`, so one entry can name many files from one repo — the list must be
577
+ written that way. **Verification item before first deploy:** confirm there is no separate
578
+ per-entry file-count or line-length limit.
579
+ 3. **Ownership.** Preloaded files belong to the *build* user and are read-only to runtime uid
580
+ 1000. Any later lazy `hf_hub_download` into the same cache fails with
581
+ `Permission denied (os error 13)` unless the cache is mirrored. Design implication: **preload
582
+ everything we need, or mirror the cache at startup — do not mix the two strategies.**
583
+
584
+ ### 6.3 🚨 BLOCKER — repo gate not accepted
585
 
586
  `Lightricks/LTX-2.5-Diffusers` is gated (`gated: auto`) and the operator's token has **not**
587
  accepted that gate. Verified response:
 
593
 
594
  **https://huggingface.co/Lightricks/LTX-2.5-Diffusers**
595
 
596
+ Until accepted, nothing touching real weights can be built or tested. Everything that does not
597
+ need real weights proceeds. Escalated to the orchestrator.
598
 
599
  ---
600
 
601
  ## 7. Architecture
602
 
603
+ ### 7.1 Module layout
604
 
605
  ```
606
+ app.py # entrypoint. `import spaces` FIRST (§8.2), then build + launch.
607
  ltx25_studio/
608
+ config.py # device/dtype selection, paths, retention cap (§2.7), env overrides
609
+ model_files.py # §5.2 — THE single source of truth: repo ids, revisions, subfolders,
610
+ # filenames, sizes, AND the dtype/quantization switch (§2.6)
611
+ errors.py # typed exception hierarchy; every error carries an actionable detail
612
+ preflight.py # §5.6 — RAM/disk headroom vs the model_files requirement;
613
+ # §1.3 — the diffusers-version fail-fast assertion
614
+ zerogpu.py # §8 — the `spaces` shim, the static @spaces.GPU policy, the
615
+ # duration callable, contextvar propagation, and the
616
+ # "never return CUDA tensors" boundary
617
+ constraints.py # §5.5 — dims/frames/CRF/sigmas. Pure functions over numbers.
618
+ pipelines.py # construction on CPU, module-scope .to("cuda"), component wiring,
619
+ # and the vae.temporal_compression_ratio tripwire (§5.5)
620
+ conditioning.py # UI inputs -> list[LTX2VideoCondition]; pixel->latent index
621
+ # conversion (§5.7 #1); uint8/PIL normalisation (§5.7 #7)
622
+ audio.py # waveform <-> mel <-> audio_vae latents; params from config (§5.4);
623
+ # vocoder decode
624
+ modes.py # §3.3 registry: 11 UI modes as data over the 4 real workflows
625
+ generate.py # §7.4 stage graph. OWNS the never-empty invariant (§5.3) and the
626
+ # per-run output directory
627
+ outputs.py # mux video+audio, single-track export, 20 GB oldest-first eviction
628
  lora/
629
+ inspect.py # header-only safetensors parse; metadata + convention detection
630
+ validate.py # validator stages 1-6 (§4.3)
631
+ apply.py # load_lora_weights, stage-7 attach assertion, both-component fusing
 
632
  ui/
 
633
  blocks.py # Gradio Blocks assembly
634
  components.py # reusable widgets (condition list, LoRA picker, constraint hints)
635
+ handlers.py # thin adapters: UI event -> one call -> UI update. No logic.
636
  ```
637
 
638
  ### 7.2 Layering rules
639
 
640
  - **Dependencies point inward.** `ui/` may import anything; `generate.py` must not import `ui/`.
641
  `model_files.py`, `errors.py` and `constraints.py` import nothing from the package.
642
+ - **`modes.py` is data, not branching.** Adding a mode is a registry entry, not an `if` in
643
+ `generate.py`.
644
+ - **`ui/handlers.py` contains no business logic.** Marshal inputs → call one function → marshal
645
+ outputs. This is what makes the app testable without a GPU and without Gradio.
646
+ - **Errors cross layers as typed exceptions.** The UI is the only place that renders one as a
647
+ string. `generate.py` never returns an error-shaped success (§5.3).
 
 
 
 
 
 
 
 
 
648
 
649
+ ### 7.3 Why this shape
650
 
651
+ The predecessor put backend, models, modes and workflow in four flat modules with `ui.py`
652
+ importing all of them. That works until the failure modes get interesting: the empty-output bug
653
+ (§5.3) lived precisely in the seam between "workflow returned something" and "UI rendered it",
654
+ and there was no layer boundary at which to assert. Here `generate.py` owns that invariant and
655
+ can be tested for it directly.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
656
 
657
+ ### 7.4 The generation stage graph — **classic pipelines PLUS an explicit diffusion decode**
658
 
659
+ This is the most consequential architectural decision in the document, because the obvious
660
+ implementation silently discards 2.5's headline quality feature.
661
 
662
+ **`LTX25AutoBlocks` docstring, verbatim:**
663
+
664
+ > *"Auto blocks for LTX-2.5 supporting text-to-video, image-to-video, condition-to-video and
665
+ > in-context (IC-LoRA) generation (joint video + audio). Identical to `LTX2AutoBlocks` except
666
+ > that the video decoder is `LTX2DiffusionVaeDecoderStep`, **since the diffusion decoder is the
667
+ > native default from LTX-2.5 on.** To decode with the convolutional VAE instead, swap the
668
+ > decode block."*
669
+
670
+ **But the classic `LTX2ConditionPipeline` / `LTX2ImageToVideoPipeline` decode with
671
+ `self.vae.decode(...)` — the convolutional VAE, NOT the diffusion decoder.** So a naive classic
672
+ call silently gives up what the model card calls *"a new diffusion video decoder [that] replaces
673
+ the VAE reconstruction stage"*. It would not error. It would just be worse, forever, and nobody
674
+ would know why.
675
+
676
+ **Decision: classic pipelines, with the diffusion decode added as an explicit stage.**
677
 
678
  ```
679
+ stage 1 LTX2ConditionPipeline(..., sigmas=DISTILLED_SIGMA_VALUES,
680
+ output_type="latent")
681
+ → latents, audio_latents
682
+
683
+ optional LTX2LatentUpsamplePipeline(latents)
684
+ → upsampled latents
685
+
686
+ stage 2 same pipeline, latents=upsampled,
687
+ sigmas=STAGE_2_DISTILLED_SIGMA_VALUES,
688
+ noise_scale=STAGE_2_DISTILLED_SIGMA_VALUES[0],
689
+ output_type="latent"
690
+ → refined latents
691
+
692
+ decode V LTX2VideoDiffusionDecodePipeline(latents=..., denormalize=False) ← see below
693
+ decode A audio_vae.decode(audio_latents) → mel → vocoder(...) → waveform
694
  ```
695
 
696
+ **`denormalize=False` is MANDATORY.** `output_type="latent"` returns *already-denormalized*
697
+ latents (§5.7 #6). Passing the default `True` **double-denormalizes and silently corrupts the
698
+ output**. This is a test case.
699
 
700
+ **Why not the modular path.** The model card's own Option C example uses the classic pipelines,
701
+ so that is the documented, lower-risk API. The modular path (`LTX25ModularPipeline` /
702
+ `LTX25AutoBlocks`, loaded from `modular_model_index.json`) is newer, and its interaction with
703
+ **ZeroGPU's tensor packing** (§8.3) and with **`load_lora_weights`** (§4) is unverified. Given
704
+ LoRA is the headline feature and ZeroGPU is the deploy target, betting both on an unverified
705
+ integration is the wrong trade.
706
 
707
+ The modular path is recorded here as **the canonical alternative**: if we move to it, the decode
708
+ block is swappable exactly as the docstring describes, and `vae/` (1.452 GB) stays in the subset
709
+ as the conv-VAE fallback either way.
710
 
711
+ ---
 
 
 
 
 
 
 
 
 
712
 
713
+ ## 8. ZeroGPU execution model
714
 
715
+ **This section replaces the conventional wisdom about diffusion-model offloading. On ZeroGPU
716
+ that wisdom is not merely suboptimal — it is counterproductive.**
 
717
 
718
+ ### 8.1 The hardware is not what the API says
719
+
720
+ ZeroGPU is **NVIDIA RTX PRO 6000 Blackwell Server Edition**, partitioned via MIG (sm_120).
721
 
722
+ | tier | VRAM | quota |
723
+ |---|---:|---|
724
+ | `large` (default) | **48 GB** | 1× |
725
+ | `xlarge` | **96 GB** | 2× |
726
 
727
+ The Spaces API still reports the legacy id string **`zero-a10g`**. **Do not read that as an
728
+ A10G** — it is a stale identifier, and sizing decisions made from it would be wrong by a factor
729
+ of several.
 
 
730
 
731
+ **VRAM is not our constraint.** 67.2 GiB of weights against `xlarge`'s ~95 GiB leaves ~28 GiB
732
+ for activations. The constraints are CPU RAM and transient disk (§2).
733
+
734
+ ### 8.2 Import order is load-bearing
735
+
736
+ ```python
737
+ import spaces # MUST be first
738
+ import torch # only after
739
+ import diffusers # only after
740
+ ```
741
+
742
+ `import spaces` **must precede** torch / diffusers / transformers, or startup fails with
743
+ `"CUDA has been initialized before importing the spaces package"`. This is why `app.py` is thin
744
+ and why the `spaces` shim lives in `zerogpu.py` with a guarded import for local dev.
745
+
746
+ ### 8.3 The packing model: build on CPU, `.to("cuda")` at module scope
747
+
748
+ ZeroGPU **replaces** CPU offload with its own disk-packing engine. The correct pattern inverts
749
+ normal GPU-box practice:
750
+
751
+ 1. Build the pipeline **on CPU**, at module scope.
752
+ 2. Call `pipe.to("cuda")` **at module scope**. That call is **intercepted**, rewritten to
753
+ `.to("cpu")`, and every tensor is packed to disk (`~/.zerogpu/tensors`, O_DIRECT). After
754
+ packing, the main process holds no weights.
755
+ 3. Apply `@spaces.GPU(...)` **statically at module import**. Dynamic or inner decoration fails
756
+ ZeroGPU's startup analyzer with *"No @spaces.GPU function detected during startup"*.
757
+ 4. Inside the decorated function real CUDA is live, and a forked worker streams
758
+ **disk → pinned host memory → VRAM**.
759
+
760
+ HF's guidance, quoted: **models must be placed on cuda at the root module level.**
761
+
762
+ **Corollary: do NOT lazy-load models inside the decorated function.** HF documents that
763
+ placements are optimised for startup; lazy loading defeats the packer entirely.
764
+
765
+ ### 8.4 `enable_model_cpu_offload()` must NOT be used
766
 
767
+ This supersedes an earlier draft of this plan, which called it mandatory. It is the opposite.
768
 
769
+ `enable_model_cpu_offload()` and `enable_sequential_cpu_offload()` are **counterproductive on
770
+ ZeroGPU**, for a specific mechanical reason: any submodule accelerate parks on CPU is **never
771
+ registered for packing**. Consequently it —
772
 
773
+ - stays resident in **main-process RAM** (which is our tightest budget, §2),
774
+ - is **fork-inherited by every worker**, multiplying that cost, and
775
+ - is **re-copied host→device on every hook fire**, on top of the packer's own streaming.
776
 
777
+ So we pay RAM we do not have, to duplicate work the platform already does better. The
778
+ `model_cpu_offload_seq` string recorded in §5.7 is upstream metadata we do not act on.
779
 
780
+ ### 8.5 Never return CUDA tensors across the `@spaces.GPU` boundary
 
 
 
 
 
 
 
 
 
 
 
 
 
781
 
782
+ Returning a CUDA tensor from a `@spaces.GPU` function makes unpickling in the main process
783
+ trigger a **blocked lazy CUDA init**, and **the call hangs** — no error, no timeout message, a
784
+ dead request.
 
 
785
 
786
+ **Rule: return file paths, or `.cpu()` tensors. Always.** `zerogpu.py` owns this boundary and it
787
+ is a test case.
788
+
789
+ ### 8.6 Tier and quota
790
+
791
+ We declare `size="xlarge"` **explicitly** rather than being silently promoted.
792
+
793
+ Packed bf16 weights (67.2 GiB) exceed `xlarge_threshold = 42.00 GiB`, so **ZeroGPU auto-promotes
794
+ us to `xlarge` and doubles quota cost with no code change** (§2.6). Declaring it explicitly means
795
+ the cost is visible in the source rather than being a surprise in the quota dashboard.
796
+
797
+ **Quota arithmetic:** PRO gives 2400 s/day. At `duration=75` on `xlarge` (2×), each run is
798
+ charged ~150 s → **~16 runs/day**. Dropping under 42.00 GiB via quantization would return us to
799
+ `large` at 1× — that is the entire quota argument for §2.6's fallback, and it is worth
800
+ approximately double the runs.
801
+
802
+ Two further traps from §5.8: the hidden **1.5× `duration_factor` on `large`** (so `duration=60`
803
+ schedules as 90 s; `xlarge` skips the factor but doubles the request instead), and the fact that
804
+ the **maximum duration is a server-side, tier-dependent, unpublished cap** — discover it, never
805
+ hardcode it. ~120 s is the empirical safe ceiling; the official Lightricks LTX Space ships
806
+ `@spaces.GPU(duration=75, size="xlarge")`.
807
+
808
+ ### 8.7 Local (Apple Silicon / MPS) is a different regime
809
+
810
+ MPS has no ZeroGPU packer, no `spaces` decorator semantics, and unified memory changes the
811
+ arithmetic entirely. Device strategy is selected in `config.py`, the `spaces` import is guarded,
812
+ and the local path is expected to be **capability-limited rather than identical**. Do not assume
813
+ a code path that works on ZeroGPU works on MPS, or vice versa.
814
+
815
+ ---
816
+
817
+ ## 9. Testing strategy
818
+
819
+ We run an **H3 TDD harness**. The gate is:
820
 
821
  ```
822
+ .venv/bin/python -m pytest -q
823
  ```
824
 
825
+ **Ownership:** a separate teammate owns `tests/**`. The implementer does not create or edit
826
+ anything under `tests/`. This plan's job is to make the units *testable*.
 
 
827
 
828
+ ### 9.1 Testable without a GPU, without weights, without network
 
829
 
830
+ | unit | how |
831
+ |---|---|
832
+ | `constraints.py` | pure functions over ints — dimension snapping, frame-count validation (with an injected compression ratio), CRF selection from a stubbed `model_type`, sigma schedule shape |
833
+ | `model_files.py` | data assertions: every entry has repo/revision/subfolder/filename; no duplicates; the §2.3 subset sums to 72.185; the dtype switch enumerates the three §2.6 configs |
834
+ | `preflight.py` | inject fake free-RAM/free-disk numbers → assert a specific error naming both numbers; assert the diffusers-version check fires on a wrong version |
835
+ | `lora/inspect.py` | **synthesised safetensors headers** — small fixtures, no real LoRA (this is why §4.3 stage 2 is header-only) |
836
+ | `lora/validate.py` | stages 1–6 against crafted good/bad headers, including the wrong-architecture case |
837
+ | `modes.py` | registry completeness (11 modes over 4 workflows); each mode's kwarg builder produces expected kwargs |
838
+ | `errors.py` | every error type's message contains the actionable detail (filename, URL, found-prefixes, measured numbers) |
839
+ | `conditioning.py` | pixel→latent index conversion (§5.7 #1); out-of-range raises (§5.7 #2) |
840
+ | `outputs.py` | 20 GB oldest-first eviction over a temp dir |
841
+ | `generate.py` | stub pipeline returning nothing → assert a specific exception, **not** an empty path |
842
+ | `zerogpu.py` | the return-boundary rejects/converts CUDA-tensor returns; `duration=` callable signature matches the decorated function |
843
 
844
+ ### 9.2 GPU-gated tests
 
 
 
845
 
846
+ Anything needing real weights is marked `gpu` and deselected by default. CI and the default local
847
+ run must be green with zero weights on disk.
848
 
849
+ ### 9.3 The invariant tests that matter most
 
850
 
851
+ Ranked by how much silent wrongness they prevent:
852
+
853
+ 1. **A LoRA for the wrong architecture is rejected** — specifically at stage 7 if it survives
854
+ stages 1–6 (§4.3).
855
+ 2. **A generation that produces no file raises** — never returns an empty path, and there is no
856
+ newest-file fallback (§5.3).
857
+ 3. **`denormalize=False` on the diffusion decode** — the double-denormalization corruption
858
+ (§7.4) is silent and total.
859
+ 4. **Pixel→latent index conversion** — off-by-8 is guaranteed without it (§5.7 #1).
860
+ 5. **Audio/video length desync** with a supplied `audio_latents` (§5.7 #11), concrete case.
861
+ 6. **The diffusers-version assertion fires** on 0.39.0 (§1.3) — otherwise the failure is late
862
+ and expensive.
863
+ 7. **`vae.temporal_compression_ratio` tripwire** raises if upstream changes it (§5.5).
864
+ 8. **`preload_from_hub` names only live shards** — assertable against the index JSON with no
865
+ download (§2.1).
866
+ 9. **`fuse_lora` passes both components** (§4.4).
867
 
868
+ ---
869
 
870
+ ## 10. Resolved questions and technical risks
 
 
 
 
 
871
 
872
+ All six open questions from the first draft are **resolved**. What remains is one ranked risk
873
+ register.
874
 
875
+ | # | Question | Resolution |
876
+ |---|---|---|
877
+ | **OQ-1** | Pin a diffusers release, or `main`? | **RESOLVED: pin the exact SHA** `ac56fa2f…` (§1.3). A SHA is more pinned than a range, not less. Plus a startup version assertion, because 0.39.0 fails *late*. |
878
+ | **OQ-2** | Does `diffusion_decoder` have a consumer? | **Moot** — the SHA provides it. Both `diffusion_decoder` (0.834) and `duration_head` (0.004) stay in the subset; duration prediction ships. |
879
+ | **OQ-3** | Is `num_frames % 8 == 1` enforced? | **RESOLVED: no.** Upstream silently floors. We raise; the modulus is a **parameter with an `8` fallback** for pre-load validation, plus a load-time tripwire (§5.5). |
880
+ | **OQ-4** | Gemma-3 or Gemma-4? | **RESOLVED: both, via a union annotation** — `Gemma3ForConditionalGeneration \| Gemma4UnifiedForConditionalGeneration`. LTX-2.5 genuinely is Gemma-4; the proof is that CRF 18 is selected by `model_type in ("gemma4_unified","gemma4")`. **Derive family and CRF at runtime; hardcode neither** (§5.7 #8). |
881
+ | **OQ-5** | Audio length from duration or latent shape? | **RESOLVED: latent shape wins.** Upstream computes it from duration, then overrides from `audio_latents.shape` when supplied. The desync is real and directional (§5.7 #11). |
882
+ | **OQ-6** | Output retention bound? | **RESOLVED: 20 GB**, oldest-first, swept on startup and after each run (§2.7). |
883
+
884
+ ### Technical risk register
885
+
886
+ | # | Risk | Why it is uncertain | How it gets settled |
887
+ |---|---|---|---|
888
+ | **1** | **bf16 fits ZeroGPU's CPU RAM and transient disk** | The ~68 GB RAM ceiling is a **community cgroup observation**, not official — HF publishes ZeroGPU RAM as `"-"` / "Dynamic resources". Transient disk peaks at ~144 GB against a 150 GB cap before the packer auto-prunes. | **Measurement on the Space, not argument.** §5.6's preflight turns a failure into a diagnosable message; §2.6's one-line dtype switch makes the fallback cheap. |
889
+ | 2 | ZeroGPU's packer handles the pipeline's tensor graph correctly | Verified for ordinary tensors; **unverified for quantised custom tensor subclasses** (bitsandbytes `Int8Params`/`Params4bit`, torchao) against the packer's `WeakTensorKeyDictionary` | only becomes live if risk #1 forces quantization — measure then |
890
+ | 3 | `preload_from_hub` can express our shard list within 10 entries | entry format is `repo_id file1,file2,…`, so one entry names many files; **a per-entry limit is unconfirmed** | verify before first deploy (§6.2) |
891
+ | 4 | Max `@spaces.GPU` duration on our tier | server-side, tier-dependent, **unpublished** | discover empirically; never hardcode (§5.8 #4) |
892
+ | 5 | Multishot via prompt structure produces usable results | no API support, and the enhancer system prompts actively discourage scene cuts (§3.4) | presented to users as unverified technique, not a feature |
893
 
894
+ ### 🚨 Live blocker
895
 
896
+ **`Lightricks/LTX-2.5-Diffusers` gate not accepted** — §6.3. One click at
897
+ https://huggingface.co/Lightricks/LTX-2.5-Diffusers. Blocks everything that touches real weights.
 
898
 
899
  ---
900
 
901
  ## 11. Build order
902
 
903
  1. ✅ **This document**, committed.
904
+ 2. ✅ **Scaffold** — venv, `pyproject.toml`, package stubs, `.gitignore`.
905
+ 3. ✅ **`requirements.txt`** — the §5.1 pin set, with **no `spaces` line**.
906
+ 4. **TDD, innermost-out**, against the test teammate's failing specs:
907
+ `errors` → `constraints` → `model_files` → `preflight` → `lora/` → `zerogpu` → `conditioning`
908
+ → `modes` → `audio` → `generate` → `outputs` → `ui`.
909
+ 5. **Blocked on §6.3** — anything that touches real weights, and every `gpu`-marked test.
ltx25_studio/preflight.py ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Startup assertions that fail loudly before anything expensive happens.
2
+
3
+ PLAN §5.6 and §1.3. Two jobs:
4
+
5
+ 1. **Version fail-fast.** ``diffusers.__version__`` must be ``"0.40.0.dev0"``
6
+ (the pinned SHA). Release 0.39.0 imports ``LTX2Pipeline`` fine and only fails
7
+ LATE, on a missing ``temporal_latent_upsampler``, after ~70 GB of download.
8
+
9
+ 2. **Resource preflight.** Measure free RAM and free disk, compare against the
10
+ requirement computed from the ``model_files`` table, and raise an error that
11
+ NAMES THE NUMBERS if short -- e.g. "need ~67.2 GiB RAM for the bf16 load, saw
12
+ 41.3 GiB". This turns an anonymous OOM, or a "Workload evicted, storage limit
13
+ exceeded (150G)", into a diagnosable message that names the fix (PLAN §2.6's
14
+ quantised fallbacks).
15
+
16
+ Both are test cases: inject fake numbers, assert the specific error.
17
+ """
ltx25_studio/zerogpu.py ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The ZeroGPU boundary: the ``spaces`` shim, GPU decoration, and return safety.
2
+
3
+ PLAN §8. This module exists because ZeroGPU inverts normal GPU-box practice and
4
+ several of its rules are silent-failure rules.
5
+
6
+ * ``import spaces`` must precede torch/diffusers/transformers, or startup dies
7
+ with "CUDA has been initialized before importing the spaces package". The
8
+ import here is guarded (``try/except ImportError``) so local dev works --
9
+ ``requirements.txt`` deliberately does NOT pin ``spaces`` (PLAN §5.1).
10
+
11
+ * ``@spaces.GPU(...)`` must be applied STATICALLY at module import. Dynamic or
12
+ inner decoration fails the startup analyzer with "No @spaces.GPU function
13
+ detected during startup".
14
+
15
+ * **Never return CUDA tensors** across the boundary. Unpickling one in the main
16
+ process triggers a blocked lazy CUDA init and THE CALL HANGS -- no error, no
17
+ timeout, a dead request. Return file paths, or ``.cpu()``. Always. This
18
+ module owns that boundary and it is a test case.
19
+
20
+ * ``enable_model_cpu_offload()`` is NOT used -- see PLAN §8.4. It is
21
+ counterproductive here, not merely suboptimal.
22
+
23
+ * The ``duration=`` callable must accept the EXACT same positional signature as
24
+ the decorated function, trailing kwargs included, or it TypeErrors at
25
+ schedule time.
26
+
27
+ * Quota attribution rides a contextvar (``gradio.context.LocalContext.request``)
28
+ that does NOT cross a plain ``threading.Thread``. Without
29
+ ``contextvars.copy_context()`` every user buckets as Space-anonymous and you
30
+ get "Space app has reached its GPU limit" at zero minutes used.
31
+
32
+ * Do NOT auto-retry a ZeroGPU timeout with a captured Gradio request: the
33
+ ``X-IP-Token`` ages past TTL during attempt 1, so attempt 2 always 401s with
34
+ "Expired ZeroGPU proxy token" -- masquerading as the root cause.
35
+ """
pyproject.toml CHANGED
@@ -7,10 +7,17 @@ requires-python = ">=3.12"
7
  license = { text = "MIT" }
8
  authors = [{ name = "Mayank Gupta", email = "techfreakworm@gmail.com" }]
9
 
10
- # Runtime dependencies are declared in requirements.txt, which is the file the
11
- # Hugging Face Space build reads. It is intentionally not written yet — see
12
- # docs/PLAN.md §5.1 (pin everything, with upper bounds) and §10 OQ-1, which must
13
- # be resolved before the diffusers pin can be chosen.
 
 
 
 
 
 
 
14
 
15
  [project.urls]
16
  Repository = "https://github.com/techfreakworm/ltx2.5-studio"
 
7
  license = { text = "MIT" }
8
  authors = [{ name = "Mayank Gupta", email = "techfreakworm@gmail.com" }]
9
 
10
+ # Runtime dependencies are declared in requirements.txt — that is the file the
11
+ # Hugging Face Space build reads, and it is the single source of truth for pins.
12
+ # See docs/PLAN.md §5.1, including why `spaces` is deliberately NOT pinned there.
13
+
14
+ [project.optional-dependencies]
15
+ # Dev tooling lives here rather than in requirements.txt so the Space image stays
16
+ # lean. Install with: .venv/bin/python -m pip install -e ".[dev]"
17
+ dev = [
18
+ "pytest>=9.0,<10",
19
+ "ruff>=0.16,<0.17",
20
+ ]
21
 
22
  [project.urls]
23
  Repository = "https://github.com/techfreakworm/ltx2.5-studio"
requirements.txt ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # LTX-2.5 Studio — runtime dependencies.
2
+ #
3
+ # Every direct dependency has a floor AND a ceiling. See docs/PLAN.md §5.1 for
4
+ # the reasoning; the short version is that the predecessor project shipped ~50
5
+ # unbounded requirements and got burned by a silent torch/torchaudio ABI break.
6
+ #
7
+ # Dev/test tooling (pytest, ruff) is NOT here — it lives in pyproject.toml's
8
+ # [project.optional-dependencies] dev extra, so the Space image stays lean.
9
+
10
+ # --- NOT PINNED, DELIBERATELY -------------------------------------------------
11
+ # `spaces` is intentionally absent. The Spaces platform pins its own version
12
+ # (live Spaces report 0.49.3 / 0.50.0 / 0.50.4 while PyPI is at 0.51.1), and a
13
+ # conflicting pin here breaks pip resolution at build time. The 2.3 repo's own
14
+ # AGENTS.md:19 states this rule and its requirements.txt:2 then violated it.
15
+ # The import is guarded in ltx25_studio/zerogpu.py so local dev works without it.
16
+ # ------------------------------------------------------------------------------
17
+
18
+ # --- torch trio: pinned exactly, and pinned TOGETHER --------------------------
19
+ # ZeroGPU supports exactly torch 2.8.0 / 2.9.1 / 2.10.0 / 2.11.0 (CUDA 12.8;
20
+ # sm_120 Blackwell needs cu128+). 2.11.0 is the ceiling.
21
+ # Trap: torchvision 0.28.0 hard-declares torch==2.13.0, so letting torchvision
22
+ # float silently drags torch past the ceiling. torchaudio must match torch
23
+ # exactly or the C++ ABI mismatch crashes the whole audio path on import.
24
+ torch==2.11.0
25
+ torchvision==0.26.0
26
+ torchaudio==2.11.0
27
+
28
+ # --- diffusers: exact SHA, PEP 508 direct reference (not #egg=) ---------------
29
+ # LTX-2.5 is not in any release. 0.39.0 imports LTX2Pipeline fine and then fails
30
+ # LATE on a missing temporal_latent_upsampler — after ~70 GB of download. This
31
+ # SHA is HEAD at 2026-08-19T12:11:04Z and contains PR #14447 (LTX-2.5 merge,
32
+ # 2026-08-11) plus #14453 and #14462. ltx25_studio/preflight.py asserts
33
+ # diffusers.__version__ == "0.40.0.dev0" at startup so a wrong revision fails
34
+ # immediately instead of expensively. See PLAN §1.3.
35
+ diffusers @ git+https://github.com/huggingface/diffusers@ac56fa2f25c1c15b4cbaa1e0b3255e9bc38c06dc
36
+
37
+ # --- model stack --------------------------------------------------------------
38
+ transformers>=5.15.1,<6
39
+ accelerate>=1.14.0,<2
40
+ peft>=0.20.0,<0.21 # LoRA backend; USE_PEFT_BACKEND gates load_lora_weights
41
+ safetensors>=0.8.0,<0.9
42
+ huggingface_hub>=1.28.0,<2
43
+ hf_xet>=1.6.0,<2
44
+
45
+ # --- UI -----------------------------------------------------------------------
46
+ # gradio 5.x is END OF LIFE (last stable 5.50.0, 2025-11-21). The Space's
47
+ # frontmatter `sdk_version` is authoritative for the installed gradio; this range
48
+ # must stay compatible with it (currently 6.24.0). Breaking changes from 5.x that
49
+ # affect us: gr.Video subtitle-tuple return removed; show_download_button /
50
+ # show_share_button -> buttons=[...]; gr.Audio show_controls ->
51
+ # show_recording_waveform; min_length / max_length -> event-listener validators.
52
+ gradio>=6.24.0,<7
53
+
54
+ # --- media I/O ----------------------------------------------------------------
55
+ # av (PyAV) is not optional: image-conditioning CRF re-compression is applied by
56
+ # default and requires it. See PLAN §5.7 #7.
57
+ imageio>=2.37.4,<3
58
+ imageio-ffmpeg>=0.6.0,<0.7
59
+ av>=18.1.0,<19
60
+
61
+ # --- numerics -----------------------------------------------------------------
62
+ # numpy 2, deliberately. Do NOT pin <2: 1.26.4 is two years EOL and nothing in
63
+ # this stack caps numpy. numpy 2.5.2 requires Python >=3.12, which is why the
64
+ # Space frontmatter must set python_version: "3.12" — the default 3.10.13 image
65
+ # silently resolves numpy and av DOWNWARD rather than failing.
66
+ numpy>=2.5.2,<3
67
+ pillow>=12.3.0,<13