Revise spec: LTX-2.3 22B distilled via diffusers, ZeroGPU H200, optional 2x upscale toggle

docs/superpowers/specs/2026-06-25-ltx-image-to-video-space-design.md

@@ -1,4 +1,4 @@
-# LTX Image-to-Video Gradio Space — Design
+# LTX-2.3 Image-to-Video Gradio Space — Design
 
 **Date:** 2026-06-25
 **Status:** Approved
@@ -6,90 +6,133 @@
 
 ## Goal
 
-A minimal Hugging Face Space where a user uploads an image and types a
-prompt, clicks **Generate**, and receives a short MP4 video. Powered by the
-LTXV 0.9.x **distilled (2B)** model running on **ZeroGPU**.
+A Hugging Face Space where a user uploads an image and types a prompt, clicks
+**Generate**, and receives a short MP4 video (with audio). It matches the model
+stack used by the `WhatDreamsCost/WhatDreamsCost-ComfyUI` "LTX Director 2"
+workflow, reimplemented as a native Gradio app using **`diffusers`** — not
+ComfyUI.
 
-The referenced project, `WhatDreamsCost/WhatDreamsCost-ComfyUI`, is a ComfyUI
-extension built around LTX Video. It is the *inspiration* for the model and
-capability only. This Space does **not** run ComfyUI — it is a clean, native
-Gradio app using the LTX-Video model directly via `diffusers`.
+## Reference workflow
+
+Derived from `docs/LTX_Director_2_Workflow_Hotfix.json`. The relevant model
+stack and sampler settings from that graph:
+
+| Component | Workflow value |
+|-----------|----------------|
+| Transformer | `ltx-2.3-22b-distilled` (22B distilled, fp8 in the Comfy graph) |
+| Text encoder | Gemma-3-12B (`gemma_3_12B_it`) + LTX text projection |
+| Video VAE | `LTX23_video_vae_bf16` |
+| Audio VAE | `LTX23_audio_vae_bf16` (LTX-2.3 jointly generates audio + video) |
+| Spatial upscaler | `ltx-2.3-spatial-upscaler-x2-1.1` (optional 2nd stage) |
+| Sampler / schedule | euler, linear_quadratic, **8 steps**, **CFG = 1** |
+| Output | **24 fps, 5 s, 120 frames**, two-stage (base → 2× upscale) |
 
 ## Decisions (from brainstorming)
 
 | Question | Decision |
 |----------|----------|
-| Relationship to ComfyUI repo | Native Gradio app using the LTX model directly (not ComfyUI) |
-| Model variant | LTXV 0.9.x **distilled (2B)** |
-| Hardware | **ZeroGPU** (`spaces` package, `@spaces.GPU` decorator, time-budgeted) |
-| UI scope | **Minimal** — image + prompt + Generate button only |
+| Relationship to ComfyUI repo | Native Gradio app via `diffusers` (not ComfyUI) |
+| Model | **LTX-2.3 22B distilled** (matches the workflow) |
+| Inference library | `diffusers` LTX-2.3 distilled image-to-video pipeline |
+| Hardware | **ZeroGPU** (H200, 70 GB) via `@spaces.GPU` |
+| UI scope | Minimal: image + prompt + Generate, plus one "2× upscale" toggle |
+| Audio | Included (LTX-2.3 generates A/V jointly) |
+| 2× upscale stage | Optional UI toggle, **default OFF** (time-budget safety) |
+
+## Feasibility (verified)
+
+- **Inference path:** `diffusers` natively supports LTX-2.3. Use
+  `diffusers/LTX-2.3-Distilled-Diffusers` (8 steps, CFG = 1) with the
+  image-to-video / IC-LoRA pipeline and the optional two-stage
+  generate→upscale recipe. No ComfyUI fp8 safetensors or custom loaders
+  required.
+- **VRAM:** ZeroGPU runs on H200 with 70 GB. The 22B distilled is ~46 GB in
+  bf16, ~23 GB in fp8; fits in 70 GB alongside Gemma-3-12B with
+  `enable_model_cpu_offload()`.
+- **GPU time budget (the real constraint):** ZeroGPU defaults to 60 s per
+  call, raised with `@spaces.GPU(duration=...)`, with a hard ceiling and a
+  daily quota (longer durations favor PRO). The base 120-frame generation is
+  the safe default; the 2× upscale stage is the slow part and is therefore an
+  opt-in toggle, default OFF.
 
 ## Architecture
 
-Single `app.py` with one Gradio `Blocks` UI. The pipeline is constructed once
-at module import; the actual generation runs inside a ZeroGPU-decorated worker
-function that moves work to the GPU.
+Single `app.py` with one Gradio `Blocks` UI. The pipeline(s) are constructed
+once at module import; generation runs inside a ZeroGPU-decorated worker.
 
 ```
 app.py
-  ├─ load pipeline (LTXImageToVideoPipeline from diffusers, bf16)
-  ├─ @spaces.GPU(duration=120) generate(image, prompt) -> mp4 path
-  └─ gr.Blocks UI: Image input + Prompt textbox + Generate button -> Video output
-requirements.txt   (diffusers, transformers, torch, spaces, imageio[ffmpeg])
-README.md          (already present; keep HF frontmatter unchanged)
+  ├─ load diffusers LTX-2.3 distilled pipeline(s) at import
+  │     • base image-to-video pipeline (8 steps, CFG=1)
+  │     • spatial 2x upscale pipeline (loaded lazily / only used when toggled)
+  │     • enable_model_cpu_offload() to fit 70 GB
+  ├─ @spaces.GPU(duration=...) generate(image, prompt, upscale) -> mp4 path
+  └─ gr.Blocks UI: Image + Prompt + Upscale checkbox + Generate -> Video
+requirements.txt   (diffusers, transformers, torch, spaces, imageio[ffmpeg], ...)
+README.md          (existing; HF Space frontmatter preserved)
 ```
 
-## Component Details
+## Component details
 
-### Model
-- `Lightricks/LTX-Video` distilled **2B** variant via the `diffusers`
-  `LTXImageToVideoPipeline`.
-- Precision: `torch.bfloat16`.
-- Loaded at module import. GPU placement handled inside the ZeroGPU worker
-  (ZeroGPU attaches the GPU only during the decorated call).
+### Models (`diffusers`)
+- LTX-2.3 22B distilled image-to-video pipeline, fp8 where available.
+- Gemma-3-12B text encoder (loaded by the pipeline).
+- Optional `ltx-2.3-spatial-upscaler-x2` pipeline for the 2× stage.
+- `enable_model_cpu_offload()` to stay within 70 GB.
+- Loaded once at module import. GPU placement happens inside the ZeroGPU
+  worker (ZeroGPU attaches the GPU only during the decorated call).
 
 ### Fixed defaults (hidden from the user)
-- Resolution: **768×512**, auto-fit to the uploaded image's aspect ratio,
-  each dimension rounded to a multiple of 32 (LTX constraint).
-- Frame count: **~96 frames @ 24 fps (~4 seconds)**.
-- Inference steps: distilled count (~8).
-- Guidance: fixed value appropriate for the distilled model.
-- Seed: **random each run**.
-
-### `generate(image, prompt)`
-- Decorated with `@spaces.GPU(duration=120)`.
-- Steps: validate inputs → resize/preprocess image to target dims → run the
-  pipeline → export frames to MP4 (`imageio` / ffmpeg) → return the file path.
-- Returns a path to a temporary MP4 for the `gr.Video` output.
+- Resolution: base resolution per the LTX-2.3 distilled recipe, auto-fit to the
+  uploaded image's aspect ratio, each dimension rounded to a multiple of 32.
+- Frames / fps: **120 frames @ 24 fps (~5 s)**.
+- Steps: **8** (distilled). Guidance: **CFG = 1**. Schedule: linear-quadratic.
+- Seed: random each run.
+
+### `generate(image, prompt, upscale)`
+- Decorated with `@spaces.GPU(duration=...)` (set near the ZeroGPU ceiling).
+- Steps: validate inputs → preprocess/resize image to target dims → run base
+  image-to-video pipeline → if `upscale` is True, run the 2× spatial-upscale
+  stage → mux generated audio → export to MP4 (`imageio` / ffmpeg) → return the
+  file path for `gr.Video`.
 
 ### UI (`gr.Blocks`)
 - `gr.Image(type="pil")` — input image.
 - `gr.Textbox` — prompt.
-- `gr.Button("Generate")` — triggers `generate`.
-- `gr.Video` — output.
-- Optional: 1–2 `gr.Examples` rows (image + prompt pairs) if example assets
-  are available.
+- `gr.Checkbox("2× high-res upscale", value=False)` — opt-in upscale stage.
+- `gr.Button("Generate")`.
+- `gr.Video` — output (MP4 with audio).
+- Optional: 1–2 `gr.Examples` rows (image + prompt) if assets are available.
 
-## Error Handling
+## Error handling
 - Empty image or empty prompt → `gr.Warning`, no generation run.
-- Pipeline / OOM / runtime exception → caught and surfaced as a `gr.Error`
-  with a short human-readable message, so the worker does not crash.
+- Pipeline / OOM / time-limit / runtime exception → caught and surfaced as a
+  `gr.Error` with a short human-readable message so the worker does not crash.
+  If the upscale stage is the cause, the message suggests retrying with the
+  toggle off.
 
-## Out of Scope (YAGNI)
-- Negative prompt, sliders, advanced accordion / extra controls.
-- ComfyUI runtime, custom nodes, node graph.
-- Video editing, timeline, keyframes, audio, IC-LoRA — these belong to the
-  ComfyUI project's scope, not this minimal demo.
+## Out of scope (YAGNI)
+- ComfyUI runtime, custom nodes, node graph, the timeline / Director UI.
+- Video editing, keyframes, retake mode, IC-LoRA style transfer, prompt relay.
+- Negative prompt, extra sliders, manual seed entry, resolution controls.
 
 ## Testing
 - Diffusion output is not meaningfully unit-testable.
 - Validation:
-  - A `generate()` smoke path with a mocked pipeline (returns dummy frames) to
-    confirm an MP4 is written and a valid path returned, and that empty-input
-    guards fire.
-  - Manual run on the deployed Space to confirm end-to-end behavior on ZeroGPU.
+  - A `generate()` smoke path with a mocked pipeline (dummy frames) confirming
+    an MP4 is written and a valid path returned, that empty-input guards fire,
+    and that the `upscale` flag selects the right code path.
+  - Manual run on the deployed Space to confirm end-to-end behavior, VRAM fit,
+    and that the default (no-upscale) path stays within the ZeroGPU time limit.
 
-## Files Touched
+## Files touched
 - `app.py` (new) — the entire application.
 - `requirements.txt` (new) — Python dependencies.
 - `README.md` (existing) — unchanged; HF Space frontmatter preserved.
+
+## References
+- diffusers LTX-2.3: https://huggingface.co/diffusers/LTX-2.3-Diffusers
+- diffusers LTX-2.3 distilled: https://huggingface.co/diffusers/LTX-2.3-Distilled-Diffusers
+- Lightricks LTX-2.3 weights: https://huggingface.co/Lightricks/LTX-2.3
+- Lightricks LTX-2 inference: https://github.com/Lightricks/LTX-2
+- ZeroGPU docs: https://huggingface.co/docs/hub/en/spaces-zerogpu