square-zero-labs commited on
Commit
20f1418
·
1 Parent(s): d56b6d6

Promote sliding window workflow fix

Browse files
Files changed (4) hide show
  1. README.md +131 -56
  2. foley-sliding-window.json +1655 -0
  3. ltx_foley_v2a/nodes.py +669 -5
  4. setup_runpod_ltx_foley.sh +140 -38
README.md CHANGED
@@ -8,51 +8,57 @@ tags:
8
  - video-to-audio
9
  - audio-generation
10
  - workflow
 
11
  ---
12
 
13
  # LTX 2.3 Foley V2A ComfyUI Workflow
14
 
15
- This repository contains a ready-to-test ComfyUI workflow for the
16
  [`FuzzPuppy/LTX-2.3-Foley-LoRA`](https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-LoRA)
17
  LoRA. The LoRA adds Foley sound effects to a silent input video using LTX-2.3:
18
- given a short video and a prompt describing the visible action, the workflow
19
  generates matching non-speech, non-music sound effects and saves a new MP4.
20
 
21
- The workflow is intentionally small: it takes one short video clip, keeps the
22
- first `89` frames, generates matching Foley audio, and saves an MP4 with the
23
- source frames plus generated audio. It is meant as the fastest public ComfyUI
24
- test for `FuzzPuppy/LTX-2.3-Foley-LoRA`, not as a full long-video stitching
25
- pipeline.
26
 
27
- ## Tutorial
28
 
 
29
  [![Watch the tutorial: using the LTX-2.3 Foley LoRA in ComfyUI](https://img.youtube.com/vi/qnHFDlrySmw/hqdefault.jpg)](https://youtu.be/qnHFDlrySmw)
30
 
31
  [Watch the tutorial on YouTube](https://youtu.be/qnHFDlrySmw)
32
 
33
  ## What Is Included
34
 
35
- - `ltx_23_foley_v2a.json`: ComfyUI workflow.
 
36
  - `setup_runpod_ltx_foley.sh`: one-command RunPod setup script.
37
  - `ltx_foley_v2a`: small helper-node package.
38
  - `tennis-no-sound.mp4`: default silent test video for RunPod setup.
39
 
40
- The workflow requires the `ltx_foley_v2a` helper-node package. If ComfyUI shows
41
- missing nodes named `LTXFoleyVideoToAudioLatent`, `LTXFoleyTrimImages`, or
42
- `LTXFoleyAudioVAEDecode`, the workflow JSON was loaded before these helper nodes
43
- were installed into `ComfyUI/custom_nodes`.
44
 
45
  The helper-node package handles the workflow-specific pieces that stock ComfyUI
46
  does not currently cover cleanly:
47
 
48
- - freezes the uploaded video as LTX video latents while leaving matching audio
 
 
 
49
  latents empty for Foley generation
50
- - trims or pads the saved video frames to the same generation window
51
- - decodes LTX audio VAE output into the Comfy audio tensor layout expected by
52
  current video saving nodes
 
 
 
53
 
54
- Prompt text, model loading, LoRA loading, sampling, video creation, and MP4
55
- saving use normal ComfyUI/LTXVideo nodes.
56
 
57
  ## Fastest RunPod Test
58
 
@@ -64,24 +70,35 @@ https://console.runpod.io/deploy?template=cw3nka7d08&ref=k7b1cgii
64
  1. In RunPod, under "Additional Filters" filter CUDA versions to CUDA 12.8.
65
  2. Select a 48 GB GPU: A40, RTX A6000, L40/L40S, or A100.
66
  3. Make sure the `ComfyUI - CUDA 12.8` template is selected.
67
- 4. The template's default volume disk is `50 GB`, which is enough for the core
68
- workflow files, but tight once caches and reruns accumulate. Change the volume
69
- disk to `100 GB` if you want more breathing room.
70
  5. Start the pod and open a terminal.
71
  6. Run:
72
 
73
  ```bash
74
- cd /workspace && curl -L https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-Workflow/resolve/main/setup_runpod_ltx_foley.sh -o setup_runpod_ltx_foley.sh && bash setup_runpod_ltx_foley.sh
 
 
 
 
 
 
 
 
 
 
75
  ```
76
 
77
- The setup script installs the nodes and models, downloads the tennis test video
78
- as `input.mp4`, restarts ComfyUI without stopping the pod, and waits until the UI
79
- responds on port `8188`.
 
 
 
80
 
81
  After the script finishes:
82
 
83
  1. Open ComfyUI from the RunPod web UI.
84
- 2. Under workflows, select `ltx_23_foley_v2a.json`.
85
  3. Hit `Run`.
86
 
87
  The default input video and prompt are already set:
@@ -103,9 +120,14 @@ It installs or refreshes:
103
 
104
  - `Lightricks/ComfyUI-LTXVideo`
105
  - `ltx_foley_v2a` helper nodes
 
106
  - `ltx_23_foley_v2a.json`
107
  - `tennis-no-sound.mp4`
108
 
 
 
 
 
109
  It downloads these model files:
110
 
111
  - Base checkpoint:
@@ -126,19 +148,12 @@ If you are not using the RunPod script:
126
  1. Install or update ComfyUI.
127
  2. Install the official LTXVideo custom nodes:
128
  `https://github.com/Lightricks/ComfyUI-LTXVideo`
129
- 3. Install the Foley helper nodes by copying or symlinking the workflow repo's
130
  `ltx_foley_v2a` folder into:
131
- `ComfyUI/custom_nodes/ltx_foley_v2a`
132
-
133
- If you only downloaded the workflow JSON, run these commands from your
134
- ComfyUI install directory to install the two helper-node files:
135
-
136
- ```bash
137
- mkdir -p custom_nodes/ltx_foley_v2a
138
- curl -L https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-Workflow/resolve/main/ltx_foley_v2a/__init__.py -o custom_nodes/ltx_foley_v2a/__init__.py
139
- curl -L https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-Workflow/resolve/main/ltx_foley_v2a/nodes.py -o custom_nodes/ltx_foley_v2a/nodes.py
140
- ```
141
- 4. Put the model files in:
142
  - checkpoint:
143
  [`ltx-2.3-22b-dev-fp8.safetensors`](https://huggingface.co/Lightricks/LTX-2.3-fp8/blob/main/ltx-2.3-22b-dev-fp8.safetensors)
144
  in `ComfyUI/models/checkpoints`
@@ -148,9 +163,21 @@ If you are not using the RunPod script:
148
  - Foley LoRA:
149
  [`ltx-2.3-foley-400-steps.safetensors`](https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-LoRA/blob/main/ltx-2.3-foley-400-steps.safetensors)
150
  in `ComfyUI/models/loras`
151
- 5. Restart ComfyUI.
152
- 6. Under workflows, select `ltx_23_foley_v2a.json`.
153
- 7. Hit `Run`.
 
 
 
 
 
 
 
 
 
 
 
 
154
 
155
  ## Workflow Defaults
156
 
@@ -159,24 +186,34 @@ If you are not using the RunPod script:
159
  - Negative prompt: anti-music/anti-vocal prompt
160
  - Conditioning size: `576x576`
161
  - Frame window: `89` frames
 
 
 
162
  - Sampling steps: `30`
163
  - Guidance: `4.0`
164
- - STG blocks: `14, 19`
 
165
  - LoRA strength: `1.0`
166
 
167
- The workflow uses the first `89` frames of the uploaded video. Shorter videos are
168
- padded by repeating the last frame. The saved video is trimmed or padded to the
169
- same frame count so the output video duration matches the generated audio.
 
 
 
 
 
 
 
170
 
171
  ## VRAM Notes
172
 
173
- Sampling is the VRAM peak. Start with a 48 GB GPU for the first reliable test.
174
  If you need to reduce memory use, try these changes in order:
175
 
176
  - reduce frames from `89` to `57`, `41`, or `25`
177
  - reduce conditioning size from `576x576` to `448x448` or `384x384`
178
  - reduce sampling steps from `30` to `20`
179
- - lower or disable STG if quality is acceptable
180
 
181
  Frame counts should stay one more than a multiple of 8:
182
 
@@ -184,7 +221,49 @@ Frame counts should stay one more than a multiple of 8:
184
  9, 17, 25, 33, 41, 49, 57, ..., 89, ..., 257
185
  ```
186
 
187
- ## Troubleshooting RunPod Setup
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
188
 
189
  If you rerun setup after a workflow or node update:
190
 
@@ -195,23 +274,19 @@ cd /workspace && bash setup_runpod_ltx_foley.sh
195
  The script will skip verified model files, refresh the workflow/helper nodes,
196
  and restart ComfyUI.
197
 
 
 
198
  If model downloads fail with authorization errors, accept the relevant Hugging
199
  Face model terms and rerun with `HF_TOKEN` set.
200
 
 
 
201
  Logs from the script-managed ComfyUI restart are written to:
202
 
203
  ```text
204
  /workspace/runpod-slim/comfyui-restart.log
205
  ```
206
 
207
- If ComfyUI reports missing `LTXFoley...` nodes after manual setup, verify that
208
- these files exist and then restart ComfyUI:
209
-
210
- ```text
211
- ComfyUI/custom_nodes/ltx_foley_v2a/__init__.py
212
- ComfyUI/custom_nodes/ltx_foley_v2a/nodes.py
213
- ```
214
-
215
  ## License Scope
216
 
217
  The files in this workflow repository are released under the Apache-2.0 license.
 
8
  - video-to-audio
9
  - audio-generation
10
  - workflow
11
+ - foley-lora
12
  ---
13
 
14
  # LTX 2.3 Foley V2A ComfyUI Workflow
15
 
16
+ This repository contains ready-to-test ComfyUI workflows for the
17
  [`FuzzPuppy/LTX-2.3-Foley-LoRA`](https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-LoRA)
18
  LoRA. The LoRA adds Foley sound effects to a silent input video using LTX-2.3:
19
+ given a video and a prompt describing the visible action, the loop workflow
20
  generates matching non-speech, non-music sound effects and saves a new MP4.
21
 
22
+ There are two workflows provided:
23
+ 1. `foley-sliding-window.json`: long-video workflow with overlapping audio windows and stitching.
24
+ 2. `ltx_23_foley_v2a.json`: original short-clip workflow.
 
 
25
 
26
+ If you want run a quick short test, use `ltx_23_foley_v2a.json`. Otherwise, use `foley-sliding-window.json` so you can generate longer audio while keeping memory under control.
27
 
28
+ ## Tutorial
29
  [![Watch the tutorial: using the LTX-2.3 Foley LoRA in ComfyUI](https://img.youtube.com/vi/qnHFDlrySmw/hqdefault.jpg)](https://youtu.be/qnHFDlrySmw)
30
 
31
  [Watch the tutorial on YouTube](https://youtu.be/qnHFDlrySmw)
32
 
33
  ## What Is Included
34
 
35
+ - `foley-sliding-window.json`: long-video workflow with overlapping audio windows and stitching.
36
+ - `ltx_23_foley_v2a.json`: original short-clip workflow.
37
  - `setup_runpod_ltx_foley.sh`: one-command RunPod setup script.
38
  - `ltx_foley_v2a`: small helper-node package.
39
  - `tennis-no-sound.mp4`: default silent test video for RunPod setup.
40
 
41
+ Both workflows require the `ltx_foley_v2a` helper-node package. If ComfyUI shows
42
+ missing nodes named `LTXFoleyForLoopOpen`, `LTXFoleyWindowSelect`,
43
+ `LTXFoleyVideoToAudioLatent`, or `LTXFoleyAudioVAEDecode`, the workflow JSON was
44
+ loaded before these helper nodes were installed into `ComfyUI/custom_nodes`.
45
 
46
  The helper-node package handles the workflow-specific pieces that stock ComfyUI
47
  does not currently cover cleanly:
48
 
49
+ - plans the window count from the uploaded video
50
+ - provides a small local ComfyUI for-loop so no external loop-node pack is needed
51
+ - splits longer videos into overlapping windows
52
+ - freezes each source window as LTX video latents while leaving matching audio
53
  latents empty for Foley generation
54
+ - decodes each audio window into the Comfy audio tensor layout expected by
 
55
  current video saving nodes
56
+ - writes each raw decoded window as a WAV before stitching so artifacts can be
57
+ checked before the final crossfade
58
+ - crossfades and stitches generated audio windows into one final track
59
 
60
+ Prompt text, model loading, LoRA loading, video creation, and MP4 saving use
61
+ normal ComfyUI/LTXVideo nodes.
62
 
63
  ## Fastest RunPod Test
64
 
 
70
  1. In RunPod, under "Additional Filters" filter CUDA versions to CUDA 12.8.
71
  2. Select a 48 GB GPU: A40, RTX A6000, L40/L40S, or A100.
72
  3. Make sure the `ComfyUI - CUDA 12.8` template is selected.
73
+ 4. The template's default volume disk is `50 GB`, which is enough for the core workflow files, but tight once caches and reruns accumulate. Change the volume disk to `100 GB` if you want more breathing room.
 
 
74
  5. Start the pod and open a terminal.
75
  6. Run:
76
 
77
  ```bash
78
+ cd /workspace
79
+ curl -L https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-Workflow/resolve/main/setup_runpod_ltx_foley.sh -o setup_runpod_ltx_foley.sh
80
+ bash setup_runpod_ltx_foley.sh
81
+ ```
82
+
83
+ The setup script installs the nodes and models, downloads the tennis test video as `input.mp4`, restarts ComfyUI without stopping the pod (with `--cache-classic`, see the Manual ComfyUI Install notes), and waits until the UI responds on port `8188`.
84
+
85
+ By default the script installs ComfyUI `v0.27.0`. To test another ComfyUI release, set `COMFYUI_CORE_REF`:
86
+
87
+ ```bash
88
+ COMFYUI_CORE_REF=v0.19.0 bash setup_runpod_ltx_foley.sh
89
  ```
90
 
91
+ To install workflow files from a different Hugging Face branch, set
92
+ `WORKFLOW_REVISION`:
93
+
94
+ ```bash
95
+ WORKFLOW_REVISION=windows bash setup_runpod_ltx_foley.sh
96
+ ```
97
 
98
  After the script finishes:
99
 
100
  1. Open ComfyUI from the RunPod web UI.
101
+ 2. Under workflows, select `foley-sliding-window.json`.
102
  3. Hit `Run`.
103
 
104
  The default input video and prompt are already set:
 
120
 
121
  - `Lightricks/ComfyUI-LTXVideo`
122
  - `ltx_foley_v2a` helper nodes
123
+ - `foley-sliding-window.json`
124
  - `ltx_23_foley_v2a.json`
125
  - `tennis-no-sound.mp4`
126
 
127
+ The script also applies a small compatibility patch to the installed
128
+ `ComfyUI-LTXVideo/pyramid_blending.py` file so current Kornia builds can import
129
+ the node pack on fresh ComfyUI installs.
130
+
131
  It downloads these model files:
132
 
133
  - Base checkpoint:
 
148
  1. Install or update ComfyUI.
149
  2. Install the official LTXVideo custom nodes:
150
  `https://github.com/Lightricks/ComfyUI-LTXVideo`
151
+ 3. Install the Foley helper nodes by placing the workflow repo's
152
  `ltx_foley_v2a` folder into:
153
+ `ComfyUI/custom_nodes/`
154
+ 4. Copy the either `foley-sliding-window.json` or `ltx_23_foley_v2a.json` into your ComfyUI user workflows folder. In a standard ComfyUI install this is:
155
+ `ComfyUI/user/default/workflows`.
156
+ 5. Put the model files in:
 
 
 
 
 
 
 
157
  - checkpoint:
158
  [`ltx-2.3-22b-dev-fp8.safetensors`](https://huggingface.co/Lightricks/LTX-2.3-fp8/blob/main/ltx-2.3-22b-dev-fp8.safetensors)
159
  in `ComfyUI/models/checkpoints`
 
163
  - Foley LoRA:
164
  [`ltx-2.3-foley-400-steps.safetensors`](https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-LoRA/blob/main/ltx-2.3-foley-400-steps.safetensors)
165
  in `ComfyUI/models/loras`
166
+ 6. Restart ComfyUI, starting it with the `--cache-classic` flag:
167
+
168
+ ```bash
169
+ python main.py --cache-classic
170
+ ```
171
+
172
+ On newer ComfyUI versions (`v0.27.0`+) the default caching mode is RAM-pressure
173
+ caching, which can evict node outputs in the middle of a run while the large
174
+ LTX models load. For `foley-sliding-window.json` that forces the window plan,
175
+ video decode, and model loaders to re-execute between windows, making long
176
+ runs much slower. `--cache-classic` keeps those outputs cached for the whole
177
+ run. The flag also exists on older releases such as `v0.19.0`, where it is
178
+ harmless.
179
+ 7. Under workflows, select `foley-sliding-window.json` or `ltx_23_foley_v2a.json`.
180
+ 8. Hit `Run`.
181
 
182
  ## Workflow Defaults
183
 
 
186
  - Negative prompt: anti-music/anti-vocal prompt
187
  - Conditioning size: `576x576`
188
  - Frame window: `89` frames
189
+ - Window overlap: `1.0` second
190
+ - Maximum windows: `16`
191
+ - Random ID: `42`
192
  - Sampling steps: `30`
193
  - Guidance: `4.0`
194
+ - Save window audio: `true`
195
+ - Window audio prefix: `ltx_foley_window`
196
  - LoRA strength: `1.0`
197
 
198
+ Advanced sampler/STG settings are visible nodes in the loop body:
199
+ sampler `euler_ancestral_cfg_pp`, STG scale `1.0`, rescale `0.7`, STG blocks
200
+ `14, 19`, max shift `2.05`, base shift `0.95`, terminal `0.1`.
201
+
202
+ The `foley-sliding-window.json` workflow uses the full uploaded video. Videos longer than the selected
203
+ window are processed as overlapping windows and stitched into one generated
204
+ audio track. Shorter videos are padded internally by repeating the last frame.
205
+ The saved MP4 uses the source frames plus the stitched generated audio.
206
+ Raw generated window WAVs are saved under ComfyUI's output directory in
207
+ `ltx_foley_windows/` and their paths are listed in the manifest output.
208
 
209
  ## VRAM Notes
210
 
211
+ Sampling is the VRAM peak.
212
  If you need to reduce memory use, try these changes in order:
213
 
214
  - reduce frames from `89` to `57`, `41`, or `25`
215
  - reduce conditioning size from `576x576` to `448x448` or `384x384`
216
  - reduce sampling steps from `30` to `20`
 
217
 
218
  Frame counts should stay one more than a multiple of 8:
219
 
 
221
  9, 17, 25, 33, 41, 49, 57, ..., 89, ..., 257
222
  ```
223
 
224
+ For l`foley-sliding-window.json`, the default `max_windows` is `16` so accidental very long inputs
225
+ fail clearly instead of running for hours. Increase it only when you expect the
226
+ extra runtime.
227
+
228
+ ## Troubleshooting
229
+
230
+ ### Missing Nodes
231
+
232
+ If ComfyUI reports missing `LTXFoley...` nodes after manual setup, verify that
233
+ these files exist and then restart ComfyUI:
234
+
235
+ ```text
236
+ ComfyUI/custom_nodes/ltx_foley_v2a/__init__.py
237
+ ComfyUI/custom_nodes/ltx_foley_v2a/nodes.py
238
+ ```
239
+
240
+ ### Models Reload Or Nodes Re-Execute Between Windows
241
+
242
+ If the log shows `planned N windows` repeating, or the checkpoint/text-encoder
243
+ reloading before every window of `foley-sliding-window.json`, ComfyUI is running
244
+ with its default RAM-pressure caching and is evicting node outputs mid-run.
245
+ Start ComfyUI with `--cache-classic` (the RunPod script already does this). The
246
+ generated audio is still correct either way — the re-execution only costs time.
247
+
248
+ ### Audio Artifacts On Some ComfyUI Versions
249
+
250
+ The workflows have been tested on ComfyUI `v0.27.0` and run
251
+ successfully there. However, on `v0.27.0` and newer ComfyUI versions generally, we have noticed that LTX-2.3 video-to-audio can produce a high-pitched squeak or audio artifacts in some generated audio.
252
+
253
+ If you notice the audio artifacts on a generation, rollback to `v0.19.0` of ComfyUI.
254
+
255
+ If you are using the RunPod setup you can rollback by simply:
256
+
257
+ ```bash
258
+ cd /workspace
259
+ COMFYUI_CORE_REF=v0.19.0 bash setup_runpod_ltx_foley.sh
260
+ ```
261
+
262
+ Then reload `foley-sliding-window.json` and run it again.
263
+
264
+ ### RunPod Setup
265
+
266
+ #### Restarting/Rerun
267
 
268
  If you rerun setup after a workflow or node update:
269
 
 
274
  The script will skip verified model files, refresh the workflow/helper nodes,
275
  and restart ComfyUI.
276
 
277
+ #### Model Downloads
278
+
279
  If model downloads fail with authorization errors, accept the relevant Hugging
280
  Face model terms and rerun with `HF_TOKEN` set.
281
 
282
+ #### Logs
283
+
284
  Logs from the script-managed ComfyUI restart are written to:
285
 
286
  ```text
287
  /workspace/runpod-slim/comfyui-restart.log
288
  ```
289
 
 
 
 
 
 
 
 
 
290
  ## License Scope
291
 
292
  The files in this workflow repository are released under the Apache-2.0 license.
foley-sliding-window.json ADDED
@@ -0,0 +1,1655 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "last_node_id": 28,
3
+ "last_link_id": 47,
4
+ "nodes": [
5
+ {
6
+ "id": 1,
7
+ "type": "LoadVideo",
8
+ "pos": [
9
+ -1600,
10
+ -420
11
+ ],
12
+ "size": [
13
+ 320,
14
+ 90
15
+ ],
16
+ "flags": {},
17
+ "order": 0,
18
+ "mode": 0,
19
+ "inputs": [],
20
+ "outputs": [
21
+ {
22
+ "name": "video",
23
+ "type": "VIDEO",
24
+ "links": [
25
+ 1
26
+ ]
27
+ }
28
+ ],
29
+ "properties": {
30
+ "Node name for S&R": "LoadVideo"
31
+ },
32
+ "widgets_values": [
33
+ "input.mp4"
34
+ ]
35
+ },
36
+ {
37
+ "id": 2,
38
+ "type": "GetVideoComponents",
39
+ "pos": [
40
+ -1220,
41
+ -420
42
+ ],
43
+ "size": [
44
+ 220,
45
+ 120
46
+ ],
47
+ "flags": {},
48
+ "order": 1,
49
+ "mode": 0,
50
+ "inputs": [
51
+ {
52
+ "name": "video",
53
+ "type": "VIDEO",
54
+ "link": 1
55
+ }
56
+ ],
57
+ "outputs": [
58
+ {
59
+ "name": "images",
60
+ "type": "IMAGE",
61
+ "links": [
62
+ 2,
63
+ 7,
64
+ 44
65
+ ]
66
+ },
67
+ {
68
+ "name": "audio",
69
+ "type": "AUDIO",
70
+ "links": null
71
+ },
72
+ {
73
+ "name": "fps",
74
+ "type": "FLOAT",
75
+ "links": [
76
+ 3,
77
+ 4,
78
+ 21,
79
+ 45
80
+ ]
81
+ },
82
+ {
83
+ "name": "bit_depth",
84
+ "type": "INT",
85
+ "links": null
86
+ }
87
+ ],
88
+ "properties": {
89
+ "Node name for S&R": "GetVideoComponents"
90
+ },
91
+ "widgets_values": []
92
+ },
93
+ {
94
+ "id": 3,
95
+ "type": "CheckpointLoaderSimple",
96
+ "pos": [
97
+ -1600,
98
+ -160
99
+ ],
100
+ "size": [
101
+ 360,
102
+ 110
103
+ ],
104
+ "flags": {},
105
+ "order": 2,
106
+ "mode": 0,
107
+ "inputs": [],
108
+ "outputs": [
109
+ {
110
+ "name": "MODEL",
111
+ "type": "MODEL",
112
+ "links": [
113
+ 10
114
+ ]
115
+ },
116
+ {
117
+ "name": "CLIP",
118
+ "type": "CLIP",
119
+ "links": null
120
+ },
121
+ {
122
+ "name": "VAE",
123
+ "type": "VAE",
124
+ "links": [
125
+ 19
126
+ ]
127
+ }
128
+ ],
129
+ "properties": {
130
+ "Node name for S&R": "CheckpointLoaderSimple"
131
+ },
132
+ "widgets_values": [
133
+ "ltx-2.3-22b-dev-fp8.safetensors"
134
+ ]
135
+ },
136
+ {
137
+ "id": 4,
138
+ "type": "LTXVAudioVAELoader",
139
+ "pos": [
140
+ -1600,
141
+ 0
142
+ ],
143
+ "size": [
144
+ 360,
145
+ 80
146
+ ],
147
+ "flags": {},
148
+ "order": 3,
149
+ "mode": 0,
150
+ "inputs": [],
151
+ "outputs": [
152
+ {
153
+ "name": "Audio VAE",
154
+ "type": "VAE",
155
+ "links": [
156
+ 20,
157
+ 35
158
+ ]
159
+ }
160
+ ],
161
+ "properties": {
162
+ "Node name for S&R": "LTXVAudioVAELoader"
163
+ },
164
+ "widgets_values": [
165
+ "ltx-2.3-22b-dev-fp8.safetensors"
166
+ ]
167
+ },
168
+ {
169
+ "id": 5,
170
+ "type": "LTXAVTextEncoderLoader",
171
+ "pos": [
172
+ -1600,
173
+ 150
174
+ ],
175
+ "size": [
176
+ 360,
177
+ 120
178
+ ],
179
+ "flags": {},
180
+ "order": 4,
181
+ "mode": 0,
182
+ "inputs": [],
183
+ "outputs": [
184
+ {
185
+ "name": "CLIP",
186
+ "type": "CLIP",
187
+ "links": [
188
+ 12,
189
+ 13
190
+ ]
191
+ }
192
+ ],
193
+ "properties": {
194
+ "Node name for S&R": "LTXAVTextEncoderLoader"
195
+ },
196
+ "widgets_values": [
197
+ "gemma_3_12B_it_fp8_scaled.safetensors",
198
+ "ltx-2.3-22b-dev-fp8.safetensors",
199
+ "default"
200
+ ]
201
+ },
202
+ {
203
+ "id": 6,
204
+ "type": "CLIPTextEncode",
205
+ "pos": [
206
+ -1180,
207
+ 140
208
+ ],
209
+ "size": [
210
+ 420,
211
+ 180
212
+ ],
213
+ "flags": {},
214
+ "order": 5,
215
+ "mode": 0,
216
+ "inputs": [
217
+ {
218
+ "name": "clip",
219
+ "type": "CLIP",
220
+ "link": 12
221
+ }
222
+ ],
223
+ "outputs": [
224
+ {
225
+ "name": "CONDITIONING",
226
+ "type": "CONDITIONING",
227
+ "links": [
228
+ 14
229
+ ]
230
+ }
231
+ ],
232
+ "properties": {
233
+ "Node name for S&R": "CLIPTextEncode"
234
+ },
235
+ "widgets_values": [
236
+ "Two men are playing tennis. No speech is present. No music is present."
237
+ ],
238
+ "title": "Positive Prompt"
239
+ },
240
+ {
241
+ "id": 7,
242
+ "type": "CLIPTextEncode",
243
+ "pos": [
244
+ -1180,
245
+ 370
246
+ ],
247
+ "size": [
248
+ 420,
249
+ 160
250
+ ],
251
+ "flags": {},
252
+ "order": 6,
253
+ "mode": 0,
254
+ "inputs": [
255
+ {
256
+ "name": "clip",
257
+ "type": "CLIP",
258
+ "link": 13
259
+ }
260
+ ],
261
+ "outputs": [
262
+ {
263
+ "name": "CONDITIONING",
264
+ "type": "CONDITIONING",
265
+ "links": [
266
+ 15
267
+ ]
268
+ }
269
+ ],
270
+ "properties": {
271
+ "Node name for S&R": "CLIPTextEncode"
272
+ },
273
+ "widgets_values": [
274
+ "music, melody, song, singing, vocals, score, soundtrack, beat, rhythm bed, instrumental backing, tinny, thin, harsh, clipped, distorted, low bitrate"
275
+ ],
276
+ "title": "Negative Prompt"
277
+ },
278
+ {
279
+ "id": 8,
280
+ "type": "LTXVConditioning",
281
+ "pos": [
282
+ -700,
283
+ 300
284
+ ],
285
+ "size": [
286
+ 220,
287
+ 90
288
+ ],
289
+ "flags": {},
290
+ "order": 7,
291
+ "mode": 0,
292
+ "inputs": [
293
+ {
294
+ "name": "positive",
295
+ "type": "CONDITIONING",
296
+ "link": 14
297
+ },
298
+ {
299
+ "name": "negative",
300
+ "type": "CONDITIONING",
301
+ "link": 15
302
+ },
303
+ {
304
+ "name": "frame_rate",
305
+ "type": "FLOAT",
306
+ "link": 3
307
+ }
308
+ ],
309
+ "outputs": [
310
+ {
311
+ "name": "positive",
312
+ "type": "CONDITIONING",
313
+ "links": [
314
+ 16
315
+ ]
316
+ },
317
+ {
318
+ "name": "negative",
319
+ "type": "CONDITIONING",
320
+ "links": [
321
+ 17
322
+ ]
323
+ }
324
+ ],
325
+ "properties": {
326
+ "Node name for S&R": "LTXVConditioning"
327
+ },
328
+ "widgets_values": [
329
+ 25.0
330
+ ]
331
+ },
332
+ {
333
+ "id": 9,
334
+ "type": "LTXFoleyWindowPlan",
335
+ "pos": [
336
+ -860,
337
+ -430
338
+ ],
339
+ "size": [
340
+ 320,
341
+ 210
342
+ ],
343
+ "flags": {},
344
+ "order": 8,
345
+ "mode": 0,
346
+ "inputs": [
347
+ {
348
+ "name": "images",
349
+ "type": "IMAGE",
350
+ "link": 2
351
+ },
352
+ {
353
+ "name": "frame_rate",
354
+ "type": "FLOAT",
355
+ "link": 4
356
+ }
357
+ ],
358
+ "outputs": [
359
+ {
360
+ "name": "window_plan",
361
+ "type": "FOLEY_WINDOW_PLAN",
362
+ "links": [
363
+ 8,
364
+ 43
365
+ ]
366
+ },
367
+ {
368
+ "name": "window_count",
369
+ "type": "INT",
370
+ "links": [
371
+ 6
372
+ ]
373
+ },
374
+ {
375
+ "name": "manifest",
376
+ "type": "STRING",
377
+ "links": null
378
+ }
379
+ ],
380
+ "properties": {
381
+ "Node name for S&R": "LTXFoleyWindowPlan"
382
+ },
383
+ "widgets_values": [
384
+ 25.0,
385
+ 89,
386
+ 1.0,
387
+ 16
388
+ ]
389
+ },
390
+ {
391
+ "id": 10,
392
+ "type": "LoraLoaderModelOnly",
393
+ "pos": [
394
+ -1180,
395
+ -160
396
+ ],
397
+ "size": [
398
+ 360,
399
+ 90
400
+ ],
401
+ "flags": {},
402
+ "order": 9,
403
+ "mode": 0,
404
+ "inputs": [
405
+ {
406
+ "name": "model",
407
+ "type": "MODEL",
408
+ "link": 10
409
+ }
410
+ ],
411
+ "outputs": [
412
+ {
413
+ "name": "MODEL",
414
+ "type": "MODEL",
415
+ "links": [
416
+ 11
417
+ ]
418
+ }
419
+ ],
420
+ "properties": {
421
+ "Node name for S&R": "LoraLoaderModelOnly"
422
+ },
423
+ "widgets_values": [
424
+ "ltx-2.3-foley-400-steps.safetensors",
425
+ 1.0
426
+ ]
427
+ },
428
+ {
429
+ "id": 11,
430
+ "type": "LTXFoleyForLoopOpen",
431
+ "pos": [
432
+ -500,
433
+ -430
434
+ ],
435
+ "size": [
436
+ 300,
437
+ 170
438
+ ],
439
+ "flags": {},
440
+ "order": 10,
441
+ "mode": 0,
442
+ "inputs": [
443
+ {
444
+ "name": "remaining",
445
+ "type": "INT",
446
+ "link": 6
447
+ },
448
+ {
449
+ "name": "audio_accumulation",
450
+ "type": "FOLEY_AUDIO_ACCUM",
451
+ "link": null,
452
+ "shape": 7
453
+ }
454
+ ],
455
+ "outputs": [
456
+ {
457
+ "name": "flow_control",
458
+ "type": "FLOW_CONTROL",
459
+ "links": [
460
+ 40
461
+ ]
462
+ },
463
+ {
464
+ "name": "remaining",
465
+ "type": "INT",
466
+ "links": [
467
+ 9
468
+ ]
469
+ },
470
+ {
471
+ "name": "audio_accumulation",
472
+ "type": "FOLEY_AUDIO_ACCUM",
473
+ "links": [
474
+ 39
475
+ ]
476
+ }
477
+ ],
478
+ "properties": {
479
+ "Node name for S&R": "LTXFoleyForLoopOpen"
480
+ },
481
+ "widgets_values": []
482
+ },
483
+ {
484
+ "id": 12,
485
+ "type": "LTXFoleyWindowSelect",
486
+ "pos": [
487
+ -140,
488
+ -430
489
+ ],
490
+ "size": [
491
+ 320,
492
+ 140
493
+ ],
494
+ "flags": {},
495
+ "order": 11,
496
+ "mode": 0,
497
+ "inputs": [
498
+ {
499
+ "name": "images",
500
+ "type": "IMAGE",
501
+ "link": 7
502
+ },
503
+ {
504
+ "name": "window_plan",
505
+ "type": "FOLEY_WINDOW_PLAN",
506
+ "link": 8
507
+ },
508
+ {
509
+ "name": "remaining",
510
+ "type": "INT",
511
+ "link": 9
512
+ }
513
+ ],
514
+ "outputs": [
515
+ {
516
+ "name": "window_images",
517
+ "type": "IMAGE",
518
+ "links": [
519
+ 18
520
+ ]
521
+ },
522
+ {
523
+ "name": "window_info",
524
+ "type": "FOLEY_WINDOW",
525
+ "links": [
526
+ 37
527
+ ]
528
+ }
529
+ ],
530
+ "properties": {
531
+ "Node name for S&R": "LTXFoleyWindowSelect"
532
+ },
533
+ "widgets_values": []
534
+ },
535
+ {
536
+ "id": 13,
537
+ "type": "LTXFoleyVideoToAudioLatent",
538
+ "pos": [
539
+ -140,
540
+ -220
541
+ ],
542
+ "size": [
543
+ 360,
544
+ 340
545
+ ],
546
+ "flags": {},
547
+ "order": 12,
548
+ "mode": 0,
549
+ "inputs": [
550
+ {
551
+ "name": "images",
552
+ "type": "IMAGE",
553
+ "link": 18
554
+ },
555
+ {
556
+ "name": "positive",
557
+ "type": "CONDITIONING",
558
+ "link": 16
559
+ },
560
+ {
561
+ "name": "negative",
562
+ "type": "CONDITIONING",
563
+ "link": 17
564
+ },
565
+ {
566
+ "name": "video_vae",
567
+ "type": "VAE",
568
+ "link": 19
569
+ },
570
+ {
571
+ "name": "audio_vae",
572
+ "type": "VAE",
573
+ "link": 20
574
+ },
575
+ {
576
+ "name": "frame_rate",
577
+ "type": "FLOAT",
578
+ "link": 21
579
+ }
580
+ ],
581
+ "outputs": [
582
+ {
583
+ "name": "positive",
584
+ "type": "CONDITIONING",
585
+ "links": [
586
+ 25
587
+ ]
588
+ },
589
+ {
590
+ "name": "negative",
591
+ "type": "CONDITIONING",
592
+ "links": [
593
+ 26
594
+ ]
595
+ },
596
+ {
597
+ "name": "av_latent",
598
+ "type": "LATENT",
599
+ "links": [
600
+ 22,
601
+ 31,
602
+ 32
603
+ ]
604
+ },
605
+ {
606
+ "name": "preview_images",
607
+ "type": "IMAGE",
608
+ "links": null
609
+ },
610
+ {
611
+ "name": "frame_rate",
612
+ "type": "FLOAT",
613
+ "links": null
614
+ },
615
+ {
616
+ "name": "frames",
617
+ "type": "INT",
618
+ "links": null
619
+ }
620
+ ],
621
+ "properties": {
622
+ "Node name for S&R": "LTXFoleyVideoToAudioLatent"
623
+ },
624
+ "widgets_values": [
625
+ 25.0,
626
+ 576,
627
+ 576,
628
+ 89
629
+ ]
630
+ },
631
+ {
632
+ "id": 14,
633
+ "type": "ModelSamplingLTXV",
634
+ "pos": [
635
+ 260,
636
+ -170
637
+ ],
638
+ "size": [
639
+ 260,
640
+ 120
641
+ ],
642
+ "flags": {},
643
+ "order": 13,
644
+ "mode": 0,
645
+ "inputs": [
646
+ {
647
+ "name": "model",
648
+ "type": "MODEL",
649
+ "link": 11
650
+ },
651
+ {
652
+ "name": "latent",
653
+ "type": "LATENT",
654
+ "link": 22
655
+ }
656
+ ],
657
+ "outputs": [
658
+ {
659
+ "name": "MODEL",
660
+ "type": "MODEL",
661
+ "links": [
662
+ 23
663
+ ]
664
+ }
665
+ ],
666
+ "properties": {
667
+ "Node name for S&R": "ModelSamplingLTXV"
668
+ },
669
+ "widgets_values": [
670
+ 2.05,
671
+ 0.95
672
+ ]
673
+ },
674
+ {
675
+ "id": 15,
676
+ "type": "LTXVApplySTG",
677
+ "pos": [
678
+ 560,
679
+ -160
680
+ ],
681
+ "size": [
682
+ 240,
683
+ 80
684
+ ],
685
+ "flags": {},
686
+ "order": 14,
687
+ "mode": 0,
688
+ "inputs": [
689
+ {
690
+ "name": "model",
691
+ "type": "MODEL",
692
+ "link": 23
693
+ }
694
+ ],
695
+ "outputs": [
696
+ {
697
+ "name": "model",
698
+ "type": "MODEL",
699
+ "links": [
700
+ 24
701
+ ]
702
+ }
703
+ ],
704
+ "properties": {
705
+ "Node name for S&R": "LTXVApplySTG"
706
+ },
707
+ "widgets_values": [
708
+ "14, 19"
709
+ ]
710
+ },
711
+ {
712
+ "id": 16,
713
+ "type": "STGGuiderNode",
714
+ "pos": [
715
+ 840,
716
+ -180
717
+ ],
718
+ "size": [
719
+ 240,
720
+ 150
721
+ ],
722
+ "flags": {},
723
+ "order": 15,
724
+ "mode": 0,
725
+ "inputs": [
726
+ {
727
+ "name": "model",
728
+ "type": "MODEL",
729
+ "link": 24
730
+ },
731
+ {
732
+ "name": "positive",
733
+ "type": "CONDITIONING",
734
+ "link": 25
735
+ },
736
+ {
737
+ "name": "negative",
738
+ "type": "CONDITIONING",
739
+ "link": 26
740
+ }
741
+ ],
742
+ "outputs": [
743
+ {
744
+ "name": "GUIDER",
745
+ "type": "GUIDER",
746
+ "links": [
747
+ 28
748
+ ]
749
+ }
750
+ ],
751
+ "properties": {
752
+ "Node name for S&R": "STGGuiderNode"
753
+ },
754
+ "widgets_values": [
755
+ 4.0,
756
+ 1.0,
757
+ 0.7
758
+ ]
759
+ },
760
+ {
761
+ "id": 17,
762
+ "type": "RandomNoise",
763
+ "pos": [
764
+ 560,
765
+ -430
766
+ ],
767
+ "size": [
768
+ 240,
769
+ 90
770
+ ],
771
+ "flags": {},
772
+ "order": 16,
773
+ "mode": 0,
774
+ "inputs": [],
775
+ "outputs": [
776
+ {
777
+ "name": "NOISE",
778
+ "type": "NOISE",
779
+ "links": [
780
+ 27
781
+ ]
782
+ }
783
+ ],
784
+ "properties": {
785
+ "Node name for S&R": "RandomNoise"
786
+ },
787
+ "widgets_values": [
788
+ 42,
789
+ "fixed"
790
+ ]
791
+ },
792
+ {
793
+ "id": 18,
794
+ "type": "KSamplerSelect",
795
+ "pos": [
796
+ 560,
797
+ -300
798
+ ],
799
+ "size": [
800
+ 240,
801
+ 80
802
+ ],
803
+ "flags": {},
804
+ "order": 17,
805
+ "mode": 0,
806
+ "inputs": [],
807
+ "outputs": [
808
+ {
809
+ "name": "SAMPLER",
810
+ "type": "SAMPLER",
811
+ "links": [
812
+ 29
813
+ ]
814
+ }
815
+ ],
816
+ "properties": {
817
+ "Node name for S&R": "KSamplerSelect"
818
+ },
819
+ "widgets_values": [
820
+ "euler_ancestral_cfg_pp"
821
+ ]
822
+ },
823
+ {
824
+ "id": 19,
825
+ "type": "LTXVScheduler",
826
+ "pos": [
827
+ 840,
828
+ 20
829
+ ],
830
+ "size": [
831
+ 240,
832
+ 160
833
+ ],
834
+ "flags": {},
835
+ "order": 18,
836
+ "mode": 0,
837
+ "inputs": [
838
+ {
839
+ "name": "latent",
840
+ "type": "LATENT",
841
+ "link": 32
842
+ }
843
+ ],
844
+ "outputs": [
845
+ {
846
+ "name": "SIGMAS",
847
+ "type": "SIGMAS",
848
+ "links": [
849
+ 30
850
+ ]
851
+ }
852
+ ],
853
+ "properties": {
854
+ "Node name for S&R": "LTXVScheduler"
855
+ },
856
+ "widgets_values": [
857
+ 30,
858
+ 2.05,
859
+ 0.95,
860
+ true,
861
+ 0.1
862
+ ]
863
+ },
864
+ {
865
+ "id": 20,
866
+ "type": "SamplerCustomAdvanced",
867
+ "pos": [
868
+ 1140,
869
+ -220
870
+ ],
871
+ "size": [
872
+ 230,
873
+ 130
874
+ ],
875
+ "flags": {},
876
+ "order": 19,
877
+ "mode": 0,
878
+ "inputs": [
879
+ {
880
+ "name": "noise",
881
+ "type": "NOISE",
882
+ "link": 27
883
+ },
884
+ {
885
+ "name": "guider",
886
+ "type": "GUIDER",
887
+ "link": 28
888
+ },
889
+ {
890
+ "name": "sampler",
891
+ "type": "SAMPLER",
892
+ "link": 29
893
+ },
894
+ {
895
+ "name": "sigmas",
896
+ "type": "SIGMAS",
897
+ "link": 30
898
+ },
899
+ {
900
+ "name": "latent_image",
901
+ "type": "LATENT",
902
+ "link": 31
903
+ }
904
+ ],
905
+ "outputs": [
906
+ {
907
+ "name": "output",
908
+ "type": "LATENT",
909
+ "links": [
910
+ 33
911
+ ]
912
+ },
913
+ {
914
+ "name": "denoised_output",
915
+ "type": "LATENT",
916
+ "links": null
917
+ }
918
+ ],
919
+ "properties": {
920
+ "Node name for S&R": "SamplerCustomAdvanced"
921
+ },
922
+ "widgets_values": []
923
+ },
924
+ {
925
+ "id": 21,
926
+ "type": "LTXVSeparateAVLatent",
927
+ "pos": [
928
+ 1420,
929
+ -220
930
+ ],
931
+ "size": [
932
+ 220,
933
+ 80
934
+ ],
935
+ "flags": {},
936
+ "order": 20,
937
+ "mode": 0,
938
+ "inputs": [
939
+ {
940
+ "name": "av_latent",
941
+ "type": "LATENT",
942
+ "link": 33
943
+ }
944
+ ],
945
+ "outputs": [
946
+ {
947
+ "name": "video_latent",
948
+ "type": "LATENT",
949
+ "links": null
950
+ },
951
+ {
952
+ "name": "audio_latent",
953
+ "type": "LATENT",
954
+ "links": [
955
+ 34
956
+ ]
957
+ }
958
+ ],
959
+ "properties": {
960
+ "Node name for S&R": "LTXVSeparateAVLatent"
961
+ },
962
+ "widgets_values": []
963
+ },
964
+ {
965
+ "id": 22,
966
+ "type": "LTXFoleyAudioVAEDecode",
967
+ "pos": [
968
+ 1680,
969
+ -220
970
+ ],
971
+ "size": [
972
+ 240,
973
+ 90
974
+ ],
975
+ "flags": {},
976
+ "order": 21,
977
+ "mode": 0,
978
+ "inputs": [
979
+ {
980
+ "name": "samples",
981
+ "type": "LATENT",
982
+ "link": 34
983
+ },
984
+ {
985
+ "name": "audio_vae",
986
+ "type": "VAE",
987
+ "link": 35
988
+ }
989
+ ],
990
+ "outputs": [
991
+ {
992
+ "name": "Audio",
993
+ "type": "AUDIO",
994
+ "links": [
995
+ 36
996
+ ]
997
+ }
998
+ ],
999
+ "properties": {
1000
+ "Node name for S&R": "LTXFoleyAudioVAEDecode"
1001
+ },
1002
+ "widgets_values": []
1003
+ },
1004
+ {
1005
+ "id": 23,
1006
+ "type": "LTXFoleyWindowAudioSave",
1007
+ "pos": [
1008
+ 1980,
1009
+ -260
1010
+ ],
1011
+ "size": [
1012
+ 300,
1013
+ 150
1014
+ ],
1015
+ "flags": {},
1016
+ "order": 22,
1017
+ "mode": 0,
1018
+ "inputs": [
1019
+ {
1020
+ "name": "audio",
1021
+ "type": "AUDIO",
1022
+ "link": 36
1023
+ },
1024
+ {
1025
+ "name": "window_info",
1026
+ "type": "FOLEY_WINDOW",
1027
+ "link": 37
1028
+ }
1029
+ ],
1030
+ "outputs": [
1031
+ {
1032
+ "name": "audio",
1033
+ "type": "AUDIO",
1034
+ "links": null
1035
+ },
1036
+ {
1037
+ "name": "window_record",
1038
+ "type": "FOLEY_WINDOW_RECORD",
1039
+ "links": [
1040
+ 38
1041
+ ]
1042
+ }
1043
+ ],
1044
+ "properties": {
1045
+ "Node name for S&R": "LTXFoleyWindowAudioSave"
1046
+ },
1047
+ "widgets_values": [
1048
+ true,
1049
+ "ltx_foley_window"
1050
+ ]
1051
+ },
1052
+ {
1053
+ "id": 24,
1054
+ "type": "LTXFoleyAudioAccumulator",
1055
+ "pos": [
1056
+ 2320,
1057
+ -300
1058
+ ],
1059
+ "size": [
1060
+ 260,
1061
+ 120
1062
+ ],
1063
+ "flags": {},
1064
+ "order": 23,
1065
+ "mode": 0,
1066
+ "inputs": [
1067
+ {
1068
+ "name": "window_record",
1069
+ "type": "FOLEY_WINDOW_RECORD",
1070
+ "link": 38
1071
+ },
1072
+ {
1073
+ "name": "accumulation",
1074
+ "type": "FOLEY_AUDIO_ACCUM",
1075
+ "link": 39
1076
+ }
1077
+ ],
1078
+ "outputs": [
1079
+ {
1080
+ "name": "accumulation",
1081
+ "type": "FOLEY_AUDIO_ACCUM",
1082
+ "links": [
1083
+ 41
1084
+ ]
1085
+ }
1086
+ ],
1087
+ "properties": {
1088
+ "Node name for S&R": "LTXFoleyAudioAccumulator"
1089
+ },
1090
+ "widgets_values": []
1091
+ },
1092
+ {
1093
+ "id": 25,
1094
+ "type": "LTXFoleyForLoopClose",
1095
+ "pos": [
1096
+ 2660,
1097
+ -430
1098
+ ],
1099
+ "size": [
1100
+ 260,
1101
+ 120
1102
+ ],
1103
+ "flags": {},
1104
+ "order": 24,
1105
+ "mode": 0,
1106
+ "inputs": [
1107
+ {
1108
+ "name": "flow_control",
1109
+ "type": "FLOW_CONTROL",
1110
+ "link": 40
1111
+ },
1112
+ {
1113
+ "name": "audio_accumulation",
1114
+ "type": "FOLEY_AUDIO_ACCUM",
1115
+ "link": 41,
1116
+ "shape": 7
1117
+ }
1118
+ ],
1119
+ "outputs": [
1120
+ {
1121
+ "name": "audio_accumulation",
1122
+ "type": "FOLEY_AUDIO_ACCUM",
1123
+ "links": [
1124
+ 42
1125
+ ]
1126
+ }
1127
+ ],
1128
+ "properties": {
1129
+ "Node name for S&R": "LTXFoleyForLoopClose"
1130
+ },
1131
+ "widgets_values": []
1132
+ },
1133
+ {
1134
+ "id": 26,
1135
+ "type": "LTXFoleyAudioStitch",
1136
+ "pos": [
1137
+ 3020,
1138
+ -380
1139
+ ],
1140
+ "size": [
1141
+ 300,
1142
+ 110
1143
+ ],
1144
+ "flags": {},
1145
+ "order": 25,
1146
+ "mode": 0,
1147
+ "inputs": [
1148
+ {
1149
+ "name": "accumulation",
1150
+ "type": "FOLEY_AUDIO_ACCUM",
1151
+ "link": 42
1152
+ },
1153
+ {
1154
+ "name": "window_plan",
1155
+ "type": "FOLEY_WINDOW_PLAN",
1156
+ "link": 43
1157
+ }
1158
+ ],
1159
+ "outputs": [
1160
+ {
1161
+ "name": "audio",
1162
+ "type": "AUDIO",
1163
+ "links": [
1164
+ 46
1165
+ ]
1166
+ },
1167
+ {
1168
+ "name": "manifest",
1169
+ "type": "STRING",
1170
+ "links": null
1171
+ }
1172
+ ],
1173
+ "properties": {
1174
+ "Node name for S&R": "LTXFoleyAudioStitch"
1175
+ },
1176
+ "widgets_values": []
1177
+ },
1178
+ {
1179
+ "id": 27,
1180
+ "type": "CreateVideo",
1181
+ "pos": [
1182
+ 3380,
1183
+ -420
1184
+ ],
1185
+ "size": [
1186
+ 260,
1187
+ 140
1188
+ ],
1189
+ "flags": {},
1190
+ "order": 26,
1191
+ "mode": 0,
1192
+ "inputs": [
1193
+ {
1194
+ "name": "images",
1195
+ "type": "IMAGE",
1196
+ "link": 44
1197
+ },
1198
+ {
1199
+ "name": "fps",
1200
+ "type": "FLOAT",
1201
+ "link": 45
1202
+ },
1203
+ {
1204
+ "name": "audio",
1205
+ "type": "AUDIO",
1206
+ "link": 46
1207
+ }
1208
+ ],
1209
+ "outputs": [
1210
+ {
1211
+ "name": "video",
1212
+ "type": "VIDEO",
1213
+ "links": [
1214
+ 47
1215
+ ]
1216
+ }
1217
+ ],
1218
+ "properties": {
1219
+ "Node name for S&R": "CreateVideo"
1220
+ },
1221
+ "widgets_values": [
1222
+ 25.0,
1223
+ 8
1224
+ ]
1225
+ },
1226
+ {
1227
+ "id": 28,
1228
+ "type": "SaveVideo",
1229
+ "pos": [
1230
+ 3700,
1231
+ -420
1232
+ ],
1233
+ "size": [
1234
+ 260,
1235
+ 110
1236
+ ],
1237
+ "flags": {},
1238
+ "order": 27,
1239
+ "mode": 0,
1240
+ "inputs": [
1241
+ {
1242
+ "name": "video",
1243
+ "type": "VIDEO",
1244
+ "link": 47
1245
+ }
1246
+ ],
1247
+ "outputs": [
1248
+ {
1249
+ "name": "video",
1250
+ "type": "VIDEO",
1251
+ "links": null
1252
+ }
1253
+ ],
1254
+ "properties": {
1255
+ "Node name for S&R": "SaveVideo"
1256
+ },
1257
+ "widgets_values": [
1258
+ "ltx_foley_loop",
1259
+ "auto",
1260
+ "auto"
1261
+ ]
1262
+ }
1263
+ ],
1264
+ "links": [
1265
+ [
1266
+ 1,
1267
+ 1,
1268
+ 0,
1269
+ 2,
1270
+ 0,
1271
+ "VIDEO"
1272
+ ],
1273
+ [
1274
+ 2,
1275
+ 2,
1276
+ 0,
1277
+ 9,
1278
+ 0,
1279
+ "IMAGE"
1280
+ ],
1281
+ [
1282
+ 3,
1283
+ 2,
1284
+ 2,
1285
+ 8,
1286
+ 2,
1287
+ "FLOAT"
1288
+ ],
1289
+ [
1290
+ 4,
1291
+ 2,
1292
+ 2,
1293
+ 9,
1294
+ 1,
1295
+ "FLOAT"
1296
+ ],
1297
+ [
1298
+ 6,
1299
+ 9,
1300
+ 1,
1301
+ 11,
1302
+ 0,
1303
+ "INT"
1304
+ ],
1305
+ [
1306
+ 7,
1307
+ 2,
1308
+ 0,
1309
+ 12,
1310
+ 0,
1311
+ "IMAGE"
1312
+ ],
1313
+ [
1314
+ 8,
1315
+ 9,
1316
+ 0,
1317
+ 12,
1318
+ 1,
1319
+ "FOLEY_WINDOW_PLAN"
1320
+ ],
1321
+ [
1322
+ 9,
1323
+ 11,
1324
+ 1,
1325
+ 12,
1326
+ 2,
1327
+ "INT"
1328
+ ],
1329
+ [
1330
+ 10,
1331
+ 3,
1332
+ 0,
1333
+ 10,
1334
+ 0,
1335
+ "MODEL"
1336
+ ],
1337
+ [
1338
+ 11,
1339
+ 10,
1340
+ 0,
1341
+ 14,
1342
+ 0,
1343
+ "MODEL"
1344
+ ],
1345
+ [
1346
+ 12,
1347
+ 5,
1348
+ 0,
1349
+ 6,
1350
+ 0,
1351
+ "CLIP"
1352
+ ],
1353
+ [
1354
+ 13,
1355
+ 5,
1356
+ 0,
1357
+ 7,
1358
+ 0,
1359
+ "CLIP"
1360
+ ],
1361
+ [
1362
+ 14,
1363
+ 6,
1364
+ 0,
1365
+ 8,
1366
+ 0,
1367
+ "CONDITIONING"
1368
+ ],
1369
+ [
1370
+ 15,
1371
+ 7,
1372
+ 0,
1373
+ 8,
1374
+ 1,
1375
+ "CONDITIONING"
1376
+ ],
1377
+ [
1378
+ 16,
1379
+ 8,
1380
+ 0,
1381
+ 13,
1382
+ 1,
1383
+ "CONDITIONING"
1384
+ ],
1385
+ [
1386
+ 17,
1387
+ 8,
1388
+ 1,
1389
+ 13,
1390
+ 2,
1391
+ "CONDITIONING"
1392
+ ],
1393
+ [
1394
+ 18,
1395
+ 12,
1396
+ 0,
1397
+ 13,
1398
+ 0,
1399
+ "IMAGE"
1400
+ ],
1401
+ [
1402
+ 19,
1403
+ 3,
1404
+ 2,
1405
+ 13,
1406
+ 3,
1407
+ "VAE"
1408
+ ],
1409
+ [
1410
+ 20,
1411
+ 4,
1412
+ 0,
1413
+ 13,
1414
+ 4,
1415
+ "VAE"
1416
+ ],
1417
+ [
1418
+ 21,
1419
+ 2,
1420
+ 2,
1421
+ 13,
1422
+ 5,
1423
+ "FLOAT"
1424
+ ],
1425
+ [
1426
+ 22,
1427
+ 13,
1428
+ 2,
1429
+ 14,
1430
+ 1,
1431
+ "LATENT"
1432
+ ],
1433
+ [
1434
+ 23,
1435
+ 14,
1436
+ 0,
1437
+ 15,
1438
+ 0,
1439
+ "MODEL"
1440
+ ],
1441
+ [
1442
+ 24,
1443
+ 15,
1444
+ 0,
1445
+ 16,
1446
+ 0,
1447
+ "MODEL"
1448
+ ],
1449
+ [
1450
+ 25,
1451
+ 13,
1452
+ 0,
1453
+ 16,
1454
+ 1,
1455
+ "CONDITIONING"
1456
+ ],
1457
+ [
1458
+ 26,
1459
+ 13,
1460
+ 1,
1461
+ 16,
1462
+ 2,
1463
+ "CONDITIONING"
1464
+ ],
1465
+ [
1466
+ 27,
1467
+ 17,
1468
+ 0,
1469
+ 20,
1470
+ 0,
1471
+ "NOISE"
1472
+ ],
1473
+ [
1474
+ 28,
1475
+ 16,
1476
+ 0,
1477
+ 20,
1478
+ 1,
1479
+ "GUIDER"
1480
+ ],
1481
+ [
1482
+ 29,
1483
+ 18,
1484
+ 0,
1485
+ 20,
1486
+ 2,
1487
+ "SAMPLER"
1488
+ ],
1489
+ [
1490
+ 30,
1491
+ 19,
1492
+ 0,
1493
+ 20,
1494
+ 3,
1495
+ "SIGMAS"
1496
+ ],
1497
+ [
1498
+ 31,
1499
+ 13,
1500
+ 2,
1501
+ 20,
1502
+ 4,
1503
+ "LATENT"
1504
+ ],
1505
+ [
1506
+ 32,
1507
+ 13,
1508
+ 2,
1509
+ 19,
1510
+ 0,
1511
+ "LATENT"
1512
+ ],
1513
+ [
1514
+ 33,
1515
+ 20,
1516
+ 0,
1517
+ 21,
1518
+ 0,
1519
+ "LATENT"
1520
+ ],
1521
+ [
1522
+ 34,
1523
+ 21,
1524
+ 1,
1525
+ 22,
1526
+ 0,
1527
+ "LATENT"
1528
+ ],
1529
+ [
1530
+ 35,
1531
+ 4,
1532
+ 0,
1533
+ 22,
1534
+ 1,
1535
+ "VAE"
1536
+ ],
1537
+ [
1538
+ 36,
1539
+ 22,
1540
+ 0,
1541
+ 23,
1542
+ 0,
1543
+ "AUDIO"
1544
+ ],
1545
+ [
1546
+ 37,
1547
+ 12,
1548
+ 1,
1549
+ 23,
1550
+ 1,
1551
+ "FOLEY_WINDOW"
1552
+ ],
1553
+ [
1554
+ 38,
1555
+ 23,
1556
+ 1,
1557
+ 24,
1558
+ 0,
1559
+ "FOLEY_WINDOW_RECORD"
1560
+ ],
1561
+ [
1562
+ 39,
1563
+ 11,
1564
+ 2,
1565
+ 24,
1566
+ 1,
1567
+ "FOLEY_AUDIO_ACCUM"
1568
+ ],
1569
+ [
1570
+ 40,
1571
+ 11,
1572
+ 0,
1573
+ 25,
1574
+ 0,
1575
+ "FLOW_CONTROL"
1576
+ ],
1577
+ [
1578
+ 41,
1579
+ 24,
1580
+ 0,
1581
+ 25,
1582
+ 1,
1583
+ "FOLEY_AUDIO_ACCUM"
1584
+ ],
1585
+ [
1586
+ 42,
1587
+ 25,
1588
+ 0,
1589
+ 26,
1590
+ 0,
1591
+ "FOLEY_AUDIO_ACCUM"
1592
+ ],
1593
+ [
1594
+ 43,
1595
+ 9,
1596
+ 0,
1597
+ 26,
1598
+ 1,
1599
+ "FOLEY_WINDOW_PLAN"
1600
+ ],
1601
+ [
1602
+ 44,
1603
+ 2,
1604
+ 0,
1605
+ 27,
1606
+ 0,
1607
+ "IMAGE"
1608
+ ],
1609
+ [
1610
+ 45,
1611
+ 2,
1612
+ 2,
1613
+ 27,
1614
+ 1,
1615
+ "FLOAT"
1616
+ ],
1617
+ [
1618
+ 46,
1619
+ 26,
1620
+ 0,
1621
+ 27,
1622
+ 2,
1623
+ "AUDIO"
1624
+ ],
1625
+ [
1626
+ 47,
1627
+ 27,
1628
+ 0,
1629
+ 28,
1630
+ 0,
1631
+ "VIDEO"
1632
+ ]
1633
+ ],
1634
+ "groups": [],
1635
+ "config": {},
1636
+ "extra": {},
1637
+ "models": [
1638
+ {
1639
+ "name": "ltx-2.3-22b-dev-fp8.safetensors",
1640
+ "url": "https://huggingface.co/Lightricks/LTX-2.3-fp8/resolve/main/ltx-2.3-22b-dev-fp8.safetensors",
1641
+ "directory": "checkpoints"
1642
+ },
1643
+ {
1644
+ "name": "gemma_3_12B_it_fp8_scaled.safetensors",
1645
+ "url": "https://huggingface.co/Comfy-Org/ltx-2/resolve/main/split_files/text_encoders/gemma_3_12B_it_fp8_scaled.safetensors",
1646
+ "directory": "text_encoders"
1647
+ },
1648
+ {
1649
+ "name": "ltx-2.3-foley-400-steps.safetensors",
1650
+ "url": "https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-LoRA/resolve/main/ltx-2.3-foley-400-steps.safetensors",
1651
+ "directory": "loras"
1652
+ }
1653
+ ],
1654
+ "version": 0.4
1655
+ }
ltx_foley_v2a/nodes.py CHANGED
@@ -1,5 +1,22 @@
1
  from __future__ import annotations
2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  def _common_upscale(images, width: int, height: int):
4
  import comfy.utils
5
 
@@ -22,6 +39,438 @@ def _audio_vae_model(audio_vae):
22
  return getattr(audio_vae, "first_stage_model", audio_vae)
23
 
24
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
  class LTXFoleyVideoToAudioLatent:
26
  CATEGORY = "LTX/Foley"
27
  RETURN_TYPES = ("CONDITIONING", "CONDITIONING", "LATENT", "IMAGE", "FLOAT", "INT")
@@ -63,10 +512,7 @@ class LTXFoleyVideoToAudioLatent:
63
  if images.shape[0] == 0:
64
  raise ValueError("No video frames were provided")
65
 
66
- source = images[:frames]
67
- if source.shape[0] < frames:
68
- pad = source[-1:].repeat((frames - source.shape[0], 1, 1, 1))
69
- source = torch.cat([source, pad], dim=0)
70
 
71
  resized = _common_upscale(source, width, height).clamp(0.0, 1.0)
72
  video_latent = video_vae.encode(resized[:, :, :, :3])
@@ -131,6 +577,208 @@ class LTXFoleyTrimImages:
131
  return (trimmed, float(frame_rate))
132
 
133
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
  class LTXFoleyAudioVAEDecode:
135
  CATEGORY = "LTX/Foley"
136
  RETURN_TYPES = ("AUDIO",)
@@ -151,7 +799,8 @@ class LTXFoleyAudioVAEDecode:
151
  if audio_latent.is_nested:
152
  audio_latent = audio_latent.unbind()[-1]
153
 
154
- audio = audio_vae.decode(audio_latent).to(audio_latent.device)
 
155
  if audio.ndim == 2:
156
  audio = audio.unsqueeze(1)
157
  elif audio.ndim != 3:
@@ -169,12 +818,27 @@ class LTXFoleyAudioVAEDecode:
169
 
170
 
171
  NODE_CLASS_MAPPINGS = {
 
 
 
172
  "LTXFoleyVideoToAudioLatent": LTXFoleyVideoToAudioLatent,
173
  "LTXFoleyTrimImages": LTXFoleyTrimImages,
 
 
 
 
 
174
  "LTXFoleyAudioVAEDecode": LTXFoleyAudioVAEDecode,
175
  }
176
  NODE_DISPLAY_NAME_MAPPINGS = {
 
 
177
  "LTXFoleyVideoToAudioLatent": "LTX Foley Video To Audio Latent",
178
  "LTXFoleyTrimImages": "LTX Foley Trim Images",
 
 
 
 
 
179
  "LTXFoleyAudioVAEDecode": "LTX Foley Audio VAE Decode",
180
  }
 
1
  from __future__ import annotations
2
 
3
+ import json
4
+ import math
5
+ import re
6
+ import wave
7
+ from pathlib import Path
8
+
9
+
10
+ class _SmartType(str):
11
+ def __ne__(self, other):
12
+ if self == "*" or other == "*":
13
+ return False
14
+ return str.__ne__(self, other)
15
+
16
+
17
+ ANY_TYPE = _SmartType("*")
18
+
19
+
20
  def _common_upscale(images, width: int, height: int):
21
  import comfy.utils
22
 
 
39
  return getattr(audio_vae, "first_stage_model", audio_vae)
40
 
41
 
42
+ def _window_specs(source_frames: int, fps: float, window_frames: int, overlap_seconds: float) -> list[dict[str, object]]:
43
+ if window_frames % 8 != 1:
44
+ raise ValueError("window_frames must satisfy window_frames % 8 == 1")
45
+ if source_frames <= 0:
46
+ raise ValueError("source frame count must be positive")
47
+ if fps <= 0:
48
+ raise ValueError("frame_rate must be positive")
49
+ if overlap_seconds < 0:
50
+ raise ValueError("overlap_seconds must be non-negative")
51
+
52
+ frames = window_frames
53
+ if source_frames <= frames:
54
+ return [{"index": 1, "start_frame": 0, "start_seconds": 0.0, "frames": frames, "duration": frames / fps}]
55
+
56
+ overlap_frames = max(0, round(overlap_seconds * fps))
57
+ overlap_frames = min(overlap_frames, frames - 8)
58
+ hop_frames = max(8, frames - overlap_frames)
59
+ last_start = max(0, source_frames - frames)
60
+
61
+ starts = list(range(0, last_start + 1, hop_frames))
62
+ if starts[-1] != last_start:
63
+ starts.append(last_start)
64
+
65
+ return [
66
+ {
67
+ "index": index,
68
+ "start_frame": start_frame,
69
+ "start_seconds": start_frame / fps,
70
+ "frames": frames,
71
+ "duration": frames / fps,
72
+ }
73
+ for index, start_frame in enumerate(starts, start=1)
74
+ ]
75
+
76
+
77
+ def _slice_or_pad_frames(images, start_frame: int, frames: int):
78
+ import torch
79
+
80
+ if images.shape[0] == 0:
81
+ raise ValueError("No video frames were provided")
82
+ if start_frame < 0:
83
+ raise ValueError("start_frame must be non-negative")
84
+ if frames <= 0:
85
+ raise ValueError("frames must be positive")
86
+
87
+ source = images[start_frame : start_frame + frames]
88
+ if source.shape[0] == 0:
89
+ source = images[-1:]
90
+ if source.shape[0] < frames:
91
+ pad = source[-1:].repeat((frames - source.shape[0], 1, 1, 1))
92
+ source = torch.cat([source, pad], dim=0)
93
+ return source
94
+
95
+
96
+ def _audio_waveform(audio: dict[str, object]):
97
+ waveform = audio.get("waveform")
98
+ if waveform is None:
99
+ raise ValueError("Expected AUDIO dict to contain waveform")
100
+ if waveform.ndim == 2:
101
+ waveform = waveform.unsqueeze(0)
102
+ elif waveform.ndim != 3:
103
+ raise ValueError(f"Expected audio waveform to have 2 or 3 dimensions, got shape {tuple(waveform.shape)}")
104
+
105
+ if waveform.shape[1] in (1, 2, 6):
106
+ return waveform
107
+ if waveform.shape[2] in (1, 2, 6):
108
+ return waveform.movedim(-1, 1)
109
+ raise ValueError(f"Could not infer audio channel dimension from waveform shape {tuple(waveform.shape)}")
110
+
111
+
112
+ def _audio_stats(audio) -> dict[str, object]:
113
+ import torch
114
+
115
+ if audio.numel() == 0:
116
+ return {"samples": 0, "rms": 0.0, "peak": 0.0}
117
+ detached = audio.detach().float()
118
+ return {
119
+ "samples": int(detached.shape[-1]),
120
+ "rms": float(torch.sqrt(torch.mean(torch.square(detached))).cpu()),
121
+ "peak": float(torch.max(torch.abs(detached)).cpu()),
122
+ }
123
+
124
+
125
+ def _accum_record_count(accumulation) -> int:
126
+ if isinstance(accumulation, dict):
127
+ return len(accumulation.get("records", []))
128
+ return 0
129
+
130
+
131
+ def _loop_next_remaining(accumulation) -> int:
132
+ """Windows still to generate, derived from the latest accumulated record.
133
+
134
+ The loop countdown is carried inside the accumulation value instead of a
135
+ graph link to the loop-open node so the ephemeral loop nodes never need a
136
+ link back into the visible graph (see LTXFoleyForLoopClose docstring).
137
+ """
138
+ records = accumulation.get("records", []) if isinstance(accumulation, dict) else []
139
+ if not records:
140
+ raise ValueError("The sliding-window loop requires at least one accumulated window record")
141
+ info = records[-1]["window_info"]
142
+ window_count = int(info.get("window_count", info.get("planned_window_count", 0)))
143
+ if window_count <= 0:
144
+ raise ValueError("Accumulated window record does not carry a window count")
145
+ return window_count - int(info["spec"]["index"])
146
+
147
+
148
+ def _safe_filename(value: str, default: str = "ltx_foley_window") -> str:
149
+ cleaned = re.sub(r"[^A-Za-z0-9._-]+", "_", str(value)).strip("._")
150
+ return cleaned or default
151
+
152
+
153
+ def _comfy_output_directory() -> Path:
154
+ try:
155
+ import folder_paths
156
+
157
+ return Path(folder_paths.get_output_directory())
158
+ except Exception:
159
+ return Path.cwd()
160
+
161
+
162
+ def _write_audio_window(audio: dict[str, object], *, prefix: str, window_index: int) -> str:
163
+ import torch
164
+
165
+ sample_rate = int(audio["sample_rate"])
166
+ waveform = _audio_waveform(audio).detach().float().cpu()[0]
167
+ waveform = waveform.clamp(-1.0, 1.0)
168
+ pcm = (waveform.movedim(0, -1) * 32767.0).round().to(torch.int16).contiguous()
169
+
170
+ output_dir = _comfy_output_directory() / "ltx_foley_windows"
171
+ output_dir.mkdir(parents=True, exist_ok=True)
172
+ path = output_dir / f"{_safe_filename(prefix)}_{window_index:03d}.wav"
173
+
174
+ with wave.open(str(path), "wb") as handle:
175
+ handle.setnchannels(int(waveform.shape[0]))
176
+ handle.setsampwidth(2)
177
+ handle.setframerate(sample_rate)
178
+ handle.writeframes(pcm.numpy().tobytes())
179
+
180
+ return str(path)
181
+
182
+
183
+ def _stitch_audio_windows(
184
+ audio_windows: list[dict[str, object]],
185
+ window_specs: list[dict[str, object]],
186
+ output_duration: float,
187
+ overlap_seconds: float,
188
+ ) -> tuple[dict[str, object], list[dict[str, object]]]:
189
+ import torch
190
+
191
+ if not audio_windows:
192
+ raise ValueError("No window audio was generated")
193
+
194
+ sample_rate = int(audio_windows[0]["sample_rate"])
195
+ waveforms = [_audio_waveform(audio).detach().float().cpu() for audio in audio_windows]
196
+ channels = int(waveforms[0].shape[1])
197
+ total_samples = max(1, math.ceil(output_duration * sample_rate))
198
+ accum = torch.zeros((1, channels, total_samples), dtype=torch.float32)
199
+ weights = torch.zeros((1, 1, total_samples), dtype=torch.float32)
200
+ overlap_samples = max(1, round(overlap_seconds * sample_rate))
201
+
202
+ stats: list[dict[str, object]] = []
203
+ for index, (waveform, spec) in enumerate(zip(waveforms, window_specs, strict=True)):
204
+ current_rate = int(audio_windows[index]["sample_rate"])
205
+ if current_rate != sample_rate:
206
+ raise ValueError(f"Mismatched audio sample rate {current_rate}; expected {sample_rate}")
207
+ if waveform.shape[1] != channels:
208
+ raise ValueError(f"Mismatched audio channel count {waveform.shape[1]}; expected {channels}")
209
+
210
+ start = round(float(spec["start_seconds"]) * sample_rate)
211
+ if start >= total_samples:
212
+ continue
213
+ available = total_samples - start
214
+ waveform = waveform[:, :, :available]
215
+ if waveform.numel() == 0:
216
+ continue
217
+
218
+ window_weight = torch.ones((1, 1, waveform.shape[-1]), dtype=torch.float32)
219
+ if index > 0:
220
+ fade = min(overlap_samples, waveform.shape[-1])
221
+ window_weight[:, :, :fade] = torch.linspace(0.0, 1.0, fade, dtype=torch.float32).view(1, 1, -1)
222
+ if index < len(waveforms) - 1:
223
+ fade = min(overlap_samples, waveform.shape[-1])
224
+ fade_out = torch.linspace(1.0, 0.0, fade, dtype=torch.float32).view(1, 1, -1)
225
+ window_weight[:, :, -fade:] = torch.minimum(window_weight[:, :, -fade:], fade_out)
226
+
227
+ end = start + waveform.shape[-1]
228
+ accum[:, :, start:end] += waveform * window_weight
229
+ weights[:, :, start:end] += window_weight
230
+ stats.append({"window_index": int(spec["index"]), **_audio_stats(waveform)})
231
+
232
+ stitched = accum / torch.clamp(weights, min=1e-6)
233
+ return {"waveform": stitched, "sample_rate": sample_rate}, stats
234
+
235
+
236
+ def _manifest_text(
237
+ *,
238
+ source_frames: int,
239
+ fps: float,
240
+ window_specs: list[dict[str, object]],
241
+ planned_window_count: int | None = None,
242
+ truncated_by_max_windows: bool = False,
243
+ audio_stats: list[dict[str, object]],
244
+ window_audio_paths: list[str],
245
+ warnings: list[str],
246
+ ) -> str:
247
+ payload = {
248
+ "version": "ltx_foley_loop",
249
+ "source_frames": int(source_frames),
250
+ "frame_rate": float(fps),
251
+ "source_duration": float(source_frames / fps),
252
+ "window_count": len(window_specs),
253
+ "planned_window_count": int(planned_window_count if planned_window_count is not None else len(window_specs)),
254
+ "truncated_by_max_windows": bool(truncated_by_max_windows),
255
+ "window_specs": window_specs,
256
+ "window_audio_stats": audio_stats,
257
+ "window_audio_paths": window_audio_paths,
258
+ "warnings": warnings,
259
+ }
260
+ return json.dumps(payload, indent=2)
261
+
262
+
263
+ class LTXFoleyForLoopOpen:
264
+ """Minimal execution-inversion for-loop.
265
+
266
+ Loop mechanics are adapted from akatz-ai/ComfyUI-Execution-Inversion
267
+ under the MIT license, Copyright (c) 2025 akatz-ai. This node deliberately
268
+ only exposes the sockets needed by the Foley workflow.
269
+
270
+ This node is a pure pass-through: the visible graph body is the first loop
271
+ iteration. Recursion happens in _LTXFoleyLoopIterator, which clones the
272
+ subgraph between this node and the accumulator and seeds the clone of this
273
+ node with plain values via remaining/initial_value0.
274
+ """
275
+
276
+ CATEGORY = "LTX/Foley/Loop"
277
+ RETURN_TYPES = (_SmartType("FLOW_CONTROL"), "INT", "FOLEY_AUDIO_ACCUM")
278
+ RETURN_NAMES = ("flow_control", "remaining", "audio_accumulation")
279
+ FUNCTION = "open"
280
+
281
+ @classmethod
282
+ def INPUT_TYPES(cls) -> dict[str, object]:
283
+ return {
284
+ "required": {
285
+ "remaining": ("INT", {"default": 1, "min": 0, "max": 100000, "step": 1}),
286
+ },
287
+ "optional": {"audio_accumulation": ("FOLEY_AUDIO_ACCUM",)},
288
+ "hidden": {"initial_value0": (ANY_TYPE,), "unique_id": "UNIQUE_ID"},
289
+ }
290
+
291
+ def open(self, remaining: int, **kwargs):
292
+ seeded = "initial_value0" in kwargs
293
+ if seeded:
294
+ remaining = kwargs["initial_value0"]
295
+ remaining = int(remaining)
296
+ print(
297
+ "[LTXFoley][trace] loop open "
298
+ f"node={kwargs.get('unique_id')} remaining={remaining} seeded={seeded} "
299
+ f"accum_records={_accum_record_count(kwargs.get('audio_accumulation'))}",
300
+ flush=True,
301
+ )
302
+ return ("stub", remaining, kwargs.get("audio_accumulation"))
303
+
304
+
305
+ class LTXFoleyForLoopClose:
306
+ """Closes the sliding-window loop.
307
+
308
+ The loop-carried audio accumulation is received as a materialized value
309
+ (not a rawLink) and handed to the recursion as an embedded constant, and
310
+ the countdown is derived from the accumulation itself. The ephemeral
311
+ iterator node this expands must never receive graph links as inputs:
312
+ links attached to freshly expanded nodes bypass the per-consumer
313
+ execution cache, so under ComfyUI's default RAM-pressure caching an
314
+ evicted upstream output re-executes the entire visible sampler chain
315
+ (observed as a duplicated first window).
316
+ """
317
+
318
+ CATEGORY = "LTX/Foley/Loop"
319
+ RETURN_TYPES = ("FOLEY_AUDIO_ACCUM",)
320
+ RETURN_NAMES = ("audio_accumulation",)
321
+ FUNCTION = "close"
322
+
323
+ @classmethod
324
+ def INPUT_TYPES(cls) -> dict[str, object]:
325
+ return {
326
+ "required": {"flow_control": (_SmartType("FLOW_CONTROL"), {"rawLink": True})},
327
+ "optional": {"audio_accumulation": ("FOLEY_AUDIO_ACCUM",)},
328
+ "hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"},
329
+ }
330
+
331
+ def close(self, flow_control, audio_accumulation=None, dynprompt=None, unique_id=None):
332
+ from comfy_execution.graph_utils import GraphBuilder, is_link
333
+
334
+ open_node = flow_control[0]
335
+ next_remaining = _loop_next_remaining(audio_accumulation)
336
+ if next_remaining <= 0:
337
+ print(
338
+ "[LTXFoley][trace] loop finished "
339
+ f"node={unique_id} accum_records={_accum_record_count(audio_accumulation)}",
340
+ flush=True,
341
+ )
342
+ return (audio_accumulation,)
343
+
344
+ anchor = dynprompt.get_node(unique_id)["inputs"].get("audio_accumulation")
345
+ if not is_link(anchor):
346
+ raise ValueError("LTXFoleyForLoopClose expects audio_accumulation to be linked from the loop body")
347
+ print(
348
+ "[LTXFoley][trace] loop close expanding "
349
+ f"node={unique_id} open_node={open_node} anchor={anchor[0]} next_remaining={next_remaining}",
350
+ flush=True,
351
+ )
352
+ graph = GraphBuilder()
353
+ iterator = graph.node(
354
+ "_LTXFoleyLoopIterator",
355
+ audio_accumulation=audio_accumulation,
356
+ open_node=open_node,
357
+ anchor_node=anchor[0],
358
+ anchor_output=anchor[1],
359
+ )
360
+ return {"result": (iterator.out(0),), "expand": graph.finalize()}
361
+
362
+
363
+ class _LTXFoleyLoopIterator:
364
+ """Runs one recursion step of the sliding-window loop.
365
+
366
+ All inputs are constants (values and node-id strings). The loop body is
367
+ discovered by walking dynprompt links upward from the anchor (the
368
+ accumulator feeding the loop close), then cloning every node between the
369
+ loop open and the anchor. The clone's open node is seeded with plain
370
+ values, and the next iterator receives the cloned anchor's output — the
371
+ only link it ever holds, and one that points inside its own expansion.
372
+ """
373
+
374
+ CATEGORY = "LTX/Foley/Internal"
375
+ RETURN_TYPES = ("FOLEY_AUDIO_ACCUM",)
376
+ RETURN_NAMES = ("audio_accumulation",)
377
+ FUNCTION = "iterate"
378
+
379
+ @classmethod
380
+ def INPUT_TYPES(cls) -> dict[str, object]:
381
+ return {
382
+ "required": {
383
+ "audio_accumulation": ("FOLEY_AUDIO_ACCUM",),
384
+ "open_node": (ANY_TYPE,),
385
+ "anchor_node": (ANY_TYPE,),
386
+ "anchor_output": ("INT", {"default": 0}),
387
+ },
388
+ "hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"},
389
+ }
390
+
391
+ def _explore_dependencies(self, node_id, dynprompt, upstream):
392
+ from comfy_execution.graph_utils import is_link
393
+
394
+ node_info = dynprompt.get_node(node_id)
395
+ if "inputs" not in node_info:
396
+ return
397
+ for value in node_info["inputs"].values():
398
+ if is_link(value):
399
+ parent_id = value[0]
400
+ if parent_id not in upstream:
401
+ upstream[parent_id] = []
402
+ self._explore_dependencies(parent_id, dynprompt, upstream)
403
+ upstream[parent_id].append(node_id)
404
+
405
+ def _collect_contained(self, node_id, upstream, contained):
406
+ if node_id not in upstream:
407
+ return
408
+ for child_id in upstream[node_id]:
409
+ if child_id not in contained:
410
+ contained[child_id] = True
411
+ self._collect_contained(child_id, upstream, contained)
412
+
413
+ def iterate(self, audio_accumulation, open_node, anchor_node, anchor_output, dynprompt=None, unique_id=None):
414
+ next_remaining = _loop_next_remaining(audio_accumulation)
415
+ if next_remaining <= 0:
416
+ print(
417
+ "[LTXFoley][trace] loop finished "
418
+ f"node={unique_id} accum_records={_accum_record_count(audio_accumulation)}",
419
+ flush=True,
420
+ )
421
+ return (audio_accumulation,)
422
+
423
+ from comfy_execution.graph_utils import GraphBuilder, is_link
424
+
425
+ upstream = {}
426
+ self._explore_dependencies(anchor_node, dynprompt, upstream)
427
+ contained = {}
428
+ self._collect_contained(open_node, upstream, contained)
429
+ contained[open_node] = True
430
+ contained[anchor_node] = True
431
+
432
+ graph = GraphBuilder()
433
+ for node_id in contained:
434
+ original_node = dynprompt.get_node(node_id)
435
+ node = graph.node(original_node["class_type"], node_id)
436
+ node.set_override_display_id(node_id)
437
+ for node_id in contained:
438
+ original_node = dynprompt.get_node(node_id)
439
+ node = graph.lookup_node(node_id)
440
+ for key, value in original_node["inputs"].items():
441
+ if is_link(value) and value[0] in contained:
442
+ parent = graph.lookup_node(value[0])
443
+ node.set_input(key, parent.out(value[1]))
444
+ else:
445
+ node.set_input(key, value)
446
+
447
+ new_open = graph.lookup_node(open_node)
448
+ # Replace the remaining link with a plain value as well, so the clone
449
+ # does not pull the (possibly evicted) window-plan chain back onto the
450
+ # execution list.
451
+ new_open.set_input("remaining", next_remaining)
452
+ new_open.set_input("initial_value0", next_remaining)
453
+ new_open.set_input("audio_accumulation", audio_accumulation)
454
+ new_anchor = graph.lookup_node(anchor_node)
455
+ recurse = graph.node(
456
+ "_LTXFoleyLoopIterator",
457
+ "Recurse",
458
+ audio_accumulation=new_anchor.out(int(anchor_output)),
459
+ open_node=new_open.id,
460
+ anchor_node=new_anchor.id,
461
+ anchor_output=int(anchor_output),
462
+ )
463
+ recurse.set_override_display_id(unique_id)
464
+ print(
465
+ "[LTXFoley][trace] loop recursing "
466
+ f"node={unique_id} next_remaining={next_remaining} "
467
+ f"accum_records={_accum_record_count(audio_accumulation)} "
468
+ f"cloned_nodes={sorted(contained)}",
469
+ flush=True,
470
+ )
471
+ return {"result": (recurse.out(0),), "expand": graph.finalize()}
472
+
473
+
474
  class LTXFoleyVideoToAudioLatent:
475
  CATEGORY = "LTX/Foley"
476
  RETURN_TYPES = ("CONDITIONING", "CONDITIONING", "LATENT", "IMAGE", "FLOAT", "INT")
 
512
  if images.shape[0] == 0:
513
  raise ValueError("No video frames were provided")
514
 
515
+ source = _slice_or_pad_frames(images, 0, frames)
 
 
 
516
 
517
  resized = _common_upscale(source, width, height).clamp(0.0, 1.0)
518
  video_latent = video_vae.encode(resized[:, :, :, :3])
 
577
  return (trimmed, float(frame_rate))
578
 
579
 
580
+ class LTXFoleyWindowPlan:
581
+ CATEGORY = "LTX/Foley"
582
+ RETURN_TYPES = ("FOLEY_WINDOW_PLAN", "INT", "STRING")
583
+ RETURN_NAMES = ("window_plan", "window_count", "manifest")
584
+ FUNCTION = "plan"
585
+
586
+ @classmethod
587
+ def INPUT_TYPES(cls) -> dict[str, object]:
588
+ return {
589
+ "required": {
590
+ "images": ("IMAGE",),
591
+ "frame_rate": ("FLOAT", {"default": 25.0, "min": 1.0, "max": 120.0, "step": 0.01}),
592
+ "window_frames": ("INT", {"default": 89, "min": 9, "max": 257, "step": 8}),
593
+ "overlap_seconds": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 10.0, "step": 0.1}),
594
+ "max_windows": ("INT", {"default": 16, "min": 1, "max": 256, "step": 1}),
595
+ }
596
+ }
597
+
598
+ def plan(self, images, frame_rate: float, window_frames: int, overlap_seconds: float, max_windows: int):
599
+ fps = float(frame_rate)
600
+ source_frames = int(images.shape[0])
601
+ all_specs = _window_specs(source_frames, fps, int(window_frames), float(overlap_seconds))
602
+ max_windows = int(max_windows)
603
+ specs = all_specs[:max_windows]
604
+ truncated = len(specs) < len(all_specs)
605
+ if not specs:
606
+ raise ValueError("max_windows must allow at least one diagnostic window")
607
+ plan = {
608
+ "version": "ltx_foley_loop_plan",
609
+ "source_frames": source_frames,
610
+ "frame_rate": fps,
611
+ "source_duration": source_frames / fps,
612
+ "window_frames": int(window_frames),
613
+ "overlap_seconds": float(overlap_seconds),
614
+ "planned_window_count": len(all_specs),
615
+ "truncated_by_max_windows": truncated,
616
+ "window_specs": specs,
617
+ }
618
+ if truncated:
619
+ print(
620
+ "[LTXFoley] diagnostic truncation: "
621
+ f"planned {len(all_specs)} windows, running first {len(specs)} because max_windows={max_windows}",
622
+ flush=True,
623
+ )
624
+ else:
625
+ print(f"[LTXFoley] planned {len(specs)} windows", flush=True)
626
+ return (plan, len(specs), json.dumps(plan, indent=2))
627
+
628
+
629
+ class LTXFoleyWindowSelect:
630
+ CATEGORY = "LTX/Foley"
631
+ RETURN_TYPES = ("IMAGE", "FOLEY_WINDOW")
632
+ RETURN_NAMES = ("window_images", "window_info")
633
+ FUNCTION = "select"
634
+
635
+ @classmethod
636
+ def INPUT_TYPES(cls) -> dict[str, object]:
637
+ return {
638
+ "required": {
639
+ "images": ("IMAGE",),
640
+ "window_plan": ("FOLEY_WINDOW_PLAN",),
641
+ "remaining": ("INT", {"default": 1, "min": 1, "max": 100000, "step": 1}),
642
+ },
643
+ "hidden": {"unique_id": "UNIQUE_ID"},
644
+ }
645
+
646
+ def select(self, images, window_plan: dict[str, object], remaining: int, unique_id=None):
647
+ specs = list(window_plan["window_specs"])
648
+ loop_index = len(specs) - int(remaining)
649
+ if loop_index < 0 or loop_index >= len(specs):
650
+ raise ValueError(f"Loop remaining={remaining} is outside the planned {len(specs)} windows")
651
+ spec = dict(specs[loop_index])
652
+ frames = int(window_plan.get("window_frames", spec["frames"]))
653
+ window_images = _slice_or_pad_frames(images, int(spec["start_frame"]), frames)
654
+ window_info = {
655
+ "spec": spec,
656
+ "window_count": len(specs),
657
+ "source_frames": int(window_plan["source_frames"]),
658
+ "frame_rate": float(window_plan["frame_rate"]),
659
+ "source_duration": float(window_plan["source_duration"]),
660
+ "overlap_seconds": float(window_plan["overlap_seconds"]),
661
+ "planned_window_count": int(window_plan.get("planned_window_count", len(specs))),
662
+ "truncated_by_max_windows": bool(window_plan.get("truncated_by_max_windows", False)),
663
+ }
664
+ print(
665
+ "[LTXFoley] selecting window "
666
+ f"{spec['index']}/{len(specs)} start_frame={spec['start_frame']} frames={frames} "
667
+ f"node={unique_id} remaining={remaining}",
668
+ flush=True,
669
+ )
670
+ return (window_images, window_info)
671
+
672
+
673
+ class LTXFoleyWindowAudioSave:
674
+ CATEGORY = "LTX/Foley"
675
+ RETURN_TYPES = ("AUDIO", "FOLEY_WINDOW_RECORD")
676
+ RETURN_NAMES = ("audio", "window_record")
677
+ FUNCTION = "save"
678
+
679
+ @classmethod
680
+ def INPUT_TYPES(cls) -> dict[str, object]:
681
+ return {
682
+ "required": {
683
+ "audio": ("AUDIO",),
684
+ "window_info": ("FOLEY_WINDOW",),
685
+ "save_audio": ("BOOLEAN", {"default": True}),
686
+ "filename_prefix": ("STRING", {"default": "ltx_foley_window"}),
687
+ }
688
+ }
689
+
690
+ def save(self, audio, window_info: dict[str, object], save_audio: bool, filename_prefix: str):
691
+ spec = dict(window_info["spec"])
692
+ path = ""
693
+ if save_audio:
694
+ path = _write_audio_window(audio, prefix=filename_prefix, window_index=int(spec["index"]))
695
+ stats = _audio_stats(_audio_waveform(audio))
696
+ print(
697
+ "[LTXFoley] window "
698
+ f"{spec['index']} audio samples={stats['samples']} rms={stats['rms']:.6f} "
699
+ f"peak={stats['peak']:.6f} path={path or '<not saved>'}",
700
+ flush=True,
701
+ )
702
+ return (audio, {"audio": audio, "window_info": window_info, "path": path})
703
+
704
+
705
+ class LTXFoleyAudioAccumulator:
706
+ CATEGORY = "LTX/Foley/Loop"
707
+ RETURN_TYPES = ("FOLEY_AUDIO_ACCUM",)
708
+ RETURN_NAMES = ("accumulation",)
709
+ FUNCTION = "accumulate"
710
+
711
+ @classmethod
712
+ def INPUT_TYPES(cls) -> dict[str, object]:
713
+ return {
714
+ "required": {"window_record": ("FOLEY_WINDOW_RECORD",)},
715
+ "optional": {"accumulation": ("FOLEY_AUDIO_ACCUM",)},
716
+ "hidden": {"unique_id": "UNIQUE_ID"},
717
+ }
718
+
719
+ def accumulate(self, window_record: dict[str, object], accumulation: dict[str, object] | None = None, unique_id=None):
720
+ records = [] if accumulation is None else list(accumulation.get("records", []))
721
+ records.append(window_record)
722
+ print(
723
+ "[LTXFoley][trace] accumulate "
724
+ f"node={unique_id} window={window_record['window_info']['spec']['index']} "
725
+ f"total_records={len(records)}",
726
+ flush=True,
727
+ )
728
+ return ({"records": records},)
729
+
730
+
731
+ class LTXFoleyAudioStitch:
732
+ CATEGORY = "LTX/Foley"
733
+ RETURN_TYPES = ("AUDIO", "STRING")
734
+ RETURN_NAMES = ("audio", "manifest")
735
+ FUNCTION = "stitch"
736
+
737
+ @classmethod
738
+ def INPUT_TYPES(cls) -> dict[str, object]:
739
+ return {
740
+ "required": {
741
+ "accumulation": ("FOLEY_AUDIO_ACCUM",),
742
+ "window_plan": ("FOLEY_WINDOW_PLAN",),
743
+ }
744
+ }
745
+
746
+ def stitch(self, accumulation: dict[str, object], window_plan: dict[str, object]):
747
+ records = list(accumulation.get("records", []))
748
+ if not records:
749
+ raise ValueError("No generated window audio records were provided")
750
+
751
+ records.sort(key=lambda item: int(item["window_info"]["spec"]["index"]))
752
+ audio_windows = [item["audio"] for item in records]
753
+ specs = [dict(item["window_info"]["spec"]) for item in records]
754
+ stitched_audio, audio_stats = _stitch_audio_windows(
755
+ audio_windows,
756
+ specs,
757
+ output_duration=float(window_plan["source_duration"]),
758
+ overlap_seconds=float(window_plan["overlap_seconds"]),
759
+ )
760
+ manifest = _manifest_text(
761
+ source_frames=int(window_plan["source_frames"]),
762
+ fps=float(window_plan["frame_rate"]),
763
+ window_specs=specs,
764
+ planned_window_count=int(window_plan.get("planned_window_count", len(specs))),
765
+ truncated_by_max_windows=bool(window_plan.get("truncated_by_max_windows", False)),
766
+ audio_stats=audio_stats,
767
+ window_audio_paths=[str(item.get("path", "")) for item in records],
768
+ warnings=(
769
+ [f"Diagnostic run truncated to {len(specs)} of {int(window_plan.get('planned_window_count', len(specs)))} planned windows"]
770
+ if bool(window_plan.get("truncated_by_max_windows", False))
771
+ else []
772
+ ),
773
+ )
774
+ print(
775
+ "[LTXFoley] stitched "
776
+ f"{len(specs)} window(s); truncated={bool(window_plan.get('truncated_by_max_windows', False))}",
777
+ flush=True,
778
+ )
779
+ return (stitched_audio, manifest)
780
+
781
+
782
  class LTXFoleyAudioVAEDecode:
783
  CATEGORY = "LTX/Foley"
784
  RETURN_TYPES = ("AUDIO",)
 
799
  if audio_latent.is_nested:
800
  audio_latent = audio_latent.unbind()[-1]
801
 
802
+ output_device = audio_latent.device
803
+ audio = audio_vae.decode(audio_latent).to(output_device)
804
  if audio.ndim == 2:
805
  audio = audio.unsqueeze(1)
806
  elif audio.ndim != 3:
 
818
 
819
 
820
  NODE_CLASS_MAPPINGS = {
821
+ "_LTXFoleyLoopIterator": _LTXFoleyLoopIterator,
822
+ "LTXFoleyForLoopOpen": LTXFoleyForLoopOpen,
823
+ "LTXFoleyForLoopClose": LTXFoleyForLoopClose,
824
  "LTXFoleyVideoToAudioLatent": LTXFoleyVideoToAudioLatent,
825
  "LTXFoleyTrimImages": LTXFoleyTrimImages,
826
+ "LTXFoleyWindowPlan": LTXFoleyWindowPlan,
827
+ "LTXFoleyWindowSelect": LTXFoleyWindowSelect,
828
+ "LTXFoleyWindowAudioSave": LTXFoleyWindowAudioSave,
829
+ "LTXFoleyAudioAccumulator": LTXFoleyAudioAccumulator,
830
+ "LTXFoleyAudioStitch": LTXFoleyAudioStitch,
831
  "LTXFoleyAudioVAEDecode": LTXFoleyAudioVAEDecode,
832
  }
833
  NODE_DISPLAY_NAME_MAPPINGS = {
834
+ "LTXFoleyForLoopOpen": "LTX Foley For Loop Open",
835
+ "LTXFoleyForLoopClose": "LTX Foley For Loop Close",
836
  "LTXFoleyVideoToAudioLatent": "LTX Foley Video To Audio Latent",
837
  "LTXFoleyTrimImages": "LTX Foley Trim Images",
838
+ "LTXFoleyWindowPlan": "LTX Foley Window Plan",
839
+ "LTXFoleyWindowSelect": "LTX Foley Window Select",
840
+ "LTXFoleyWindowAudioSave": "LTX Foley Window Audio Save",
841
+ "LTXFoleyAudioAccumulator": "LTX Foley Audio Accumulator",
842
+ "LTXFoleyAudioStitch": "LTX Foley Audio Stitch",
843
  "LTXFoleyAudioVAEDecode": "LTX Foley Audio VAE Decode",
844
  }
setup_runpod_ltx_foley.sh CHANGED
@@ -1,9 +1,12 @@
1
  #!/usr/bin/env bash
2
  set -Eeuo pipefail
3
 
4
- LTXVIDEO_REPO="${LTXVIDEO_REPO:-https://github.com/Lightricks/ComfyUI-LTXVideo.git}"
5
- WORKFLOW_REPO="${WORKFLOW_REPO:-FuzzPuppy/LTX-2.3-Foley-Workflow}"
6
- WORKFLOW_BASE_URL="${WORKFLOW_BASE_URL:-https://huggingface.co/${WORKFLOW_REPO}/resolve/main}"
 
 
 
7
 
8
  log() {
9
  printf '\n[%s] %s\n' "$(date +%H:%M:%S)" "$*"
@@ -14,12 +17,18 @@ die() {
14
  exit 1
15
  }
16
 
 
 
 
 
 
17
  download_file() {
18
  local url="$1"
19
  local target="$2"
20
- local token="${HF_TOKEN:-${HUGGINGFACE_TOKEN:-}}"
21
  local curl_args=(-fL --retry 5 --retry-delay 2)
22
 
 
23
  mkdir -p "$(dirname "$target")"
24
  if [[ -n "$token" ]]; then
25
  curl_args+=(-H "Authorization: Bearer $token")
@@ -32,10 +41,11 @@ download_model_file() {
32
  local target_path="$2"
33
  local expected_sha256="$3"
34
  local partial_path="${target_path}.part"
35
- local token="${HF_TOKEN:-${HUGGINGFACE_TOKEN:-}}"
36
  local curl_args=(-fL --retry 5 --retry-delay 2 -C -)
37
  local attempt
38
 
 
39
  if [[ -s "$target_path" ]]; then
40
  if printf '%s %s\n' "$expected_sha256" "$target_path" | sha256sum -c - >/dev/null 2>&1; then
41
  printf 'verified: %s\n' "$target_path"
@@ -69,23 +79,12 @@ download_model_file() {
69
  die "Downloaded file failed SHA-256 verification: $target_path"
70
  }
71
 
72
- build_comfyui_args() {
73
- COMFYUI_ARGS=(--listen 0.0.0.0 --port 8188 --enable-cors-header)
74
-
75
- if [[ -s "$ARGS_FILE" ]]; then
76
- local line
77
- local line_args
78
- while IFS= read -r line; do
79
- [[ -z "$line" || "$line" == \#* ]] && continue
80
- read -r -a line_args <<< "$line"
81
- COMFYUI_ARGS+=("${line_args[@]}")
82
- done < "$ARGS_FILE"
83
- fi
84
- }
85
-
86
  restart_comfyui() {
87
  log "Restarting ComfyUI"
88
- build_comfyui_args
 
 
 
89
 
90
  local pid
91
  while IFS= read -r pid; do
@@ -107,7 +106,7 @@ restart_comfyui() {
107
 
108
  (
109
  cd "$COMFYUI_DIR"
110
- nohup "$PYTHON_BIN" main.py "${COMFYUI_ARGS[@]}" > "$LOG_FILE" 2>&1 &
111
  printf '%s\n' "$!" > "$PID_FILE"
112
  )
113
 
@@ -127,13 +126,113 @@ restart_comfyui() {
127
  die "ComfyUI process started, but port 8188 did not become ready. Check $LOG_FILE"
128
  }
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  COMFYUI_DIR="${COMFYUI_DIR:-/workspace/runpod-slim/ComfyUI}"
131
  [[ -f "$COMFYUI_DIR/main.py" ]] || die "Could not find ComfyUI at $COMFYUI_DIR. Use RunPod's official ComfyUI - CUDA 12.8 template or set COMFYUI_DIR=/path/to/ComfyUI."
132
  CUSTOM_NODES_DIR="$COMFYUI_DIR/custom_nodes"
133
  MODELS_DIR="$COMFYUI_DIR/models"
134
  INPUT_DIR="$COMFYUI_DIR/input"
135
  VENV_DIR="$COMFYUI_DIR/.venv-cu128"
136
- ARGS_FILE="/workspace/runpod-slim/comfyui_args.txt"
137
  LOG_FILE="/workspace/runpod-slim/comfyui-restart.log"
138
  PID_FILE="/workspace/runpod-slim/comfyui-restart.pid"
139
  PYTHON_BIN="$VENV_DIR/bin/python"
@@ -148,6 +247,8 @@ fi
148
 
149
  log "Using ComfyUI at $COMFYUI_DIR"
150
  log "Using Python at $PYTHON_BIN"
 
 
151
 
152
  mkdir -p "$CUSTOM_NODES_DIR" "$MODELS_DIR/checkpoints" "$MODELS_DIR/loras" "$MODELS_DIR/text_encoders" "$INPUT_DIR"
153
 
@@ -156,36 +257,32 @@ command -v git >/dev/null 2>&1 || die "git is required. RunPod's official ComfyU
156
  command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg is required. RunPod's official ComfyUI - CUDA 12.8 template includes it."
157
  command -v sha256sum >/dev/null 2>&1 || die "sha256sum is required. RunPod's official ComfyUI - CUDA 12.8 template includes it."
158
 
159
- if [[ "${UPDATE_COMFYUI:-0}" == "1" ]]; then
160
- log "Updating ComfyUI"
161
- git -C "$COMFYUI_DIR" pull --ff-only || log "ComfyUI git pull was skipped or failed; continuing with the installed version."
162
- else
163
- log "Keeping the RunPod template's pinned ComfyUI install"
164
- fi
165
 
166
- log "Installing official LTXVideo custom nodes"
167
- if [[ -d "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/.git" ]]; then
168
- git -C "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo" pull --ff-only || log "LTXVideo git pull was skipped or failed; continuing with the installed version."
169
- else
170
- rm -rf "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo"
171
- git clone --depth 1 "$LTXVIDEO_REPO" "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo"
172
- fi
173
 
174
  if [[ -f "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/requirements.txt" ]]; then
175
- "$PYTHON_BIN" -m pip install -r "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/requirements.txt"
176
  fi
177
  if [[ -f "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/install.py" ]]; then
178
  "$PYTHON_BIN" "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/install.py" || log "LTXVideo install.py returned non-zero; continuing after requirements install."
179
  fi
 
180
 
181
  log "Installing LTX Foley helper node and workflow from $WORKFLOW_REPO"
182
  NODE_DEST="$CUSTOM_NODES_DIR/ltx_foley_v2a"
183
  WORKFLOW_DEST="$COMFYUI_DIR/user/default/workflows"
184
  rm -rf "$NODE_DEST"
185
  mkdir -p "$NODE_DEST" "$WORKFLOW_DEST"
 
 
 
 
186
  download_file "$WORKFLOW_BASE_URL/ltx_foley_v2a/__init__.py" "$NODE_DEST/__init__.py"
187
  download_file "$WORKFLOW_BASE_URL/ltx_foley_v2a/nodes.py" "$NODE_DEST/nodes.py"
188
  download_file "$WORKFLOW_BASE_URL/ltx_23_foley_v2a.json" "$WORKFLOW_DEST/ltx_23_foley_v2a.json"
 
189
  download_file "$WORKFLOW_BASE_URL/tennis-no-sound.mp4" "$INPUT_DIR/input.mp4"
190
 
191
  log "Downloading LTX 2.3 Foley model files"
@@ -198,7 +295,7 @@ download_model_file \
198
  "$MODELS_DIR/text_encoders/gemma_3_12B_it_fp8_scaled.safetensors" \
199
  "60216ce97c01c3a8753c2dfd0a89fc76e16fbe446d1a32ef8f1b528ac8bae466"
200
  download_model_file \
201
- "https://huggingface.co/FuzzPuppy/LTX-2.3-Foley/resolve/main/ltx-2.3-foley-400-steps.safetensors" \
202
  "$MODELS_DIR/loras/ltx-2.3-foley-400-steps.safetensors" \
203
  "12b37ffff8444ac1348a8c9fb24843a6bd70e77cf4202ac39b1d056b81fa41c9"
204
 
@@ -209,9 +306,14 @@ cat <<EOF
209
 
210
  Next steps:
211
  1. Open the RunPod HTTP service on port 8188.
212
- 2. Load workflow: user/default/workflows/ltx_23_foley_v2a.json
 
213
  3. Queue the workflow. The tennis test video is already installed as input.mp4 and the prompt is prefilled.
214
 
 
 
 
 
215
  If model download fails with 401/403, accept the model licenses on Hugging Face and rerun with:
216
  export HF_TOKEN=hf_...
217
  bash $0
 
1
  #!/usr/bin/env bash
2
  set -Eeuo pipefail
3
 
4
+ LTXVIDEO_REPO="https://github.com/Lightricks/ComfyUI-LTXVideo.git"
5
+ LTXVIDEO_REVISION="4f45fd6c222eb06eb3e46605da62e7c889e4be5c"
6
+ COMFYUI_CORE_REF="${COMFYUI_CORE_REF:-v0.27.0}"
7
+ WORKFLOW_REPO="FuzzPuppy/LTX-2.3-Foley-Workflow"
8
+ WORKFLOW_REVISION="${WORKFLOW_REVISION:-main}"
9
+ WORKFLOW_BASE_URL="https://huggingface.co/${WORKFLOW_REPO}/resolve/${WORKFLOW_REVISION}"
10
 
11
  log() {
12
  printf '\n[%s] %s\n' "$(date +%H:%M:%S)" "$*"
 
17
  exit 1
18
  }
19
 
20
+ hf_token() {
21
+ local token="${HF_TOKEN:-${HUGGINGFACE_TOKEN:-}}"
22
+ printf '%s' "$token" | tr -d '[:space:]'
23
+ }
24
+
25
  download_file() {
26
  local url="$1"
27
  local target="$2"
28
+ local token
29
  local curl_args=(-fL --retry 5 --retry-delay 2)
30
 
31
+ token="$(hf_token)"
32
  mkdir -p "$(dirname "$target")"
33
  if [[ -n "$token" ]]; then
34
  curl_args+=(-H "Authorization: Bearer $token")
 
41
  local target_path="$2"
42
  local expected_sha256="$3"
43
  local partial_path="${target_path}.part"
44
+ local token
45
  local curl_args=(-fL --retry 5 --retry-delay 2 -C -)
46
  local attempt
47
 
48
+ token="$(hf_token)"
49
  if [[ -s "$target_path" ]]; then
50
  if printf '%s %s\n' "$expected_sha256" "$target_path" | sha256sum -c - >/dev/null 2>&1; then
51
  printf 'verified: %s\n' "$target_path"
 
79
  die "Downloaded file failed SHA-256 verification: $target_path"
80
  }
81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  restart_comfyui() {
83
  log "Restarting ComfyUI"
84
+ # --cache-classic: the default RAM-pressure cache evicts node outputs
85
+ # mid-prompt while the large AV models load, which forces the sliding-window
86
+ # loop expansion to re-execute the entire upstream graph on every iteration.
87
+ local comfyui_args=(--listen 0.0.0.0 --port 8188 --enable-cors-header --cache-classic)
88
 
89
  local pid
90
  while IFS= read -r pid; do
 
106
 
107
  (
108
  cd "$COMFYUI_DIR"
109
+ nohup "$PYTHON_BIN" main.py "${comfyui_args[@]}" > "$LOG_FILE" 2>&1 &
110
  printf '%s\n' "$!" > "$PID_FILE"
111
  )
112
 
 
126
  die "ComfyUI process started, but port 8188 did not become ready. Check $LOG_FILE"
127
  }
128
 
129
+ patch_ltxvideo_kornia_import() {
130
+ local target="$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/pyramid_blending.py"
131
+ [[ -f "$target" ]] || return
132
+
133
+ "$PYTHON_BIN" - "$target" <<'PY'
134
+ from pathlib import Path
135
+ import sys
136
+
137
+ path = Path(sys.argv[1])
138
+ text = path.read_text(encoding="utf-8")
139
+ old = """from kornia.geometry.transform.pyramid import (
140
+ PyrUp,
141
+ build_laplacian_pyramid,
142
+ build_pyramid,
143
+ find_next_powerof_two,
144
+ is_powerof_two,
145
+ pad,
146
+ )
147
+ """
148
+ new = """from kornia.geometry.transform.pyramid import (
149
+ PyrUp,
150
+ build_laplacian_pyramid,
151
+ build_pyramid,
152
+ find_next_powerof_two,
153
+ is_powerof_two,
154
+ )
155
+ from torch.nn.functional import pad
156
+ """
157
+ if old in text:
158
+ path.write_text(text.replace(old, new), encoding="utf-8")
159
+ print(f"patched kornia pad import: {path}")
160
+ elif "from torch.nn.functional import pad" in text:
161
+ print(f"kornia pad import already patched: {path}")
162
+ else:
163
+ print(f"warning: expected kornia import block not found in {path}", file=sys.stderr)
164
+ PY
165
+ }
166
+
167
+ ensure_pip() {
168
+ if "$PYTHON_BIN" -m pip --version >/dev/null 2>&1; then
169
+ return
170
+ fi
171
+
172
+ log "Bootstrapping pip for $PYTHON_BIN"
173
+ "$PYTHON_BIN" -m ensurepip --upgrade || true
174
+ if ! "$PYTHON_BIN" -m pip --version >/dev/null 2>&1; then
175
+ die "Could not run pip with $PYTHON_BIN. The RunPod Python environment is missing pip/ensurepip."
176
+ fi
177
+ }
178
+
179
+ pip_install() {
180
+ ensure_pip
181
+ "$PYTHON_BIN" -m pip install "$@"
182
+ }
183
+
184
+ install_comfyui_core_requirements() {
185
+ local requirements="$COMFYUI_DIR/requirements.txt"
186
+
187
+ [[ -f "$requirements" ]] || die "Could not find ComfyUI requirements file: $requirements"
188
+
189
+ log "Installing ComfyUI core requirements for $COMFYUI_CORE_REF"
190
+ pip_install --upgrade pip setuptools wheel
191
+ pip_install --upgrade -r "$requirements"
192
+ }
193
+
194
+ install_comfyui_core() {
195
+ log "Installing ComfyUI core at $COMFYUI_CORE_REF"
196
+ if [[ "$COMFYUI_CORE_REF" == v* ]]; then
197
+ git -C "$COMFYUI_DIR" fetch --force origin "refs/tags/$COMFYUI_CORE_REF:refs/tags/$COMFYUI_CORE_REF"
198
+ else
199
+ git -C "$COMFYUI_DIR" fetch --force origin "$COMFYUI_CORE_REF"
200
+ fi
201
+ if ! git -C "$COMFYUI_DIR" checkout -f "$COMFYUI_CORE_REF"; then
202
+ die "Could not check out ComfyUI core ref: $COMFYUI_CORE_REF"
203
+ fi
204
+
205
+ log "ComfyUI core resolved to $(git -C "$COMFYUI_DIR" describe --tags --always --dirty)"
206
+ }
207
+
208
+ install_ltxvideo_nodes() {
209
+ local target="$CUSTOM_NODES_DIR/ComfyUI-LTXVideo"
210
+
211
+ log "Installing official LTXVideo custom nodes at $LTXVIDEO_REVISION"
212
+ if [[ ! -d "$target/.git" ]]; then
213
+ rm -rf "$target"
214
+ git clone "$LTXVIDEO_REPO" "$target"
215
+ fi
216
+
217
+ if ! git -C "$target" fetch origin "$LTXVIDEO_REVISION"; then
218
+ git -C "$target" fetch origin
219
+ fi
220
+
221
+ if ! git -C "$target" checkout -f "$LTXVIDEO_REVISION"; then
222
+ log "Existing LTXVideo checkout could not switch revisions; re-cloning."
223
+ rm -rf "$target"
224
+ git clone "$LTXVIDEO_REPO" "$target"
225
+ git -C "$target" fetch origin "$LTXVIDEO_REVISION" || git -C "$target" fetch origin
226
+ git -C "$target" checkout -f "$LTXVIDEO_REVISION"
227
+ fi
228
+ }
229
+
230
  COMFYUI_DIR="${COMFYUI_DIR:-/workspace/runpod-slim/ComfyUI}"
231
  [[ -f "$COMFYUI_DIR/main.py" ]] || die "Could not find ComfyUI at $COMFYUI_DIR. Use RunPod's official ComfyUI - CUDA 12.8 template or set COMFYUI_DIR=/path/to/ComfyUI."
232
  CUSTOM_NODES_DIR="$COMFYUI_DIR/custom_nodes"
233
  MODELS_DIR="$COMFYUI_DIR/models"
234
  INPUT_DIR="$COMFYUI_DIR/input"
235
  VENV_DIR="$COMFYUI_DIR/.venv-cu128"
 
236
  LOG_FILE="/workspace/runpod-slim/comfyui-restart.log"
237
  PID_FILE="/workspace/runpod-slim/comfyui-restart.pid"
238
  PYTHON_BIN="$VENV_DIR/bin/python"
 
247
 
248
  log "Using ComfyUI at $COMFYUI_DIR"
249
  log "Using Python at $PYTHON_BIN"
250
+ log "Requested ComfyUI core ref: $COMFYUI_CORE_REF"
251
+ log "Requested workflow revision: $WORKFLOW_REVISION"
252
 
253
  mkdir -p "$CUSTOM_NODES_DIR" "$MODELS_DIR/checkpoints" "$MODELS_DIR/loras" "$MODELS_DIR/text_encoders" "$INPUT_DIR"
254
 
 
257
  command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg is required. RunPod's official ComfyUI - CUDA 12.8 template includes it."
258
  command -v sha256sum >/dev/null 2>&1 || die "sha256sum is required. RunPod's official ComfyUI - CUDA 12.8 template includes it."
259
 
260
+ install_comfyui_core
261
+ install_comfyui_core_requirements
 
 
 
 
262
 
263
+ install_ltxvideo_nodes
 
 
 
 
 
 
264
 
265
  if [[ -f "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/requirements.txt" ]]; then
266
+ pip_install --upgrade -r "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/requirements.txt"
267
  fi
268
  if [[ -f "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/install.py" ]]; then
269
  "$PYTHON_BIN" "$CUSTOM_NODES_DIR/ComfyUI-LTXVideo/install.py" || log "LTXVideo install.py returned non-zero; continuing after requirements install."
270
  fi
271
+ patch_ltxvideo_kornia_import
272
 
273
  log "Installing LTX Foley helper node and workflow from $WORKFLOW_REPO"
274
  NODE_DEST="$CUSTOM_NODES_DIR/ltx_foley_v2a"
275
  WORKFLOW_DEST="$COMFYUI_DIR/user/default/workflows"
276
  rm -rf "$NODE_DEST"
277
  mkdir -p "$NODE_DEST" "$WORKFLOW_DEST"
278
+ rm -f \
279
+ "$WORKFLOW_DEST/ltx_23_foley_v2a_v3_loop.json" \
280
+ "$WORKFLOW_DEST/ltx_23_foley_v2a_legacy_audio_vae.json" \
281
+ "$WORKFLOW_DEST/ltx_23_foley_v2a_v1_wav_debug.json"
282
  download_file "$WORKFLOW_BASE_URL/ltx_foley_v2a/__init__.py" "$NODE_DEST/__init__.py"
283
  download_file "$WORKFLOW_BASE_URL/ltx_foley_v2a/nodes.py" "$NODE_DEST/nodes.py"
284
  download_file "$WORKFLOW_BASE_URL/ltx_23_foley_v2a.json" "$WORKFLOW_DEST/ltx_23_foley_v2a.json"
285
+ download_file "$WORKFLOW_BASE_URL/foley-sliding-window.json" "$WORKFLOW_DEST/foley-sliding-window.json"
286
  download_file "$WORKFLOW_BASE_URL/tennis-no-sound.mp4" "$INPUT_DIR/input.mp4"
287
 
288
  log "Downloading LTX 2.3 Foley model files"
 
295
  "$MODELS_DIR/text_encoders/gemma_3_12B_it_fp8_scaled.safetensors" \
296
  "60216ce97c01c3a8753c2dfd0a89fc76e16fbe446d1a32ef8f1b528ac8bae466"
297
  download_model_file \
298
+ "https://huggingface.co/FuzzPuppy/LTX-2.3-Foley-LoRA/resolve/main/ltx-2.3-foley-400-steps.safetensors" \
299
  "$MODELS_DIR/loras/ltx-2.3-foley-400-steps.safetensors" \
300
  "12b37ffff8444ac1348a8c9fb24843a6bd70e77cf4202ac39b1d056b81fa41c9"
301
 
 
306
 
307
  Next steps:
308
  1. Open the RunPod HTTP service on port 8188.
309
+ 2. Load workflow: user/default/workflows/foley-sliding-window.json
310
+ Short-clip workflow: user/default/workflows/ltx_23_foley_v2a.json
311
  3. Queue the workflow. The tennis test video is already installed as input.mp4 and the prompt is prefilled.
312
 
313
+ This run used:
314
+ COMFYUI_CORE_REF=$COMFYUI_CORE_REF
315
+ WORKFLOW_REVISION=$WORKFLOW_REVISION
316
+
317
  If model download fails with 401/403, accept the model licenses on Hugging Face and rerun with:
318
  export HF_TOKEN=hf_...
319
  bash $0