Kijai

/

LTX2.3_comfy

like 391

Any workflow to add audio and lipsync a silent video, like InfiniteTalk v2v can do but with ltx?

Should be possible with a bit of inpainting ...
Will try make an inpaint workflow soon ;-)

Just a quick test run as proof of concept, in low res single pass (so quality will be much better in a full workflow):

Are you also getting a wonky splash logo that shows up on the last several frames of every generation? it's happening to me every time and it makes it very difficult to get a clean extend flow going...

Are you also getting a wonky splash logo that shows up on the last several frames

This was an error with their first released upscale model.
Try the 1.1 version

https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-spatial-upscaler-x2-1.1.safetensors

Would it be possible to create a workflow that utilizes both Retake and your Video Extender logic to generate whole new sections in the middle of videos? Not replace or edit them, but add to them. E.g. I have a 10 second video, and I want to add 10 additional seconds to the middle (starting at the 5 second mark), then seamlessly blend it into the last 5 seconds, thereby creating a 20 second extended version.

It seems like LTXV AudioVideoMask would be well suited for this. Just not sure how to use two of them, or do it in a way that's VRAM-friendly...

Should be possible yeah. Will give it a try ;-)

Btw, is it possible to extend a video using PromptRelay for consistency on long video extension? 🤔

Btw, is it possible to extend a video using PromptRelay for consistency on long video extension? 🤔

Might be, the tricky part is the overlapping frames (around 3 seconds or so).
But if you set the first segment in prompt relay to be quite long (and for sure longer than 3s), it should work.

Been trying to find a sweet spot with that for extending videos, so hopefully a wf soon ;-)

Hey Rune, thanks for everything you do. I have a few questions regarding this specific workflow. Like anr2me said, I'm trying to feed in a reference video, extend it, but also maintain the detail as opposed to having it degrade. When I try to do multiple V2V extensions, the quality degrades over time no matter what I try. Would your suggestion about using prompt relay fix this, potentially?

Could try extract first frame of the original video and use that as a guiding late tent perhaps.
The workflow here was originally just for a short little extension, say input some 5s Wan video to make it 20 sec or so.... LTX is trained on videos up to 20 sec (with the sweet spot around 10-15 ish). So if you go beyond that, it will degrade in quality

Will take a look and see if there was some guiding latent in this one ;-) perhaps make a variant specifically for longer extensions, where it extends multiple times of 10-15 sec videos

LTX-2.3 - V2V "Extend Any Video Multi-Extend" for longer extended videos

I'm trying to feed in a reference video, extend it, but also maintain the detail as opposed to having it degrade.

Try the new "Extend Any Video - Multi Extend for Long Video" workflow, made for longer video where wf essentially does a multi extend of 10-20 sec per group/window...
(3 extend groups in the workflow, making a minute long video or so - but you can copy paste groups, and extend further than those already in the workflow).

This should work better for long videos ;-)
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Video-2-Video

Potentially it will degrade if you go too "crazy" long, but should hold up for minute or two long videos..

There is one caveat. Each group has a reference input from previous group (last few seconds of video and audio).
Things not present in that reference segment, the model will forget. (except those in the reference input image.. those are remembered all the way)

Should be possible yeah. Will give it a try ;-)

have you tried?

Would be a combo of retake and extend. Essentially you could first to the re-take and edit the middle part. Then extend (2 separate workflows, available)
But will try combine it, forgot about that one

there is an IC lora for this seamless transitions I found https://huggingface.co/siraxe/MergeGreen_IC-lora_ltx2.3

there is an IC lora for this seamless transitions I found https://huggingface.co/siraxe/MergeGreen_IC-lora_ltx2.3

yes its an interesting lora, but more for first last middle frame workflows.
Where the middle frame is a green square.

Can get some really nice transitions with that ;-)

I did test it and work, the example workflow method of concatenating the batch is terrible but it works nicely, it even complete the sentence if it's cutted on dirty

Yes I gave that Merge Green lora a test run today as well. Can get some interesting transitions with it.
Will upload a workflow, a little undecided if using the nodes in the example workflow, or simply just basic mask and batch images (that are natively supported)

Critters meet their distant cousin, Yoda

a little merge green lora test run ;-)

Yes I gave that Merge Green lora a test run today as well. Can get some interesting transitions with it.
Will upload a workflow, a little undecided if using the nodes in the example workflow, or simply just basic mask and batch images (that are natively supported)

the example nodes are terrible, I refer to the one ComfyUI-SA-Nodes-QQ (sorry missed your reply)

I refer to the one ComfyUI-SA-Nodes-QQ

yes those are perfectly ok to use, but the lora repro has a workflow already for those nodes.
My workflow is an alternative one, where its less dependencies, using most of all native comfy nodes as far as thats possible

That being said i might tweak it a bit, it was just a "for fun lora" test.