100percentrobot/LTX-2.3-Audio-Reactive-LORA

LTX-2.3 Audio Reactive LORA V1

Still early stages and really just a proof of concept. Created to increase the responsiveness and synchronization of musical elements to changing visual elements within generated videos, AKA "Audio Reactive" content.

V1 shows a marked improvement over the base model, but further improvements are expected with more fine tuning.

Trained exclusively on custom synthetic data (which I may open source for training other multimodal models). I expect V2 to be much improved with a broader, less abstract sampling of audio reactive content and adjusted training settings.

If you want to submit some training samples I am accepting high quality well labeled data.

Recommended LORA weight :

• 1.0-2.0, recommended 1.4 or above for better motion.

Prompt template:

• A continuous audio-reactive video that transitions smoothly from [Phase 1: Initial Subject & Action] to [Phase 2: First Evolution & Motion], then warps/morphs into [Phase 3: Second Evolution & Morphing] before warping/morphing into [Phase 4: Final Chaotic/Complex State], with every [Type of Visual Motion/Deformation] perfectly synchronized to the [Musical Element 1], [Musical Element 2], and [Musical Element 3] of the [Music Genre & Vibe] track.

It's recommended to have 2-4 different 'phases' for optimal motion since this is the template the (current v1) training data uses, which I expect to improve with future versions and more diverse data. The base LTX-2.3 model still has some issues with low motion on shorter videos so 10-20 seconds produces better results. I usually find that batches of 4-8 will have some semi-useable results.

I2V was trained into the LORA but I haven't tested it extensively, and I expect motion to be less stable.

Trained with Ostris AI Toolkit Runpod template.

Trigger words

You should use continuous audio-reactive video to trigger the image generation.

You should use audio-reactive to trigger the image generation.

Download model

Download them in the Files & versions tab.