rambrogi/ltx2.3-cinematic-conductor

LTX-2.3 · Cinematic Conductor 🎬🎧

An audio-reactive LoRA for LTX-2.3 where the music directs the film.

Most audio LoRAs make objects bounce on the beat. Cinematic Conductor treats the audio as a score: it drives the cinematic treatment of a scene you provide — the build of energy, the release on the drop, the settle afterward — as camera motion, exposure, light and atmosphere. Give it one base scene and three different tracks and you get three different films.

Same ridge, driven by a riser→drop track: tension builds, then detonates on the drop.

• Track: Audio (also relevant to VFX / Creative & Fun)
• Trigger token: audio reactive
• Base model: LTX-2.3 (22B)
• Type: Audio→Video LoRA, used in image + audio (first-frame) mode
• Trained with: the official LTX Trainer
• Training data: 100% synthetic & procedurally generated — zero third-party footage, zero IP/privacy exposure

---

The idea: prompt says what, audio says when & how hard

The LoRA is built around an explicit division of labor:

Channel	Responsibility
Text prompt	The content & style — the scene, palette, mood ("a jagged mountain ridge, cold steel-blue")
Audio track	The performance — timing, intensity, the energy trajectory (build → drop → settle)
First-frame image	The anchor — the exact scene the music will conduct

The LoRA learns the mapping from a track's energy trajectory to a coherent cinematic response, and lays it on top of whatever scene you anchor. Because content comes from the prompt/anchor and dynamics come from the audio, the same scene re-scores completely differently for a riser, a swell, or a groove.

---

Showcase — one scene, three tracks, three films

Cold mountain ridge (top→bottom: riser→detonation, swell→bloom, groove→pulse):

Warm ocean sunset — proving it is not scene-specific (audio modulates wave energy, foam and light):

Video reels (each clip carries its own audio):

• REEL_master_2scenes.mp4 — the full 30s showcase (both scenes × three tracks)
• REEL_ridge_3films.mp4 — ridge, three tracks
• REEL_ocean_3films.mp4 — ocean, three tracks

---

Usage

Use with the LTX-2.3 a2vid_two_stage pipeline in image + audio mode. Anchor a first frame, hand it an audio track, and describe the scene (start the prompt with the trigger audio reactive). A short cue in the prompt matching the track's shape ("…building tension that detonates on the drop") sharpens the result.

python -m ltx_pipelines.a2vid_two_stage \
  --checkpoint-path      ltx-2.3-22b-dev.safetensors \
  --distilled-lora       ltx-2.3-22b-distilled-lora-384-1.1.safetensors 0.6 \
  --spatial-upsampler-path ltx-2.3-spatial-upscaler-x2-1.1.safetensors \
  --gemma-root           gemma-3-12b-it \
  --lora                 ltx2.3-cinematic-conductor.safetensors 1.0 \
  --image                base_scene.png 0 1.0 33 \
  --audio-path           your_track.wav \
  --prompt "audio reactive, a jagged mountain ridge silhouetted against a vast sky, cold, hard, high-contrast steel-blue. the camera, lighting and atmosphere are conducted by the music: building tension that detonates on the drop, with cinematic camera moves and exposure responding to the sound." \
  --negative-prompt "worst quality, inconsistent motion, blurry, distorted, text, letters, logo, watermark" \
  --width 320 --height 576 --num-frames 121 --frame-rate 24 \
  --num-inference-steps 25 --seed 42 \
  --output-path out.mp4

Tips

• Start prompts with audio reactive.
• Provide a first-frame image — this anchors the scene so the treatment is applied to it rather than replacing it.
• Match a one-line prompt cue to the track's shape (riser/detonation, swell/bloom, groove/pulse).
• Scenes with open negative space (sky, water) give the audio more room for dramatic events.

---

Training

• Framework: LTX Trainer (official), LoRA on LTX-2.3-22B.
• LoRA: rank 32, alpha 32, targets to_q/to_k/to_v/to_out.0.
• Conditioning: audio_to_video + first_frame (p=0.5) so the model learns to conduct a provided scene.
• Checkpoint: step 500 (chosen pre-overfit; later steps drift toward look-collapse and text artifacts).
• Compute: single A100-80GB on Modal.

Data — fully synthetic, zero IP risk

The dataset is 120 procedurally-generated clips. Audio is synthesized as a set of energy-trajectory archetypes (riser_to_drop, pure_riser, swell_to_decay, steady_groove, multi_onset, breakdown_drift); the matching video is rendered with NumPy across randomized scenes and color grades, with camera, exposure, bloom and atmosphere driven by the audio envelope. This guarantees:

• Perfect audio↔video sync (both come from the same signal),
• No third-party footage, music, faces, or logos — clean under the JAM rules,
• Precise control over the behavior being taught.

---

Limitations & honest notes

• Best used image + audio. In pure audio-only text-to-video (no first-frame anchor) the scene is under-constrained and can drift toward the learned aesthetic.
• Audio alone is a tasteful modulator, not a magician — pairing it with a matched prompt cue produces the dramatic "different films" effect.
• Drama is scene-dependent: open skies/water allow big events; dense scenes get subtler energy modulation.
• Occasional sky artifacts; mitigated by the negative prompt and re-rolling the seed.

Acknowledgements

Built for the LTX LoRA Jam with LTX-2.3, LTX Trainer, and the LTX pipelines by Lightricks.