jdopensource

/

JoyAI-Echo

like 164

Follow

JD.com Open Source 323

First — thank you for open-sourcing JoyAI-Echo. Genuine minute-scale, memory-consistent audio+video with cross-shot identity is rare and having it runnable at home at all is a gift. I've been driving it hard for a found-footage web series on consumer hardware (an RTX 5090 32 GB and a 3090 24 GB), and I wanted to (1) contribute a patch with fixes + features that made the multishot path far more usable for me, and (2) share some honest field notes, including one that's actually a compliment to your architecture.

I've put a small patch package (fixes/features layered on the community GGUF node pack, with a README) here: https://huggingface.co/joeygambino/joyai-echo-multishot-patch. Everything is drop-in over an existing install; new widgets append at the end of existing nodes so saved graphs keep their values.

What's in the patch

Fixes

• Cross-shot memory could silently switch off. With audio memory disabled, the video memory-bank save was gated on the audio latent being present, so the bank never filled — identity quietly reset every shot (memory_size=0). Decoupled: storage is now unconditional, and the audio-memory flag gates only injection.
• GGUF text-encoder loading — meta-strip of the unused vision/projection heads and a device-unify pin for the embeddings processor, so a text-only Gemma-3 GGUF loads without "Cannot copy out of meta tensor" / addmm device-mismatch.

Features

• Per-domain negative steering — since the DMD pipeline has no CFG, split the embedding-space lever into separate video (kills burned-in captions) and audio (kills invented score) channels, norm-preserving. Also documents the scene-lock failure mode when the video scale is pushed too high.
• Passthrough mode on the enhancer — feed finished {"prompts":[...]} JSON with no LLM call / no key.
• Reference-image conditioning — up to 4 identity references prepended as video-only conditioning clips (never bank slots — silent bank slots injected audio noise), plus a head-trim for the reference-morph flash at shot starts.
• Shot transitions — cut / dissolve / VHS-glitch boundary treatment.
• fp8 transformer option — quantizes the DiT linears to float8_e4m3fn at load from the bf16 checkpoint (via your vendored QuantizationPolicy.fp8_cast), roughly halving DiT weight memory while keeping the memory training and leaving the VAEs/text-encoder in bf16.
• Tiled VAE decode — temporal-chunked decode so a long high-res shot (e.g. 241f at 1280×736) doesn't hard-abort the VAE at the end on a 24–32 GB card.
• A model-file dropdown and a few automation nodes (unified prompt-source dropdown, an auto reference-picker by character name, and a None-tolerant image batcher) that let me queue long batches without rewiring.

Details and apply instructions are in the README. Happy to turn any of this into a proper PR if useful.

Field notes (offered constructively)

The big positive — your graph elevates the base model. The most striking thing I found: running the base LTX 2.3 distilled checkpoint through your multishot nodes produced noticeably better audio (dialogue especially) and video than the same model does in my standard ComfyUI LTX graphs — where I'd get a "Speak-and-Spell" robotic voice on roughly every other clip. In your graph that mostly went away. My best guess is the combination of no CFG on the audio branch, the separate per-domain audio/video conditioning off the Gemma encoder, and the bf16 end-to-end audio VAE + vocoder path. Whatever the reason, the pipeline design itself seems to be doing real work, independent of which DiT sits in it. That's a genuine result and a credit to how the graph is built.

Where I struggled — native voice/lip-sync fidelity. In my hands, JoyAI-Echo's own synthesized dialogue tended to come out with a persistent metallic/synthetic quality and some lip-sync drift that I couldn't fully dial out — to the point where, on identical scripts through the same graph, base LTX 2.3's voice read as more natural to blind reviewers than Echo's did. I want to stress this is very likely something on my end, and I'd genuinely welcome guidance:

• Most of my Echo runs were GGUF-quantized. Heads-up for the GGUF community: the common Q8 conversions I tried were missing four tensors — text_embedding_projection.{audio,video}_aggregate_embed.{weight,bias} — which live top-level in the release safetensors and get dropped by standard converters. With them missing you get robotic voice, prompt text being read aloud, and identity loss, while the load log looks clean. Rebuilding the GGUF with those four tensors preserved fixed the worst of it. This may explain a lot of "Echo sounds robotic" reports.
• Even on the bf16 release I couldn't match the base model on voice naturalness, so I suspect I'm still misconfiguring something. I was often running above the 480p→720p native band (up to 736×1280) with heavy continuous dialogue.

So my open questions for the team:

1. What resolution / frames-per-shot / dialogue density do you consider the model's sweet spot for best voice quality specifically?
2. Are there settings (memory size, any audio-side knob) that most affect lip-sync tightness on dialogue-heavy shots?
3. Any timeline on Echo-SR, and is native above ~720p expected to degrade voice as well as picture?

Thanks again for the release and for reading. Even where I hit walls, this is the most capable open multishot A/V model I've been able to run locally, and the graph has quietly improved everything else I've put through it.

*Attribution: this patch layers on the community ComfyUI_JoyAI_Echo_GGUF_Nodes pack (the "Rebels" GGUF loader wrapper around JoyAI-Echo), not on official JoyAI code — the two edited libs/ files are your vendored ltx_distillation with small additions. JoyAI-Echo is under the LTX-2 Community License (research / non-commercial); nothing here changes that, and I'm not redistributing any model weights or the Gemma tokenizer assets. *