| | --- |
| | license: mit |
| | pipeline_tag: image-to-video |
| | tags: |
| | - lip-sync |
| | - talking-head |
| | - face-animation |
| | - musetalk |
| | - safetensors |
| | --- |
| | |
| | # MuseTalk V15 UNet — AEmotionStudio Mirror |
| |
|
| | **Mirror of the MuseTalk V15 UNet weights** for use with [ComfyUI-FFMPEGA](https://github.com/AEmotionStudio/ComfyUI-FFMPEGA). |
| |
|
| | ## About |
| |
|
| | [MuseTalk](https://github.com/TMElyralab/MuseTalk) is a real-time, high-quality lip sync model that synchronizes lip movements in video to match provided audio. It supports: |
| |
|
| | - **Video + Audio lip sync** — make a person in a video speak new dialogue |
| | - **Image + Audio talking head** — animate a portrait photo with speech audio |
| | - **Multi-face support** — sync multiple faces in a single video |
| | - **Batch inference** — process multiple frames simultaneously for speed |
| |
|
| | ## Files |
| |
|
| | | File | Precision | Size | Description | |
| | |------|-----------|------|-------------| |
| | | `musetalkV15/unet_fp16.safetensors` | fp16 | ~1.6 GB | **Recommended** — half-precision UNet weights | |
| | | `musetalkV15/unet.safetensors` | fp32 | ~3.2 GB | Full-precision UNet weights (fallback) | |
| | | `musetalkV15/musetalk.json` | — | < 1 KB | Model configuration | |
| |
|
| | ## Usage with ComfyUI-FFMPEGA |
| |
|
| | This model is **auto-downloaded** when you use the `lip_sync` skill in [ComfyUI-FFMPEGA](https://github.com/AEmotionStudio/ComfyUI-FFMPEGA). |
| |
|
| | ### Example Prompts |
| |
|
| | ``` |
| | Lip sync this video to the provided audio |
| | ``` |
| | ``` |
| | Make the person's lips match the speech |
| | ``` |
| | ``` |
| | Dub this video with the new voiceover |
| | ``` |
| |
|
| | The fp16 variant is preferred by default when `use_float16` is enabled (default). Falls back to fp32 if fp16 is unavailable. |
| |
|
| | ### Manual Download |
| |
|
| | If auto-download is disabled, download the files and place them in: |
| | ``` |
| | ComfyUI/models/musetalk/musetalkV15/ |
| | ``` |
| |
|
| | ### Additional Dependencies |
| |
|
| | MuseTalk also requires these models (auto-downloaded from HuggingFace on first use): |
| | - **SD-VAE** (`stabilityai/sd-vae-ft-mse`) — ~335 MB |
| | - **Whisper-tiny** (`openai/whisper-tiny`) — ~75 MB |
| |
|
| | ## VRAM Requirements |
| |
|
| | - **Minimum**: ~4 GB |
| | - **Recommended**: ~6 GB |
| | - Uses subprocess isolation to prevent CUDA memory leaks |
| |
|
| | ## License |
| |
|
| | - **MuseTalk code**: [MIT License](https://github.com/TMElyralab/MuseTalk/blob/main/LICENSE) |
| | - **SD-VAE**: [CreativeML Open RAIL-M](https://huggingface.co/stabilityai/sd-vae-ft-mse/blob/main/LICENSE) |
| | - **Whisper**: [MIT License](https://github.com/openai/whisper/blob/main/LICENSE) |
| |
|
| | ## Citation |
| |
|
| | ```bibtex |
| | @article{zhang2024musetalk, |
| | title={MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting}, |
| | author={Zhang, Yue and Liu, Minhao and Chen, Zhaokang and Wu, Bin and others}, |
| | journal={arXiv preprint arXiv:2410.10122}, |
| | year={2024} |
| | } |
| | ``` |
| |
|
