Mirror MuseTalk V1.5 + deps from upstream (A.I.M.I Stage 2)

README.md

@@ -0,0 +1,50 @@
+---
+license: mit
+tags:
+  - lip-sync
+  - musetalk
+  - talking-head
+  - mirror
+library_name: pytorch
+---
+
+# MuseTalk Mirror (A.I.M.I)
+
+Mirror of [TMElyralab/MuseTalk](https://huggingface.co/TMElyralab/MuseTalk) V1.5 plus its inference-time dependencies, re-hosted for stable URLs inside the [A.I.M.I](https://aimi.app) desktop product. Contents are unmodified.
+
+MuseTalk re-syncs the lips of an existing video to match a new audio track (mouth-region editing, rest of frame passes through). Pairs with our TTS + Voice-Clone stack for full "text → lip-synced video" workflows.
+
+## Files
+
+| Folder / File | Upstream | Size | Purpose |
+|---|---|---|---|
+| `musetalkV15/unet.pth` | TMElyralab/MuseTalk | 3.24 GB | MuseTalk V1.5 UNet weights |
+| `musetalkV15/musetalk.json` | TMElyralab/MuseTalk | 748 B | UNet config |
+| `sd-vae-ft-mse/diffusion_pytorch_model.bin` | stabilityai/sd-vae-ft-mse | 319 MB | VAE for face latents |
+| `sd-vae-ft-mse/config.json` | stabilityai/sd-vae-ft-mse | 547 B | VAE config |
+| `whisper/pytorch_model.bin` | openai/whisper-tiny | 144 MB | Audio feature extraction (tiny) |
+| `dwpose/dw-ll_ucoco_384.pth` | yzd-v/DWPose | 388 MB | Face bbox + pose detection |
+| `face-parse-bisent/79999_iter.pth` | ManyOtherFunctions/face-parse-bisent | 51 MB | BiSeNet face-region parser |
+| `face-parse-bisent/resnet18-5c106cde.pth` | pytorch.org/models | 45 MB | ResNet18 backbone for face-parser |
+
+Total: ~4.1 GB.
+
+## Licenses
+
+| Component | License |
+|---|---|
+| MuseTalk | MIT (Tencent Music Entertainment Lyra Lab) |
+| SD-VAE-ft-MSE | CreativeML Open RAIL-M (Stability AI) |
+| Whisper | MIT (OpenAI) |
+| DWPose | Apache 2.0 |
+| face-parse-bisent | MIT |
+| ResNet18 (pretrained) | BSD-3-Clause (PyTorch / Facebook) |
+
+All components are commercial-use-compatible. Redistributed unchanged. See upstream repos for full license texts.
+
+## Attribution
+
+- **MuseTalk**: Yue Zhang, Minhao Liu, Zhaokang Chen, Bin Wu, Yubin Zeng, Chao Zhan, Yingjie He, Junxin Huang, Wenjiang Zhou — *MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting* (2024).
+- **Whisper**: Alec Radford et al. — *Robust Speech Recognition via Large-Scale Weak Supervision* (OpenAI, 2022).
+- **DWPose**: Zhendong Yang, Ailing Zeng, Chun Yuan, Yu Li — *Effective Whole-body Pose Estimation with Two-stages Distillation* (ICCV 2023).
+- **BiSeNet**: Changqian Yu et al. — *BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation* (ECCV 2018).

dwpose/dw-ll_ucoco_384.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d9408b13cd863c4e95a149dd31232f88f2a12aa6cf8964ed74d7d97748c7a07
+size 406878486

face-parse-bisent/79999_iter.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:468e13ca13a9b43cc0881a9f99083a430e9c0a38abd935431d1c28ee94b26567
+size 53289463

face-parse-bisent/resnet18-5c106cde.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c106cde386e87d4033832f2996f5493238eda96ccf559d1d62760c4de0613f8
+size 46827520

musetalkV15/musetalk.json

@@ -0,0 +1,36 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.6.0.dev0",
+  "act_fn": "silu",
+  "attention_head_dim": 8,
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "cross_attention_dim": 384,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 8,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "out_channels": 4,
+  "sample_size": 64,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ]
+}

musetalkV15/unet.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ebf6c98c181e20838e4c0054e96e944ac60d5d692cc01db42839fe11b787007
+size 3400074924

sd-vae-ft-mse/config.json

@@ -0,0 +1,29 @@
+{
+  "_class_name": "AutoencoderKL",
+  "_diffusers_version": "0.4.2",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    512,
+    512
+  ],
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "in_channels": 3,
+  "latent_channels": 4,
+  "layers_per_block": 2,
+  "norm_num_groups": 32,
+  "out_channels": 3,
+  "sample_size": 256,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ]
+}

sd-vae-ft-mse/diffusion_pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b4889b6b1d4ce7ae320a02dedaeff1780ad77d415ea0d744b476155c6377ddc
+size 334707217

whisper/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9607f98a2b22d9e229ae43c52ecea79dcede9e0c5cfae67e8da6eda86d8aac1d
+size 151095027