Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -9,17 +9,34 @@ tags:
|
|
| 9 |
|
| 10 |
# MMAudio Models (FP16 Safetensors)
|
| 11 |
|
| 12 |
-
FP16 `.safetensors` conversions of [MMAudio](https://github.com/hkchengrex/MMAudio) model checkpoints for use with [ComfyUI-FFMPEGA](https://github.com/AEmotionStudio/ComfyUI-FFMPEGA).
|
| 13 |
|
| 14 |
## Models
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|------|-------------|-----------|---------------|
|
| 18 |
-
| `mmaudio_large_44k_v2.safetensors` | MMAudio large model (44kHz, v2) | 1,966 MB | 3,932 MB |
|
| 19 |
-
| `v1-44.safetensors` | VAE decoder (44kHz) | 583 MB | 1,165 MB |
|
| 20 |
-
| `synchformer_state_dict.safetensors` | Synchformer temporal encoder | 453 MB | 906 MB |
|
| 21 |
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
## Usage
|
| 25 |
|
|
@@ -28,17 +45,22 @@ These models are automatically downloaded by the `generate_audio` skill in Comfy
|
|
| 28 |
**Manual installation:**
|
| 29 |
```
|
| 30 |
ComfyUI/models/mmaudio/
|
| 31 |
-
├── mmaudio_large_44k_v2.
|
| 32 |
-
├── v1-44.
|
| 33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
```
|
| 35 |
|
| 36 |
## License
|
| 37 |
|
| 38 |
-
> ⚠️ **CC-BY-NC 4.0** —
|
| 39 |
> Commercial use of the models is restricted. The code that loads/runs them is MIT/GPL-3.0.
|
|
|
|
|
|
|
| 40 |
|
| 41 |
-
##
|
| 42 |
|
| 43 |
-
|
| 44 |
-
Paper: *Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis* (CVPR 2025)
|
|
|
|
| 9 |
|
| 10 |
# MMAudio Models (FP16 Safetensors)
|
| 11 |
|
| 12 |
+
FP16 `.safetensors` conversions of [MMAudio](https://github.com/hkchengrex/MMAudio) model checkpoints and dependencies for use with [ComfyUI-FFMPEGA](https://github.com/AEmotionStudio/ComfyUI-FFMPEGA).
|
| 13 |
|
| 14 |
## Models
|
| 15 |
|
| 16 |
+
### MMAudio Core
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
| File | Description | FP16 Size |
|
| 19 |
+
|------|-------------|-----------|
|
| 20 |
+
| `mmaudio_large_44k_v2.safetensors` | MMAudio large model (44kHz, v2) | 1,966 MB |
|
| 21 |
+
| `v1-44.safetensors` | VAE decoder (44kHz) | 583 MB |
|
| 22 |
+
| `synchformer_state_dict.safetensors` | Synchformer temporal encoder | 453 MB |
|
| 23 |
+
|
| 24 |
+
### Dependencies
|
| 25 |
+
|
| 26 |
+
| File | Description | Size |
|
| 27 |
+
|------|-------------|------|
|
| 28 |
+
| `apple_DFN5B-CLIP-ViT-H-14-384_fp16.safetensors` | CLIP vision encoder (fp16) | 1,882 MB |
|
| 29 |
+
| `bigvgan_v2_44khz_128band_512x/` | BigVGAN v2 vocoder (44kHz) | ~467 MB |
|
| 30 |
+
|
| 31 |
+
All models converted from original FP32 sources → **FP16** `.safetensors` (50% size reduction).
|
| 32 |
+
|
| 33 |
+
## Sources
|
| 34 |
+
|
| 35 |
+
All models were downloaded from their **original sources** and converted by us:
|
| 36 |
+
|
| 37 |
+
- **MMAudio**: [hkchengrex/MMAudio](https://huggingface.co/hkchengrex/MMAudio)
|
| 38 |
+
- **CLIP**: [apple/DFN5B-CLIP-ViT-H-14-384](https://huggingface.co/apple/DFN5B-CLIP-ViT-H-14-384)
|
| 39 |
+
- **BigVGAN**: [nvidia/bigvgan_v2_44khz_128band_512x](https://huggingface.co/nvidia/bigvgan_v2_44khz_128band_512x)
|
| 40 |
|
| 41 |
## Usage
|
| 42 |
|
|
|
|
| 45 |
**Manual installation:**
|
| 46 |
```
|
| 47 |
ComfyUI/models/mmaudio/
|
| 48 |
+
├── mmaudio_large_44k_v2.safetensors
|
| 49 |
+
├── v1-44.safetensors
|
| 50 |
+
├── synchformer_state_dict.safetensors
|
| 51 |
+
├── apple_DFN5B-CLIP-ViT-H-14-384_fp16.safetensors
|
| 52 |
+
└── bigvgan_v2_44khz_128band_512x/
|
| 53 |
+
├── config.json
|
| 54 |
+
└── bigvgan_generator.pt
|
| 55 |
```
|
| 56 |
|
| 57 |
## License
|
| 58 |
|
| 59 |
+
> ⚠️ **CC-BY-NC 4.0** — MMAudio model checkpoints are licensed under [Creative Commons Attribution-NonCommercial 4.0](https://creativecommons.org/licenses/by-nc/4.0/).
|
| 60 |
> Commercial use of the models is restricted. The code that loads/runs them is MIT/GPL-3.0.
|
| 61 |
+
>
|
| 62 |
+
> BigVGAN is licensed under MIT. CLIP (DFN5B) is licensed under Apple's research license.
|
| 63 |
|
| 64 |
+
## Paper
|
| 65 |
|
| 66 |
+
*Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis* (CVPR 2025)
|
|
|