| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - audiocraft |
| - melodyflow |
| - music-generation |
| - music-editing |
| - flow-matching |
| language: |
| - en |
| --- |
| |
| # MelodyFlow — AEmotionStudio mirror |
|
|
| 1:1 mirror of [facebook/melodyflow-t24-30secs](https://huggingface.co/facebook/melodyflow-t24-30secs). Used by the MAESTRO / Æmotion Studio AI Workstation's **MelodyFlow** panel (Design → MelodyFlow). |
|
|
| ## License — Non-Commercial |
|
|
| **Weights:** CC-BY-NC-4.0. Generated outputs may NOT be used in commercial projects, paid releases, or client work. |
|
|
| **Code (audiocraft):** MIT. MelodyFlow's inference code lives in the [`facebook/MelodyFlow`](https://huggingface.co/spaces/facebook/MelodyFlow) HuggingFace Space — Meta uploaded it there but never merged it into audiocraft `main`. MAESTRO vendors that Space's `audiocraft/` subtree under `backend/ai/melodyflow_pkg/`. The non-commercial clause attaches only to the weights and to anything derived from running them. |
|
|
| ## Format |
|
|
| This mirror keeps the upstream `.bin` layout (PyTorch pickle) verbatim — `state_dict.bin` (the flow-matching DiT language model) plus `compression_state_dict.bin` (the EnCodec compression model, 2-channel / 32 kHz). We do NOT convert to safetensors here because the vendored audiocraft loader expects pickled `{xp.cfg, best_state}` packages and reads the OmegaConf cfg blob alongside the tensor dict in one `torch.load` call. Splitting cfg into a sidecar would require a custom loader — deferred. |
|
|
| PyTorch 2.6+'s default `weights_only=True` rejects these pickles (numpy scalars in `xp.cfg`). MAESTRO's runner wraps the load in a `_TorchLoadWeightsOnlyShim` context manager; vanilla audiocraft users on torch ≥ 2.6 will hit the same issue and need a similar shim. |
|
|
| ## Loading |
|
|
| ```python |
| # Requires the facebook/MelodyFlow Space's audiocraft subtree on PYTHONPATH |
| # (the upstream audiocraft PyPI release does NOT include MelodyFlow). |
| from audiocraft.models import MelodyFlow |
| model = MelodyFlow.get_pretrained('AEmotionStudio/melodyflow-models', device='cuda') |
| |
| # Generate from text alone: |
| model.set_generation_params(solver='midpoint', steps=64, duration=10.0) |
| wav = model.generate(descriptions=['cinematic strings']) |
| |
| # OR edit a source clip via regularized latent inversion: |
| import torchaudio |
| src, sr = torchaudio.load('source.wav') # MelodyFlow's EnCodec is stereo |
| if src.shape[0] == 1: src = src.repeat(2, 1) |
| src = src.unsqueeze(0).to('cuda') |
| import torch |
| with torch.no_grad(): |
| prompt_tokens = model.encode_audio(src) |
| model.set_editing_params(solver='euler', steps=25, regularize=True, |
| regularize_iters=4, lambda_kl=0.2) |
| edited = model.edit(prompt_tokens=prompt_tokens, |
| descriptions=['solo piano with reverb'], |
| src_descriptions=['gentle arpeggio']) |
| torchaudio.save('edited.wav', edited[0].cpu(), model.sample_rate) |
| ``` |
|
|
| ## Citation |
|
|
| MelodyFlow is described in: |
|
|
| > Le Lan, G., Nagaraja, V., Chang, E., Kant, D., Ni, Z., Shi, Y., Iandola, F., & Chandra, V. (2024). **High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching**. arXiv:2407.03648. |
|
|
