| --- |
| base_model: |
| - "" |
| frameworks: |
| - Pytorch |
| license: Apache License 2.0 |
| tags: [] |
| tasks: |
| - audio-generation |
| --- |
| |
| # DiffSynth-Music |
|
|
| [](https://github.com/modelscope/DiffSynth-Studio) [](https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music) [](https://arxiv.org/abs/2609.12774) |
|
|
| ## Introduction |
|
|
| DiffSynth-Music is a suite of controllable music generation models built on [ACE-Step-1.5](https://modelscope.cn/models/ACE-Step/acestep-v15-xl-sft), trained with the [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) framework, and powered by [Diffusion-Templates](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) for controllable generation. It supports five control modes: |
|
|
| |Control Mode|Model Component|Description| |
| |-|-|-| |
| |Beats|`template_control/model.safetensors`|Aligns the beat of the generated music tightly with the input beat sequence.| |
| |Vocals|`template_control/model.safetensors`|Keeps the vocals consistent with the input vocal track, while the model generates the accompaniment.| |
| |Accompany|`template_control/model.safetensors`|Keeps the accompaniment consistent with the input accompaniment, while the model generates the vocals.| |
| |Prosody|`template_prosody/model.safetensors`|Controls the prosody of the vocal part, matching the timing and vocal style of every syllable to the input prosody.| |
| |Reference (experimental)|`template_reference/model.safetensors`|Generates new music by referencing the style, melody, singing style, timbre, and other characteristics of the input audio.| |
|
|
| ## Inference Code and Examples |
|
|
| ### Installation: |
|
|
| ``` |
| git clone https://github.com/modelscope/DiffSynth-Studio.git |
| cd DiffSynth-Studio |
| pip install -e .[audio] |
| ``` |
|
|
| ### Loading the Models |
|
|
| ```python |
| import torch, torchaudio |
| from diffsynth.pipelines.diffsynth_music import DiffSynthMusicPipeline, ModelConfig |
| from diffsynth.diffusion.template import TemplatePipeline |
| from diffsynth.core.data.operators import LoadMultiTrackAudio |
| from diffsynth.utils.music_tools import extract_prosody, generate_click |
| from modelscope import snapshot_download |
| |
| pipe = DiffSynthMusicPipeline.from_pretrained( |
| torch_dtype=torch.bfloat16, |
| device="cuda", |
| model_configs=[ |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="transformer/model.safetensors"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="conditioner/model.safetensors"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/model.safetensors"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="vae/model.safetensors"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="track_separator/model.safetensors", computation_dtype=torch.float32), |
| ], |
| tokenizer_config=ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/"), |
| ) |
| template = TemplatePipeline.from_pretrained( |
| torch_dtype=torch.bfloat16, |
| device="cuda", |
| model_configs=[ |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_control/"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_prosody/"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_reference/"), |
| ], |
| ) |
| ``` |
|
|
| <details> |
| <summary>Loading the models on low-VRAM GPUs</summary> |
|
|
| ```python |
| import torch, torchaudio |
| from diffsynth.pipelines.diffsynth_music import DiffSynthMusicPipeline, ModelConfig |
| from diffsynth.diffusion.template import TemplatePipeline |
| from diffsynth.core.data.operators import LoadMultiTrackAudio |
| from diffsynth.utils.music_tools import extract_prosody, generate_click |
| from modelscope import snapshot_download |
| |
| vram_config = { |
| "offload_dtype": "disk", |
| "offload_device": "disk", |
| "onload_dtype": "disk", |
| "onload_device": "disk", |
| "preparing_dtype": torch.bfloat16, |
| "preparing_device": "cuda", |
| "computation_dtype": torch.bfloat16, |
| "computation_device": "cuda", |
| } |
| vram_config_cpu = { |
| "offload_dtype": torch.bfloat16, |
| "offload_device": "cpu", |
| "onload_dtype": torch.bfloat16, |
| "onload_device": "cpu", |
| "preparing_dtype": torch.bfloat16, |
| "preparing_device": "cuda", |
| "computation_dtype": torch.bfloat16, |
| "computation_device": "cuda", |
| } |
| vram_config_fp32 = { |
| "offload_dtype": torch.float32, |
| "offload_device": "cpu", |
| "onload_dtype": torch.float32, |
| "onload_device": "cpu", |
| "preparing_dtype": torch.float32, |
| "preparing_device": "cuda", |
| "computation_dtype": torch.float32, |
| "computation_device": "cuda", |
| } |
| |
| pipe = DiffSynthMusicPipeline.from_pretrained( |
| torch_dtype=torch.bfloat16, |
| device="cuda", |
| model_configs=[ |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="transformer/model.safetensors", **vram_config), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="conditioner/model.safetensors", **vram_config), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/model.safetensors", **vram_config), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="vae/model.safetensors", **vram_config_cpu), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="track_separator/model.safetensors", **vram_config_fp32), |
| ], |
| tokenizer_config=ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/"), |
| vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, |
| ) |
| template = TemplatePipeline.from_pretrained( |
| torch_dtype=torch.bfloat16, |
| device="cuda", |
| model_configs=[ |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_control/"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_prosody/"), |
| ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_reference/"), |
| ], |
| lazy_loading=True, |
| ) |
| ``` |
|
|
| </details> |
|
|
| ### Basic Generation Capability |
|
|
| DiffSynth-Music inherits its basic text-to-music generation capability from [ACE-Step-1.5](https://modelscope.cn/models/ACE-Step/acestep-v15-xl-sft). |
|
|
| <audio controls><source src="./assets/audio_1_output.mp3" type="audio/mpeg"></audio> |
|
|
| <details> |
| <summary>Lyrics</summary> |
|
|
| ``` |
| [Intro] |
| |
| 清新海风里有我们旅途 |
| 漆黑海浪上有帆依呀远征 |
| 风暴的咆哮不把恐惧藏水手的胸襟 |
| 祈祷你像无畏的领航人 |
| 懂也不懂的守护航程 |
| 你在甲板上留下的刻痕 |
| 是我梦的风景 |
| |
| 我要送你永不沉的信念 |
| 升起代表勇的黑旗幡 |
| 我要送你永不沉的誓言 |
| 锚连着锚把七海踏遍 |
| 你就是烈焰 |
| 你就是烈焰 |
| 我的血未寒 |
| 不灭的烽火燃在你身边 |
| 我的血未寒 |
| |
| 怒海的狂涛总是起了又平 |
| 凝望指着罗盘的星辰 |
| 我要把酒全都灌进骨里 |
| 陪我一起远行 |
| |
| 我要送你永不沉的信念 |
| 升起代表勇的黑旗幡 |
| 我要送你永不沉的誓言 |
| 锚连着锚把七海踏遍 |
| 你就是烈焰 |
| 你就是烈焰 |
| 我的血未寒 |
| 不灭的烽火燃在你身边 |
| 我的血未寒 |
| |
| 祈祷你像无畏的领航人 |
| 懂也不懂的守护航程 |
| 你在甲板上留下的刻痕 |
| 是我梦的风景 |
| |
| 我要送你永不沉的信念 |
| 升起代表勇的黑旗幡 |
| 我要送你永不沉的誓言 |
| 锚连着锚把七海踏遍 |
| 你就是烈焰 |
| 你就是烈焰 |
| 我的血未寒 |
| 不灭的烽火燃在你身边 |
| 我的血未寒 |
| |
| 我要送你永不沉的信念 |
| 升起代表勇的黑旗幡 |
| 我要送你永不沉的誓言 |
| 锚连着锚把七海踏遍 |
| 你就是烈焰 |
| 你就是烈焰 |
| 我的血未寒 |
| 不灭的烽火燃在你身边 |
| 我的血未寒 |
| ``` |
|
|
| </details> |
|
|
| <details> |
| <summary>Code</summary> |
|
|
| ```python |
| lyrics = "[Intro]\n\n清新海风里有我们旅途\n漆黑海浪上有帆依呀远征\n风暴的咆哮不把恐惧藏水手的胸襟\n祈祷你像无畏的领航人\n懂也不懂的守护航程\n你在甲板上留下的刻痕\n是我梦的风景\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n怒海的狂涛总是起了又平\n凝望指着罗盘的星辰\n我要把酒全都灌进骨里\n陪我一起远行\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n祈祷你像无畏的领航人\n懂也不懂的守护航程\n你在甲板上留下的刻痕\n是我梦的风景\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n" |
| prompt = "An explosive, high-energy pop-rock track with a strong anime theme song feel." |
| snapshot_download("DiffSynth-Studio/DiffSynth-Music", allow_file_pattern="assets/audio_reference.mp3", local_dir="data") |
| |
| audio = template( |
| pipe, |
| prompt=prompt, negative_prompt=pipe.default_negative_prompt, |
| lyrics=lyrics, |
| duration=240, |
| seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, |
| ) |
| torchaudio.save("audio_1_output.mp3", audio, 48000) |
| ``` |
|
|
| </details> |
|
|
| ### Beats |
|
|
| A click track is generated at a fixed BPM (beats per minute) and fed into the Template model as input. Used together with the `bpm` parameter of the base model, it strictly aligns the beat of the generated music. |
|
|
| |Input|Output|Output (with beats)| |
| |-|-|-| |
| |<audio controls><source src="./assets/audio_2_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_2_output.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_2_output_with_beats.mp3" type="audio/mpeg"></audio>| |
|
|
| <details> |
| <summary>Code</summary> |
|
|
| ```python |
| bpm = 120 |
| duration = 240 |
| beats = generate_click(bpm, duration=duration) |
| torchaudio.save("audio_2_input.mp3", beats, 48000) |
| audio = template( |
| pipe, |
| prompt=prompt, negative_prompt=pipe.default_negative_prompt, |
| lyrics=lyrics, |
| duration=duration, |
| seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, |
| bpm=bpm, |
| template_inputs=[{"model_id": 0, "audio": beats}], |
| negative_template_inputs=[{"model_id": 0, "audio": beats * 0}], |
| ) |
| torchaudio.save("audio_2_output.mp3", audio, 48000) |
| torchaudio.save("audio_2_output_with_beats.mp3", audio + beats, 48000) |
| ``` |
|
|
| </details> |
|
|
| ### Vocals & Accompany |
|
|
| [Demucs](https://github.com/facebookresearch/demucs) is used to separate a song into its vocal and instrumental tracks. The model then completes the full song from the input track. |
|
|
| |Input|Output| |
| |-|-| |
| |<audio controls><source src="./assets/audio_3_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_3_output.mp3" type="audio/mpeg"></audio>| |
| |<audio controls><source src="./assets/audio_4_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_4_output.mp3" type="audio/mpeg"></audio>| |
|
|
| <details> |
| <summary>Code</summary> |
|
|
| ```python |
| # Vocals Control |
| audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") |
| vocals = pipe.extract_track(audio, track="vocals") |
| torchaudio.save("audio_3_input.mp3", vocals, 48000) |
| audio = template( |
| pipe, |
| prompt=prompt, negative_prompt=pipe.default_negative_prompt, |
| lyrics="", |
| duration=vocals.shape[1] / 48000, |
| seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, |
| template_inputs=[{"model_id": 0, "audio": vocals}], |
| negative_template_inputs=[{"model_id": 0, "audio": vocals}], |
| target_audio=vocals, target_track="vocals", |
| ) |
| torchaudio.save("audio_3_output.mp3", audio, 48000) |
| |
| # Accompaniment Music Control |
| audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") |
| music = pipe.extract_track(audio, track=["drums", "bass", "other"]) |
| torchaudio.save("audio_4_input.mp3", music, 48000) |
| audio = template( |
| pipe, |
| prompt=prompt, negative_prompt=pipe.default_negative_prompt, |
| lyrics=lyrics, |
| duration=music.shape[1] / 48000, |
| seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, |
| template_inputs=[{"model_id": 0, "audio": music}], |
| negative_template_inputs=[{"model_id": 0, "audio": music}], |
| target_audio=music, target_track=["drums", "bass", "other"], |
| ) |
| torchaudio.save("audio_4_output.mp3", audio, 48000) |
| ``` |
|
|
| </details> |
|
|
| ### Prosody |
|
|
| Prosody is rhythmic information extracted from the vocal track, in which the pronunciation of every syllable is blurred out while its timing and pitch are preserved. |
|
|
| |Input|Output| |
| |-|-| |
| |<audio controls><source src="./assets/audio_5_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_5_output.mp3" type="audio/mpeg"></audio>| |
|
|
| <details> |
| <summary>Code</summary> |
|
|
| ```python |
| audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") |
| vocals = pipe.extract_track(audio, track="vocals") |
| prosody = extract_prosody(vocals) |
| torchaudio.save("audio_5_input.mp3", prosody, 48000) |
| audio = template( |
| pipe, |
| prompt=prompt, negative_prompt=pipe.default_negative_prompt, |
| lyrics=lyrics, |
| duration=prosody.shape[1] / 48000, |
| seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, |
| template_inputs=[{"model_id": 1, "audio": prosody}], |
| negative_template_inputs=[{"model_id": 1, "audio": prosody}], |
| ) |
| torchaudio.save("audio_5_output.mp3", audio, 48000) |
| ``` |
|
|
| </details> |
|
|
| ### Reference (Experimental) |
|
|
| Reference is an experimental feature. The model analyzes the style, melody, singing style, timbre, and other characteristics of the loudest segment of the input audio, and generates new music accordingly. |
|
|
| |Input|Output| |
| |-|-| |
| |<audio controls><source src="./assets/audio_6_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_6_output.mp3" type="audio/mpeg"></audio>| |
|
|
| <details> |
| <summary>Code</summary> |
|
|
| ```python |
| audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") |
| torchaudio.save("audio_6_input.mp3", audio, 48000) |
| audio = template( |
| pipe, |
| prompt="Music", negative_prompt="", # The timbre is controlled by the reference audio. |
| lyrics=lyrics, |
| duration=200, |
| seed=42, tiled=True, cfg_scale=4, num_inference_steps=100, |
| template_inputs=[{"model_id": 2, "audio": audio}], |
| ) |
| torchaudio.save("audio_6_output.mp3", audio, 48000) |
| ``` |
|
|
| </details> |
|
|
| ## Training Code |
|
|
| Please refer to the [DiffSynth-Studio documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/DiffSynth-Music.html). |
|
|
|
|
