Hugging Face's logo

DiffSynth-Studio
/
DiffSynth-Music

Safetensors
Model card Files
xet
 Community
2
DiffSynth-Music
File size: 15,318 Bytes
26ba8eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
---
base_model:
- ""
frameworks:
- Pytorch
license: Apache License 2.0
tags: []
tasks:
- audio-generation
---

# DiffSynth-Music

[![GitHub](https://img.shields.io/badge/GitHub-%23121011.svg?logo=github&logoColor=white)](https://github.com/modelscope/DiffSynth-Studio) [![Model](https://img.shields.io/badge/Model-8A2BE2.svg?logo=modelscope&logoColor=white)](https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music) [![Report](https://img.shields.io/badge/Report-%23E34F26.svg?logo=readthedocs&logoColor=white)](https://arxiv.org/abs/2609.12774)

## Introduction

DiffSynth-Music is a suite of controllable music generation models built on [ACE-Step-1.5](https://modelscope.cn/models/ACE-Step/acestep-v15-xl-sft), trained with the [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) framework, and powered by [Diffusion-Templates](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) for controllable generation. It supports five control modes:

|Control Mode|Model Component|Description|
|-|-|-|
|Beats|`template_control/model.safetensors`|Aligns the beat of the generated music tightly with the input beat sequence.|
|Vocals|`template_control/model.safetensors`|Keeps the vocals consistent with the input vocal track, while the model generates the accompaniment.|
|Accompany|`template_control/model.safetensors`|Keeps the accompaniment consistent with the input accompaniment, while the model generates the vocals.|
|Prosody|`template_prosody/model.safetensors`|Controls the prosody of the vocal part, matching the timing and vocal style of every syllable to the input prosody.|
|Reference (experimental)|`template_reference/model.safetensors`|Generates new music by referencing the style, melody, singing style, timbre, and other characteristics of the input audio.|

## Inference Code and Examples

### Installation:

```
git clone https://github.com/modelscope/DiffSynth-Studio.git  
cd DiffSynth-Studio
pip install -e .[audio]
```

### Loading the Models

```python
import torch, torchaudio
from diffsynth.pipelines.diffsynth_music import DiffSynthMusicPipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.core.data.operators import LoadMultiTrackAudio
from diffsynth.utils.music_tools import extract_prosody, generate_click
from modelscope import snapshot_download

pipe = DiffSynthMusicPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="transformer/model.safetensors"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="conditioner/model.safetensors"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/model.safetensors"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="vae/model.safetensors"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="track_separator/model.safetensors", computation_dtype=torch.float32),
    ],
    tokenizer_config=ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/"),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_control/"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_prosody/"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_reference/"),
    ],
)
```

<details>
<summary>Loading the models on low-VRAM GPUs</summary>

```python
import torch, torchaudio
from diffsynth.pipelines.diffsynth_music import DiffSynthMusicPipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.core.data.operators import LoadMultiTrackAudio
from diffsynth.utils.music_tools import extract_prosody, generate_click
from modelscope import snapshot_download

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
vram_config_cpu = {
    "offload_dtype": torch.bfloat16,
    "offload_device": "cpu",
    "onload_dtype": torch.bfloat16,
    "onload_device": "cpu",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
vram_config_fp32 = {
    "offload_dtype": torch.float32,
    "offload_device": "cpu",
    "onload_dtype": torch.float32,
    "onload_device": "cpu",
    "preparing_dtype": torch.float32,
    "preparing_device": "cuda",
    "computation_dtype": torch.float32,
    "computation_device": "cuda",
}

pipe = DiffSynthMusicPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="transformer/model.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="conditioner/model.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/model.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="vae/model.safetensors", **vram_config_cpu),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="track_separator/model.safetensors", **vram_config_fp32),
    ],
    tokenizer_config=ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_control/"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_prosody/"),
        ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_reference/"),
    ],
    lazy_loading=True,
)
```

</details>

### Basic Generation Capability

DiffSynth-Music inherits its basic text-to-music generation capability from [ACE-Step-1.5](https://modelscope.cn/models/ACE-Step/acestep-v15-xl-sft).

<audio controls><source src="./assets/audio_1_output.mp3" type="audio/mpeg"></audio>

<details>
<summary>Lyrics</summary>

```
[Intro]

清新海风里有我们旅途
漆黑海浪上有帆依呀远征
风暴的咆哮不把恐惧藏水手的胸襟
祈祷你像无畏的领航人
懂也不懂的守护航程
你在甲板上留下的刻痕
是我梦的风景

我要送你永不沉的信念
升起代表勇的黑旗幡
我要送你永不沉的誓言
锚连着锚把七海踏遍
你就是烈焰
你就是烈焰
我的血未寒
不灭的烽火燃在你身边
我的血未寒

怒海的狂涛总是起了又平
凝望指着罗盘的星辰
我要把酒全都灌进骨里
陪我一起远行

我要送你永不沉的信念
升起代表勇的黑旗幡
我要送你永不沉的誓言
锚连着锚把七海踏遍
你就是烈焰
你就是烈焰
我的血未寒
不灭的烽火燃在你身边
我的血未寒

祈祷你像无畏的领航人
懂也不懂的守护航程
你在甲板上留下的刻痕
是我梦的风景

我要送你永不沉的信念
升起代表勇的黑旗幡
我要送你永不沉的誓言
锚连着锚把七海踏遍
你就是烈焰
你就是烈焰
我的血未寒
不灭的烽火燃在你身边
我的血未寒

我要送你永不沉的信念
升起代表勇的黑旗幡
我要送你永不沉的誓言
锚连着锚把七海踏遍
你就是烈焰
你就是烈焰
我的血未寒
不灭的烽火燃在你身边
我的血未寒
```

</details>

<details>
<summary>Code</summary>

```python
lyrics = "[Intro]\n\n清新海风里有我们旅途\n漆黑海浪上有帆依呀远征\n风暴的咆哮不把恐惧藏水手的胸襟\n祈祷你像无畏的领航人\n懂也不懂的守护航程\n你在甲板上留下的刻痕\n是我梦的风景\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n怒海的狂涛总是起了又平\n凝望指着罗盘的星辰\n我要把酒全都灌进骨里\n陪我一起远行\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n祈祷你像无畏的领航人\n懂也不懂的守护航程\n你在甲板上留下的刻痕\n是我梦的风景\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n"
prompt = "An explosive, high-energy pop-rock track with a strong anime theme song feel."
snapshot_download("DiffSynth-Studio/DiffSynth-Music", allow_file_pattern="assets/audio_reference.mp3", local_dir="data")

audio = template(
    pipe,
    prompt=prompt, negative_prompt=pipe.default_negative_prompt,
    lyrics=lyrics,
    duration=240,
    seed=42, tiled=True, cfg_scale=4, num_inference_steps=50,
)
torchaudio.save("audio_1_output.mp3", audio, 48000)
```

</details>

### Beats

A click track is generated at a fixed BPM (beats per minute) and fed into the Template model as input. Used together with the `bpm` parameter of the base model, it strictly aligns the beat of the generated music.

|Input|Output|Output (with beats)|
|-|-|-|
|<audio controls><source src="./assets/audio_2_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_2_output.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_2_output_with_beats.mp3" type="audio/mpeg"></audio>|

<details>
<summary>Code</summary>

```python
bpm = 120
duration = 240
beats = generate_click(bpm, duration=duration)
torchaudio.save("audio_2_input.mp3", beats, 48000)
audio = template(
    pipe,
    prompt=prompt, negative_prompt=pipe.default_negative_prompt,
    lyrics=lyrics,
    duration=duration,
    seed=42, tiled=True, cfg_scale=4, num_inference_steps=50,
    bpm=bpm,
    template_inputs=[{"model_id": 0, "audio": beats}],
    negative_template_inputs=[{"model_id": 0, "audio": beats * 0}],
)
torchaudio.save("audio_2_output.mp3", audio, 48000)
torchaudio.save("audio_2_output_with_beats.mp3", audio + beats, 48000)
```

</details>

### Vocals & Accompany

[Demucs](https://github.com/facebookresearch/demucs) is used to separate a song into its vocal and instrumental tracks. The model then completes the full song from the input track.

|Input|Output|
|-|-|
|<audio controls><source src="./assets/audio_3_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_3_output.mp3" type="audio/mpeg"></audio>|
|<audio controls><source src="./assets/audio_4_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_4_output.mp3" type="audio/mpeg"></audio>|

<details>
<summary>Code</summary>

```python
# Vocals Control
audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3")
vocals = pipe.extract_track(audio, track="vocals")
torchaudio.save("audio_3_input.mp3", vocals, 48000)
audio = template(
    pipe,
    prompt=prompt, negative_prompt=pipe.default_negative_prompt,
    lyrics="",
    duration=vocals.shape[1] / 48000,
    seed=42, tiled=True, cfg_scale=4, num_inference_steps=50,
    template_inputs=[{"model_id": 0, "audio": vocals}],
    negative_template_inputs=[{"model_id": 0, "audio": vocals}],
    target_audio=vocals, target_track="vocals",
)
torchaudio.save("audio_3_output.mp3", audio, 48000)

# Accompaniment Music Control
audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3")
music = pipe.extract_track(audio, track=["drums", "bass", "other"])
torchaudio.save("audio_4_input.mp3", music, 48000)
audio = template(
    pipe,
    prompt=prompt, negative_prompt=pipe.default_negative_prompt,
    lyrics=lyrics,
    duration=music.shape[1] / 48000,
    seed=42, tiled=True, cfg_scale=4, num_inference_steps=50,
    template_inputs=[{"model_id": 0, "audio": music}],
    negative_template_inputs=[{"model_id": 0, "audio": music}],
    target_audio=music, target_track=["drums", "bass", "other"],
)
torchaudio.save("audio_4_output.mp3", audio, 48000)
```

</details>

### Prosody

Prosody is rhythmic information extracted from the vocal track, in which the pronunciation of every syllable is blurred out while its timing and pitch are preserved.

|Input|Output|
|-|-|
|<audio controls><source src="./assets/audio_5_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_5_output.mp3" type="audio/mpeg"></audio>|

<details>
<summary>Code</summary>

```python
audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3")
vocals = pipe.extract_track(audio, track="vocals")
prosody = extract_prosody(vocals)
torchaudio.save("audio_5_input.mp3", prosody, 48000)
audio = template(
    pipe,
    prompt=prompt, negative_prompt=pipe.default_negative_prompt,
    lyrics=lyrics,
    duration=prosody.shape[1] / 48000,
    seed=42, tiled=True, cfg_scale=4, num_inference_steps=50,
    template_inputs=[{"model_id": 1, "audio": prosody}],
    negative_template_inputs=[{"model_id": 1, "audio": prosody}],
)
torchaudio.save("audio_5_output.mp3", audio, 48000)
```

</details>

### Reference (Experimental)

Reference is an experimental feature. The model analyzes the style, melody, singing style, timbre, and other characteristics of the loudest segment of the input audio, and generates new music accordingly.

|Input|Output|
|-|-|
|<audio controls><source src="./assets/audio_6_input.mp3" type="audio/mpeg"></audio>|<audio controls><source src="./assets/audio_6_output.mp3" type="audio/mpeg"></audio>|

<details>
<summary>Code</summary>

```python
audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3")
torchaudio.save("audio_6_input.mp3", audio, 48000)
audio = template(
    pipe,
    prompt="Music", negative_prompt="", # The timbre is controlled by the reference audio.
    lyrics=lyrics,
    duration=200,
    seed=42, tiled=True, cfg_scale=4, num_inference_steps=100,
    template_inputs=[{"model_id": 2, "audio": audio}],
)
torchaudio.save("audio_6_output.mp3", audio, 48000)
```

</details>

## Training Code

Please refer to the [DiffSynth-Studio documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/DiffSynth-Music.html).