OzzyGT/YuE2-Modular

YuE2 Modular Diffusers

A 69-second song generated with the code below, seed 831001.

Modular Diffusers blocks for m-a-p/YuE2-3B: full songs from a style description and lyrics, planned as an ABC score first, as 48 kHz stereo audio. The model code and weights are in OzzyGT/YuE2-3B-Diffusers.

Note: This model requires tiktoken and the example uses soundfile, install them with pip install tiktoken soundfile.

Licensing

The code is Apache-2.0, adapted from multimodal-art-projection/YuE. The weights are CC BY-NC 4.0 (non-commercial); see the model repository.

Sample song

The song above was generated with the following code:

import soundfile as sf
import torch
from diffusers import ModularPipelineBlocks


blocks = ModularPipelineBlocks.from_pretrained(  # load the blocks first to avoid warnings
    "OzzyGT/YuE2-Modular",
    trust_remote_code=True,
    components_repo="OzzyGT/YuE2-3B-Diffusers",
    trust_components_code=True,
)
pipe = blocks.init_pipeline()
pipe.load_components(dtype={"transformer": torch.bfloat16, "vae": torch.float32})
pipe.to("cuda")

style = "English, warm piano pop, expressive female voice, acoustic piano, rounded bass and light drums, lyrical memorable melody, unhurried phrasing, 88 BPM"
lyrics = """[Verse]
Neon fades along the lane
Footsteps keep the time of rain
Fold the night and leave it here
Morning has a sky to clear

[Chorus]
Let the day come into view
Every road begins with you
Hold a little room for light
We will sing beyond the night"""

state = pipe(style=style, lyrics=lyrics, seed=831001, use_cuda_graph=True)
sf.write("song.flac", state.get("audios")[0].T.numpy(), state.get("sample_rate"), subtype="PCM_24")