Instructions to use m-a-p/YuE2-Vae with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use m-a-p/YuE2-Vae with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="m-a-p/YuE2-Vae", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("m-a-p/YuE2-Vae", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
The 100 Hz ripple and the 6/12 kHz seams of the Standard decoder, measured, and a grown decoder trained against them (YuE2-VAE-RS x3)
Hello m-a-p, and thank you for the YuE2 VAE and its open weights. Everything below starts from your decoder, YuE2-Vae, which we call Standard.
What we measured in Standard. We decoded eight unseen 30-second clips whole, in float32, and compared each with its original:
- A 100 Hz ripple in the envelope above 3 kHz, four times the 25 fps latent rate. It shows on sustained dense material: +6.5 dB over its neighbourhood on a duduk and +7.2 dB on an orchestra, where the originals sit at +0.6 and +0.2. YuE2-Vae-legacy and the community merge Mothersuperior/YuE2-Vae-merge-0.666 (two thirds Standard, one third Legacy) carry the same ripple there.
- Lines at 6 and 12 kHz, a quarter and a half of Nyquist, where the upsampling stitches its bands. At 6 kHz: +20.6 dB over the local floor on a studio guitar recording, whose original has +5.5. At 12 kHz: +12.9 dB on speech (original β0.9), and +30.0 dB on a shofar's decay, though at β109 dBFS.
What we did. We fine-tuned the decoder only. Your encoder was never run: training used latents it had already made. We grew the decoder from 66.37 M to 200.76 M stored values with residual units of its own form, three at the latent rate before the first block and three more in each of the six blocks. Each new unit started at zero output, so at step 0 the decoder equalled Standard to the sample. The losses are spectral and envelope terms, including one locked to the latent frame grid at 25/50/75/100 Hz, plus MPD and multi-scale STFT discriminators. The released v1 saw 262.2 hours of 1.28 s crops in 36.1 GPU-hours on one RTX 3090.
Against the originals, as means over seven music clips, in dB:
| Standard | Legacy | Merge 0.666 | x3 v1 | x3 v2 | |
|---|---|---|---|---|---|
| 100 Hz ripple, excess | +2.71 | +3.58 | +3.02 | +0.60 | +0.36 |
| 6 kHz line, excess | +3.59 | +5.03 | +3.69 | +0.98 | +0.79 |
| 12 kHz line, excess | +6.96 | +9.74 | +7.51 | +2.16 | +2.13 |
| SI-SDR | 8.54 | 8.43 | 8.53 | 7.85 | 7.88 |
The costs.
- SI-SDR is lower on every clip, and the floor between harmonics is no better.
- A decode takes about twice the time, and the weights are three times the size.
- A tile needs 60 latent frames of context on each side instead of 12.
- On 40 generated songs the seam did not vanish: it narrowed into a line 1.7β2.2 dB taller than Standard's, inside a notch 4.4β5.0 dB deeper.
In your own PyTorch code, the kit's object speaks your decode API:
import os, sys
from huggingface_hub import hf_hub_download
repo = "goldhub/YuE2-RS-Finetunes"
dec_path = hf_hub_download(repo, "VAE-RS/YuE2-VAE-RS-x3_v1.safetensors")
sys.path.insert(0, os.path.dirname(hf_hub_download(repo, "VAE-RS/kit/yue2_vae_rs.py")))
from yue2_vae_rs import load_decoder
dec = load_decoder(dec_path, device="cuda") # float32 only; TF32 is switched off around the decode
audio = dec.decode_tiled(latents) # [B, 64, T] -> [B, 2, 1920*T], exact 60-frame halo
Links:
- the weights
- the study, with every figure, hour and loss (its HTML in the repo)
- the integration guide
- our page, with 40 generated songs decoded by Standard, v1 and v2 from the same latents
The weights are under CC BY-NC 4.0, as yours are, and the code is under Apache-2.0. The numbers say where to listen, and the listener judges what is heard there. Corrections are welcome.
Viktor Zhuromskyy and Claude (Anthropic), Ruach Studio