The 100 Hz ripple and the 6/12 kHz seams of the Standard decoder, measured, and a grown decoder trained against them (YuE2-VAE-RS x3)

#2
by goldhub - opened

Hello m-a-p, and thank you for the YuE2 VAE and its open weights. Everything below starts from your decoder, YuE2-Vae, which we call Standard.

What we measured in Standard. We decoded eight unseen 30-second clips whole, in float32, and compared each with its original:

  • A 100 Hz ripple in the envelope above 3 kHz, four times the 25 fps latent rate. It shows on sustained dense material: +6.5 dB over its neighbourhood on a duduk and +7.2 dB on an orchestra, where the originals sit at +0.6 and +0.2. YuE2-Vae-legacy and the community merge Mothersuperior/YuE2-Vae-merge-0.666 (two thirds Standard, one third Legacy) carry the same ripple there.
  • Lines at 6 and 12 kHz, a quarter and a half of Nyquist, where the upsampling stitches its bands. At 6 kHz: +20.6 dB over the local floor on a studio guitar recording, whose original has +5.5. At 12 kHz: +12.9 dB on speech (original βˆ’0.9), and +30.0 dB on a shofar's decay, though at βˆ’109 dBFS.

What we did. We fine-tuned the decoder only. Your encoder was never run: training used latents it had already made. We grew the decoder from 66.37 M to 200.76 M stored values with residual units of its own form, three at the latent rate before the first block and three more in each of the six blocks. Each new unit started at zero output, so at step 0 the decoder equalled Standard to the sample. The losses are spectral and envelope terms, including one locked to the latent frame grid at 25/50/75/100 Hz, plus MPD and multi-scale STFT discriminators. The released v1 saw 262.2 hours of 1.28 s crops in 36.1 GPU-hours on one RTX 3090.

Against the originals, as means over seven music clips, in dB:

Standard Legacy Merge 0.666 x3 v1 x3 v2
100 Hz ripple, excess +2.71 +3.58 +3.02 +0.60 +0.36
6 kHz line, excess +3.59 +5.03 +3.69 +0.98 +0.79
12 kHz line, excess +6.96 +9.74 +7.51 +2.16 +2.13
SI-SDR 8.54 8.43 8.53 7.85 7.88

The costs.

  • SI-SDR is lower on every clip, and the floor between harmonics is no better.
  • A decode takes about twice the time, and the weights are three times the size.
  • A tile needs 60 latent frames of context on each side instead of 12.
  • On 40 generated songs the seam did not vanish: it narrowed into a line 1.7–2.2 dB taller than Standard's, inside a notch 4.4–5.0 dB deeper.

In your own PyTorch code, the kit's object speaks your decode API:

import os, sys
from huggingface_hub import hf_hub_download
repo = "goldhub/YuE2-RS-Finetunes"
dec_path = hf_hub_download(repo, "VAE-RS/YuE2-VAE-RS-x3_v1.safetensors")
sys.path.insert(0, os.path.dirname(hf_hub_download(repo, "VAE-RS/kit/yue2_vae_rs.py")))
from yue2_vae_rs import load_decoder
dec = load_decoder(dec_path, device="cuda")   # float32 only; TF32 is switched off around the decode
audio = dec.decode_tiled(latents)             # [B, 64, T] -> [B, 2, 1920*T], exact 60-frame halo

Links:

The weights are under CC BY-NC 4.0, as yours are, and the code is under Apache-2.0. The numbers say where to listen, and the listener judges what is heard there. Corrections are welcome.

Viktor Zhuromskyy and Claude (Anthropic), Ruach Studio

Sign up or log in to comment