ytm-vocal-separator
ONNX graphs of the Mel-Band RoFormer vocal separator, exported for ytm (a YouTube Music terminal player) to run with ONNX Runtime for karaoke. Two models share one graph contract:
- ALL vocals: Kim's Mel-Band RoFormer, weights included (fp16 for GPUs, int8 for CPUs).
- LEAD vocal: a weight-less graph of the same architecture for aufr33 & viperx's karaoke model. Its weights are not hosted here. ytm downloads the original checkpoint from its upstream release and binds its tensors into this graph at load time.
Credits and licences
- Weights (ALL): Kim Mel-Band RoFormer by Kimberley Jensen (KimberleyJSN), revision
ac9b0614ab3cd7f77219e18ba494dfd93956c348,MelBandRoformer.ckptsha25687201f4d31afb5bc79993230fc49446918425574db48c01c405e44f365c7559e. MIT licence (the model card'slicense: mit). These files are a format conversion of those weights (fp16 conversion and int8 quantization; no training). - Architecture and code: Mel-Band RoFormer as implemented in
Music-Source-Separation-Training (ZFTurbo, MIT) @
6fdad13686bb7e84d029ec66f9dbe76e65c93ebf, based on lucidrains' BS-RoFormer (MIT). Paper: Wang et al., "Mel-Band RoFormer for Music Source Separation" (2023). - Karaoke model by aufr33 & viperx. Upstream checkpoint
https://github.com/TRvlvr/model_repo/releases/download/all_public_uvr_models/mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt(sha2561de20d459332fe8869aeb01327a31df0032262706e1365114e852dc271779813), config from TRvlvr/application_data @adea29c9fdbd2fa115a208965d00a6f03cdfde96. Its weights are not hosted here:sep-lead-graph-*.onnxhold no weights from it, only the network structure from the MIT code above, with every weight left as a named placeholder.
The MIT licence (for Kim's weights and the MSST code this export is made from):
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Copyright notices: weights © Kimberley Jensen (KimberleyJSN); MSST code © 2024 Roman Solovyev (ZFTurbo); BS-RoFormer code © Phil Wang (lucidrains).
Files
| File | What |
|---|---|
sep-all-fp16.onnx |
ALL, fp16 compute (RMSNorm divisions in fp32, so silence gives no NaN); inputs and outputs fp32. For GPUs. |
sep-all-int8.onnx |
ALL, int8 weights: ONNX Runtime dynamic quantization of every MatMul with a weight, per-tensor QInt8, full 8-bit range (reduce_range=False). For CPUs; exact on CPUs with AVX512-VNNI or AVX-VNNI, while older x86 CPUs without VNNI may saturate in ONNX Runtime's u8s8 kernels (not measured). |
sep-lead-graph-fp32.onnx |
LEAD, fp32, no weights. |
sep-lead-graph-fp16.onnx |
LEAD, fp16 compute (as sep-all-fp16), no weights. |
sep-lead-graph-*.weights.json |
[{"name", "shape", "ckpt_dtype", "graph_dtype"}]: every placeholder, named by the checkpoint's state-dict key. |
ref-input.f32, ref-*.f32 |
One synthetic 8 s stereo chunk (interleaved f32le) and each graph's vocal estimate for it, for tests of ports (LEAD with the upstream checkpoint bound in). |
SHA256SUMS |
sha256 of every file above. |
Exported with opset 17 (IR version 8), torch 2.14.1, onnx 1.23.2, onnxruntime 1.31.0, onnxconverter-common 1.16.0.
Graph contract (all four graphs)
| Name | Type | Shape | |
|---|---|---|---|
| input | spec |
float32 | [1, 4, 1025, 801]: the STFT of one 8 s stereo chunk (352,800 samples at 44.1 kHz), planes (L.re, L.im, R.re, R.im) |
| output | mask |
float32 | [1, 4, 1025, 801]: the complex vocal mask, same planes |
- STFT exactly as
torch.stft(n_fft=2048, hop_length=441, win_length=2048, window=torch.hann_window(2048), center=True, pad_mode="reflect", normalized=False). - Vocals = iSTFT(spec × mask) to the chunk length, with bin 0 zeroed first (
zero_dc). Accompaniment = mix − vocals. - Long audio: 8 s chunks, step 4 s (or 8 s), linear fades of 0.8 s, overlap-add divided by the summed
weights (MSST
demix()), stopping at the first chunk that reaches the end.
Binding the LEAD weights
Every placeholder is an initializer with data_location = EXTERNAL pointing at a file named
bound-at-load-from-upstream-checkpoint that does not exist. Read each tensor named in <graph>.weights.json from the upstream
checkpoint (a PyTorch zip: data.pkl + raw little-endian storages), cast it to graph_dtype (fp32, or
fp16 with round-to-nearest-even), and hand all of them to ONNX Runtime with AddExternalInitializers
(Python: SessionOptions.add_external_initializers(names, ortvalues)) before creating the session.
Without that the graph does not load (the placeholder file does not exist); AddInitializer does not
help, as ONNX Runtime validates external data paths before it applies those. A tensor of another data
type, or a name the graph does not have, is refused by ONNX Runtime too.
The rotary_embed.freqs entries of the checkpoint are not bound: they are the default rotary frequencies,
and the graph holds the cos/sin tables computed from them.
Verification (ytm's export script)
MUSDB18 7-s test previews (50 tracks), museval BSSEval v4 SDR, median over 1-s frames per track, then median over tracks (the SiSEC 2018 aggregation behind the published 12.08):
| Graph | EP | Overlap | Mode | Vocals SDR | Accompaniment SDR |
|---|---|---|---|---|---|
| sep-all-fp32 (work) | cuda | 2 | track | 12.08 | 15.90 |
| sep-all-fp16 | cuda | 2 | track | 12.08 | 15.90 |
| sep-all-int8 | cpu | 2 | track | 11.92 | 15.86 |
| sep-all-fp16 | cuda | 2 | concat | 11.96 | 15.74 |
| sep-all-fp16 | cuda | 1 | concat | 11.77 | 15.70 |
| sep-all-int8 | cpu | 2 | concat | 11.91 | 15.73 |
| sep-all-int8 | cpu | 1 | concat | 11.73 | 15.67 |
| sep-lead-graph-fp16 + ckpt | cuda | 2 | track | 7.14 | 12.68 |
track: each preview separated alone; concat: the 50 previews concatenated (6 min), so chunk seams count. sep-all-fp32 is the conversion source (not published). LEAD is scored against MUSDB's full vocal stem (lead and backing), so its low score is by design: it leaves backing vocals in the accompaniment.
Parity against the PyTorch original (fp32), one 8 s chunk of a song, through the host pipeline:
| Graph | EP | SNR |
|---|---|---|
| sep-all-fp32 (work) | cuda | 72.3 dB |
| sep-all-fp16 | cuda | 64.3 dB |
| sep-all-int8 | cpu | 30.1 dB |
| sep-lead-graph-fp32 + ckpt | cuda | 72.2 dB |
| sep-lead-graph-fp32 + ckpt | cpu | 80.4 dB |
| sep-lead-graph-fp16 + ckpt | cuda | 64.7 dB |
Non-finite output values on digital silence, 1e-6 noise and a half-silent chunk, every graph: 0.