README.md

---
license: apache-2.0
tags:
  - audio
  - audio-to-audio
  - source-separation
  - music-source-separation
  - spleeter
  - onnx
library_name: onnx
pipeline_tag: audio-to-audio
---

# Spleeter 4stems — ONNX

Deezer's [Spleeter](https://github.com/deezer/spleeter) **4stems** — vocals,
drums, bass, other — converted to ONNX.

Deezer ship 4stems as a TensorFlow 1 checkpoint from October 2019.
[sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) ship an excellent ONNX
export of **2stems** and state plainly: *"We only support the `2-stem` model at
present."* This is the missing half.

| file | size | notes |
|---|---|---|
| `{vocals,drums,bass,other}.fp16.onnx` | 19.7 MB each | **use these** — self-contained |
| `{vocals,drums,bass,other}.onnx` | 39.4 MB each | fp32, self-contained |

Each stem is an independent U-Net (9,826,759 params). Every file is
self-contained — no external-data sidecars.

## Verification

| | |
|---|---|
| TF vs PyTorch port, per stem | vocals 2.6e-04 · drums 1.3e-02 · bass 1.1e-03 · other 2.6e-03 |
| `sum(4 stems) − mix` | **−153.1 dB** |
| Inference, all four stems | 1.69 s for 30 s of 44.1 kHz stereo (~18× realtime, CPU) |

The −153.1 dB is the one to trust. Spleeter's ratio masks sum to exactly 1 by
construction, so the stems *must* reconstruct the mix — and they do, to machine
precision. That single assertion tests the STFT, the mask, the band extension
and the iSTFT at once, without anyone listening to anything.

## ⚠️ These models use ELU. This is the whole trick.

If you plan to run these through an existing Spleeter port, read this first.
Spleeter reads its activations from config:

```jsonc
// configs/2stems/base_config.json
"model": { "type": "unet.unet", "params": {} }
//                              ^^ defaults: LeakyReLU(0.2) + ReLU

// configs/4stems/base_config.json
"model": { "type": "unet.unet", "params": {
    "conv_activation": "ELU", "deconv_activation": "ELU" } }
```

2stems and 4stems have **identical architecture and identical weight shapes**,
kernel for kernel. So a 2stems-shaped implementation loads 4stems weights
**without a murmur** and returns garbage. That is why sherpa-onnx only publish
2stems: their `unet.py` hardcodes LeakyReLU/ReLU, and nothing about the shapes
tells you.

It presented as a max error of **945 against a TF output whose entire range was
178** — the error larger than the signal. With ELU: **945 → 2.6e-04**.

The activations are recorded in each file's ONNX metadata (`conv_activation`,
`deconv_activation`) so the next person doesn't have to find this the hard way.

## Interface

```
input   x : float32 [2, num_splits, 512, 1024]   # [channels, splits, frames, bins]
output  y : float32 [2, num_splits, 512, 1024]   # that stem's magnitude estimate
```

`num_splits` is dynamic. The graph takes **magnitudes and returns magnitudes** —
no complex numbers cross the boundary, which is why this ports cleanly where
time-domain models don't. The STFT, the mask and the iSTFT are yours to do.

### The STFT contract

The forward transform must produce the magnitudes the net was trained on:
**periodic Hann, frame 4096, hop 1024**, 44.1 kHz stereo. `np.hanning` is
*symmetric* and is **not** this — it differs by one sample, and that sample is
the difference between matching training-time spectrograms and merely
resembling them.

Feed `abs(stft(x))[..., :1024]` (1024 of 2049 bins), padded and partitioned to
512-frame splits.

### The mask

Spleeter's soft ratio mask, across all four stems:

```python
total = sum(e ** 2 for e in estimates.values()) + 1e-10
mask  = (estimate ** 2 + 1e-10 / 4) / total
```

Apply it to the **original complex STFT** — you keep the original phase, so
there is nothing to reconstruct. You cannot cherry-pick one stem: the
denominator needs all four.

### Band extension — use `average`, not `zeros`

The net models 1024 of 2049 bins (to ~11 kHz). Everything above needs a value.
Spleeter's default `zeros` **discards** it, which is a −23 dB hole in the
reconstruction. `average` carries the per-frame mean up and reconstructs
exactly (the −153.1 dB above is with `average`; with `zeros` it is −23.0 dB).

## Usage

Complete and runnable — the graphs are only half a separator, so here is the
other half.

```python
import numpy as np, onnxruntime as ort, soundfile as sf

N_FFT, HOP, T, F, BINS = 4096, 1024, 512, 1024, 2049
PAD = N_FFT - HOP
W = np.hanning(N_FFT + 1)[:-1]      # PERIODIC. np.hanning(N_FFT) is symmetric
STEMS = ("vocals", "drums", "bass", "other")

def stft(x):
    n = int(np.ceil((PAD + len(x)) / HOP))
    p = np.zeros((n - 1) * HOP + N_FFT)
    p[PAD:PAD + len(x)] = x          # front pad: see note below
    idx = np.arange(N_FFT)[None, :] + HOP * np.arange(n)[:, None]
    return np.fft.rfft(p[idx] * W, axis=-1)

def istft(spec, length):
    frames = np.fft.irfft(spec, n=N_FFT, axis=-1)
    total = (len(spec) - 1) * HOP + N_FFT
    out, wsum = np.zeros(total), np.zeros(total)
    for i in range(len(spec)):
        at = i * HOP
        out[at:at + N_FFT] += frames[i] * W
        wsum[at:at + N_FFT] += W ** 2
    out = np.divide(out, wsum, out=np.zeros_like(out), where=wsum > 1e-8)
    return out[PAD:PAD + length]

wave, sr = sf.read("song.wav", dtype="float64")   # 44.1 kHz stereo
assert sr == 44100 and wave.shape[1] == 2
n = len(wave)

spec = np.stack([stft(wave[:, c]) for c in range(2)])
frames = spec.shape[1]
splits = int(np.ceil(frames / T))
mag = np.zeros((2, splits * T, F), dtype=np.float32)
mag[:, :frames] = np.abs(spec[:, :, :F])
net_in = mag.reshape(2, splits, T, F)

est = {}
for s in STEMS:                       # all four: the mask needs every one
    sess = ort.InferenceSession(f"{s}.fp16.onnx", providers=["CPUExecutionProvider"])
    out = sess.run(["y"], {"x": net_in})[0]
    est[s] = out.reshape(2, -1, F)[:, :frames]

stems = {}
denom = sum(e ** 2 for e in est.values()) + 1e-10
for s, e in est.items():
    mask = (e ** 2 + 1e-10 / len(est)) / denom
    # extend 1024 -> 2049 bins with the per-frame mean ("average", not "zeros")
    tail = np.repeat(mask.mean(axis=-1, keepdims=True), BINS - F, axis=-1)
    full = np.concatenate([mask, tail], axis=-1)
    # applied to the ORIGINAL complex spectrum: the phase is already correct
    stems[s] = np.stack([istft(spec[c] * full[c], n) for c in range(2)], axis=-1)

# The invariant. Check this, not your ears -- see below.
res = 10 * np.log10(np.mean((sum(stems.values()) - wave) ** 2) / np.mean(wave ** 2))
print(f"sum(stems) - mix = {res:.1f} dB")     # -153 dB

for s, y in stems.items():
    sf.write(f"out_{s}.wav", y, sr, subtype="FLOAT")
```

### Check the invariant, not your ears

The four masks sum to 1 by construction, so `sum(stems)` **must** equal the mix.
It does, to **−153.1 dB**. If yours doesn't, your window or your hop is wrong,
and no amount of listening will tell you which — every one of those mistakes
produces audio that sounds approximately right.

Two ways to measure it wrongly, both learned the hard way:

- **Measure it in memory, before writing.** `sf.write` defaults to `PCM_16`, and
  16-bit quantisation of four stems costs ~75 dB on its own — enough to turn
  −153 into −77 and send you hunting a bug that isn't there. Hence
  `subtype="FLOAT"` above.
- **`zeros` is not a bug.** With Spleeter's default `mask_extension`, −23 dB is
  the *correct* answer: that is the >11 kHz band being discarded, exactly as
  asked.

### Why the front pad

`stft` above prepends `N_FFT-HOP` zeros, which Spleeter does not. Alignment is
not part of the contract — the U-Net is convolutional and translation-equivariant
in time — and without the pad, reconstruction is exact in theory and broken in
practice: on the ramp-in a periodic Hann is ~1e-7, so `W²` is ~1e-13, and
dividing by it turns float noise into the one stretch of signal with no
redundancy to spare. Unpadded, the round-trip error is **2.3**. Padded, it is
**1e-15**.

This is also why there is no `WINDOW_COMPENSATION_FACTOR` here. Spleeter's 2/3
constant and its `inverse_stft_window_fn` exist to undo TensorFlow-specific
normalisation; plain weighted overlap-add inverts this forward with no constants
at all.

## Source

Conversion scripts, the full pipeline, the tests, and the debugging trail that
found the ELU:
**[github.com/madewith-bestpractice/spleeter-4stems-onnx](https://github.com/madewith-bestpractice/spleeter-4stems-onnx)**

## License and attribution

**Apache-2.0**, and deliberately the stricter of the two licences in play, since
these artefacts carry both:

- **The weights are Deezer's Spleeter, MIT.** Spleeter's README licenses "the
  code"; the authors' own peer-reviewed JOSS paper (Hennequin, Khlif, Voituret &
  Moussallam, JOSS 5(50):2154, 2020, [doi:10.21105/joss.02154](https://doi.org/10.21105/joss.02154))
  states: *"Spleeter source code and pre-trained models are available on github
  and distributed under a MIT license."* That ambiguity is real and unresolved —
  [deezer/spleeter#898](https://github.com/deezer/spleeter/issues/898) has been
  open and unanswered since 2024-04-26 — so it is stated here rather than
  glossed. Deezer trained on their own licensed catalogue (the "Bean" dataset)
  and released the weights *because* the data could not be released, so unlike
  models trained on MUSDB18 there is no non-commercial dataset term upstream.
- **The graph derives from [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
  (Apache-2.0)**, whose PyTorch reimplementation of Spleeter's U-Net these were
  exported through. Per Apache-2.0 §4(b): that U-Net was **modified** — its two
  activation functions were parameterised instead of hardcoded.

## Credits

- **[Deezer](https://github.com/deezer/spleeter)** — Spleeter, and for training
  on a licensed catalogue and releasing the weights *because* the data couldn't
  be. That decision is the only reason this model is usable rather than merely
  good.
- **[sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)** (Xiaomi, Fangjun
  Kuang) — the TF→PyTorch→ONNX route, the U-Net port, and the TF-vs-torch
  assertion that made the ELU bug findable instead of silent. This is a
  four-stem extension of their work, not a replacement for it.

```bibtex
@article{spleeter2020,
  doi = {10.21105/joss.02154},
  author = {Romain Hennequin and Anis Khlif and Felix Voituret and Manuel Moussallam},
  title = {Spleeter: a fast and efficient music source separation tool with pre-trained models},
  journal = {Journal of Open Source Software},
  volume = {5}, number = {50}, pages = {2154}, year = {2020}
}
```