HTDemucs 4-stem, ONNX, fp16, split

Hybrid Transformer Demucs (htdemucs) music source separation (drums, bass, other, vocals) as nine ONNX graphs run in sequence. Prepared from StemSplitio/htdemucs-onnx (htdemucs_fp16weights.onnx) so the model fits comfortably in GPU memory, notably under DirectML.

Why

The source export has about 25,000 nodes, most of them runtime shape arithmetic (Shape, Range, ScatterND, ...). Its intermediate tensors total about 16.5 GB per 7.8 s chunk. DirectML allocated close to that, so the model failed with E_OUTOFMEMORY on a 16 GB card.

How it was made

  1. Fixed input shape: mix pinned to [1, 2, 343980] (7.8 s, stereo, 44.1 kHz), as the model already requires.
  2. Constant folding: ONNX Runtime basic graph optimisation, saved offline. This brings it to 1,444 nodes, with all shape arithmetic folded away. The output is bit-identical to the source.
  3. fp16: converted with onnxconverter-common, with inputs and outputs kept fp32. InstanceNormalization, LayerNormalization, Softmax, ReduceMean and Erf stay fp32.
  4. Split: cut in topological order into 9 parts, each with at most about 2 GB of intermediates. Cut points were chosen where few tensors cross, at most 53 MB handed between parts.

Accuracy

Measured on one 7.8 s music chunk against the source model (fp32 activations):

stem SNR vs source
drums 61.2 dB
bass 63.1 dB
other 58.1 dB
vocals 62.4 dB

On CPU, peak process memory fell from 6.45 GB to 3.88 GB at similar speed.

Files

  • part0.onnx … part8.onnx: the graphs, 173 MB in total.
  • parts.json: runs in order. Each entry lists file, inputs and outputs (tensor names). An entry's inputs are graph input mix or outputs of earlier parts.
  • SHA256SUMS

Usage

Input mix: float32 [1, 2, 343980]. Output of the last part: float32 [1, 4, 2, 343980], with stems in the order drums, bass, other, vocals. Longer audio is processed in overlapping chunks, as with the source model (its infer.py uses 25% overlap with a linear crossfade).

import json, onnxruntime as ort
parts = json.load(open("parts.json"))
env = {"mix": mix}  # float32 [1, 2, 343980]
for p in parts:
    s = ort.InferenceSession(p["file"], providers=["DmlExecutionProvider", "CPUExecutionProvider"])
    env.update(zip(p["outputs"], s.run(p["outputs"], {k: env[k] for k in p["inputs"]})))
stems = env[parts[-1]["outputs"][0]]

To bound GPU memory across a whole song, load one part at a time: run every chunk through it, release it, then load the next.

License and credit

MIT, as the source export. HTDemucs by Alexandre Défossez, Simon Rouard et al. (facebookresearch/demucs, MIT). ONNX export by StemSplitio.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for killameep/htdemucs-onnx-fp16-split

Quantized
(2)
this model