HTDemucs 4-stem, ONNX, fp16, split
Hybrid Transformer Demucs (htdemucs) music source separation (drums, bass, other, vocals) as nine ONNX graphs run in sequence. Prepared from StemSplitio/htdemucs-onnx (htdemucs_fp16weights.onnx) so the model fits comfortably in GPU memory, notably under DirectML.
Why
The source export has about 25,000 nodes, most of them runtime shape arithmetic (Shape, Range, ScatterND, ...). Its intermediate tensors total about 16.5 GB per 7.8 s chunk. DirectML allocated close to that, so the model failed with E_OUTOFMEMORY on a 16 GB card.
How it was made
- Fixed input shape:
mixpinned to[1, 2, 343980](7.8 s, stereo, 44.1 kHz), as the model already requires. - Constant folding: ONNX Runtime basic graph optimisation, saved offline. This brings it to 1,444 nodes, with all shape arithmetic folded away. The output is bit-identical to the source.
- fp16: converted with
onnxconverter-common, with inputs and outputs kept fp32.InstanceNormalization,LayerNormalization,Softmax,ReduceMeanandErfstay fp32. - Split: cut in topological order into 9 parts, each with at most about 2 GB of intermediates. Cut points were chosen where few tensors cross, at most 53 MB handed between parts.
Accuracy
Measured on one 7.8 s music chunk against the source model (fp32 activations):
| stem | SNR vs source |
|---|---|
| drums | 61.2 dB |
| bass | 63.1 dB |
| other | 58.1 dB |
| vocals | 62.4 dB |
On CPU, peak process memory fell from 6.45 GB to 3.88 GB at similar speed.
Files
part0.onnx…part8.onnx: the graphs, 173 MB in total.parts.json: runs in order. Each entry listsfile,inputsandoutputs(tensor names). An entry's inputs are graph inputmixor outputs of earlier parts.SHA256SUMS
Usage
Input mix: float32 [1, 2, 343980]. Output of the last part: float32 [1, 4, 2, 343980], with stems in the order drums, bass, other, vocals. Longer audio is processed in overlapping chunks, as with the source model (its infer.py uses 25% overlap with a linear crossfade).
import json, onnxruntime as ort
parts = json.load(open("parts.json"))
env = {"mix": mix} # float32 [1, 2, 343980]
for p in parts:
s = ort.InferenceSession(p["file"], providers=["DmlExecutionProvider", "CPUExecutionProvider"])
env.update(zip(p["outputs"], s.run(p["outputs"], {k: env[k] for k in p["inputs"]})))
stems = env[parts[-1]["outputs"][0]]
To bound GPU memory across a whole song, load one part at a time: run every chunk through it, release it, then load the next.
License and credit
MIT, as the source export. HTDemucs by Alexandre Défossez, Simon Rouard et al. (facebookresearch/demucs, MIT). ONNX export by StemSplitio.
Model tree for killameep/htdemucs-onnx-fp16-split
Base model
StemSplitio/htdemucs-onnx