Mark models as fp16 quantization of Politrees/UVR_resources (base_model)

aa27ab1 verified 9 days ago

5.74 kB

license: mit
base_model:
  - Politrees/UVR_resources
tags:
  - coreml
  - apple-silicon
  - apple-neural-engine
  - audio-source-separation
  - music-source-separation
  - mdx-net
  - uvr
  - vocals
  - instrumental
  - fp16
  - mlprogram
  - on-device
pipeline_tag: audio-to-audio

UVR MDX-Net CoreML — on-device music stem separation for iOS

On-device vocals / instrumental (music stem) separation models — the fp16 quantization of the Ultimate Vocal Remover (UVR) MDX-Net checkpoints converted to CoreML .mlpackage files (fp16, mlprogram) for the Apple Neural Engine / GPU / CPU on iOS and macOS. The source ONNX models are hosted in Politrees/UVR_resources.

Each .mlpackage is the learned core only — the STFT/iSTFT live in your app's DSP, so the whole signal pipeline stays under your control and runs entirely on-device. No cloud, no network.

Models

Model file	Source (UVR MDX-Net)	Stem	Size	`n_fft`	`hop`	`dim_f`
`UVR_MDXNET_9482.mlpackage`	UVR-MDXNET 9482	vocals	~15 MB	4096	1024	2048
`UVR-MDX-NET-Voc_FT.mlpackage`	UVR-MDX-NET Voc FT	vocals	~32 MB	6144	1024	3072
`UVR-MDX-NET-Inst_HQ_3.mlpackage`	UVR-MDX-NET Inst HQ 3	instrumental	~32 MB	6144	1024	3072

All three share the same I/O contract: NCHW, static dim_t = 256, input [1, 4, dim_f, 256] complex-as-channels [L_re, L_im, R_re, R_im] → output of the same shape = the predicted stem's spectrogram. Real fp16 I/O with fp16 compute, mlprogram, minimum_deployment_target = iOS16.

⚠️ Inst HQ 3 inverts the residual. It is architecturally identical to Voc FT but predicts the instrumental, so the free residual is vocals = mix − model(mix) — the opposite polarity from 9482 / Voc FT. A caller that assumes "output = vocals" will get the stems swapped.

Download

# huggingface_hub (Python)
pip install huggingface_hub
python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download("gyoom-sa/UVR-MDX-CoreML", local_dir="./UVR-MDX-CoreML")
PY

# or the CLI
hf download gyoom-sa/UVR-MDX-CoreML --local-dir ./UVR-MDX-CoreML

The .mlpackage folders keep their on-disk structure in this repo, so git clone / hf download yields ready-to-compile packages. Add them to your Xcode target (or call MLModel.compileModel(at:) at runtime) and they compile to .mlmodelc.

Usage (iOS / Swift)

The model operates on a spectrogram, not raw audio. Build the STFT in your app (Apple's Accelerate / vDSP), feed the complex planes to the model, then iSTFT the output. Per model: 9482 n_fft 4096, hop 1024, dim_f 2048; Voc FT / Inst HQ 3 n_fft 6144, hop 1024, dim_f 3072. Use a periodic Hann window, center=True reflect padding, unnormalized STFT, drop the Nyquist bin (dim_f = n_fft/2), and pack planes in order [L_re, L_im, R_re, R_im] with flat row-major index ((plane*dim_f)+bin)*dim_t + frame.

import CoreML

// Accelerator is a LOAD-TIME choice. .all == ANE → GPU → CPU (the baked default).
let config = MLModelConfiguration()
config.computeUnits = .all               // .cpuAndGPU | .cpuOnly also available
let model = try MLModel(contentsOf: compiledURL, configuration: config)

// Per chunk. dim_f = 2048 (9482) or 3072 (Voc FT / Inst HQ 3); dim_t = 256. Real fp16 I/O.
let input = try MLMultiArray(shape: [1, 4, dimF as NSNumber, dimT as NSNumber],
                             dataType: .float16)
let p = input.dataPointer.bindMemory(to: Float16.self, capacity: input.count)
// pack the STFT: p[((plane*dimF)+bin)*dimT + frame] = value   (plane 0..3 = L_re,L_im,R_re,R_im)

let out = try model.prediction(from: MLDictionaryFeatureProvider(dictionary: ["input": input]))
let stem = out.featureValue(for: "output")!.multiArrayValue!     // fp16 [1,4,dim_f,256] → iSTFT

For the stem models, vocals = model(mix); for Inst HQ 3, instrumental = model(mix) and vocals = mix − instrumental. The equal-power "shift trick" denoise (0.5·model(x) − 0.5·model(−x)) is optional.

Notes & limitations

STFT is outside the graph — these are the learned core only; the app must reproduce the DSP contract above, or the stems silently degrade or swap.
fp16 is quality-safe for MDX — activations stay far below the fp16 ceiling, so the ~73 dB output floor is well above audible noise.
Residual polarity differs per model (see the Inst HQ 3 warning).
These are fp16-quantized, on-device artifacts of the UVR MDX-Net models (base model: Politrees/UVR_resources), not models loadable in transformers or via HF Inference. pipeline_tag: audio-to-audio is set for task-based discoverability only.

Reproducibility

The exact conversion pipeline (export_mdx_coreml.py + pinned requirements.txt) and the full technical write-up (precision, compute units, verification SNR, iOS DSP contract) live in the export/ folder of this repo. example.mp3 is provided for a quick smoke test.

License & attribution

The model weights are derived from the Ultimate Vocal Remover (UVR) MDX-Net models by Anjok07 (UVR-MDXNET 9482) and Kimberley Jensen (UVR-MDX-NET Voc FT / Inst HQ 3), distributed under the MIT License via https://github.com/TRvlvr/model_repo and mirrored at Politrees/UVR_resources (this repo's base_model). This repository redistributes fp16-quantized, format-converted (ONNX → CoreML fp16) copies under the same MIT terms. See LICENSE.