| --- |
| license: apache-2.0 |
| tags: |
| - audio |
| - audio-embedding |
| - clap |
| - laion-clap |
| - coreml |
| - on-device |
| library_name: coreml |
| base_model: laion/larger_clap_music |
| --- |
| |
| # LAION-CLAP (Music) → Core ML |
|
|
| On-device audio-embedding model for Apple Silicon Macs. Converted from |
| [laion/larger_clap_music](https://huggingface.co/laion/larger_clap_music) |
| (HTSAT-base audio encoder + audio projection) to a self-contained Core ML |
| `.mlpackage`, int8-quantized. |
|
|
| Used by [Gridshift](https://gridshift.studio) for sample similarity search — |
| "find samples that sound like this kick" — and (in a later phase) text-to-sample |
| retrieval. |
|
|
| ## Input / output contract |
|
|
| ``` |
| audio: fp32 tensor [1, 480000] 10 s mono @ 48 kHz, peak-normalized to [-1, 1] |
| embedding: fp32 tensor [1, 512] L2-normalized, cosine = dot product |
| ``` |
|
|
| Mel-spectrogram preprocessing is baked into the model graph (via |
| [convmelspec](https://github.com/adobe-research/convmelspec) STFT), so the |
| client does zero DSP preprocessing — just supply raw audio samples. |
|
|
| ## Accuracy vs PyTorch reference (5 synthetic signals) |
|
|
| | signal | cos(ref, coreml) | |
| |---------------|-----------------:| |
| | sine 440 Hz | 0.99851 | |
| | sine 220 Hz | 0.99746 | |
| | white noise | 0.99977 | |
| | silence | 0.99986 | |
| | clipped noise | 0.99977 | |
|
|
| Pairwise distance structure between signals is preserved with max drift |
| 0.004 (threshold ≤ 0.02), so relative similarity rankings between samples |
| remain intact through the int8 quantization. |
|
|
| ## Handling audio of different lengths |
|
|
| The Core ML graph is shape-rigid at 480 000 samples (10 s). The client is |
| expected to preprocess: |
|
|
| - **≤ 10 s**: zero-pad on the right. |
| - **< 200 ms** (short one-shots): repeat-pad to ~500 ms, then zero-pad. Prevents |
| padding from dominating the embedding. |
| - **> 10 s** (loops): sliding-window 3× with 50 % overlap, then mean-pool the |
| three 512-d embeddings. Re-normalize to unit length. |
|
|
| ## License and attribution |
|
|
| Apache-2.0, inherited from upstream LAION-CLAP. Please cite: |
|
|
| ``` |
| Wu et al., "Large-scale Contrastive Language-Audio Pretraining with Feature |
| Fusion and Keyword-to-Caption Augmentation", 2022. |
| https://arxiv.org/abs/2211.06687 |
| ``` |
|
|
| ## Conversion details |
|
|
| Conversion was done with the script at |
| `app/ml/clap/convert_to_coreml.py` in the Gridshift source tree, using: |
|
|
| - PyTorch 2.11 + torch.export |
| - coremltools 9.0 MLProgram backend |
| - int8 symmetric weight quantization |
| - bicubic → bilinear interp swap for Core ML compat (minimal accuracy impact) |
| - CLAP window-size patch for `torch.jit.is_tracing` branch divergence |
| - Fixed input shape [1, 480000] baked into the graph |
|
|
| Target: macOS 14+. |
|
|
