Update README.md

README.md

@@ -10,4 +10,131 @@ tags:
   - htsat
   - core-ml
   - apple-silicon
----
+---
+
+# larger-clap-general-coreml
+
+Native Core ML (`.mlpackage`) build of [`laion/larger_clap_general`](https://huggingface.co/laion/larger_clap_general)'s **audio encoder + projection head**. Takes raw 48 kHz mono waveform, returns an L2-normalized 512-d embedding compatible with the original CLAP joint embedding space.
+
+`larger_clap_general` is trained on **general audio, music and speech** — pair it with the upstream text encoder for zero-shot audio classification or open-vocabulary retrieval.
+
+Built because `ort`'s CoreML execution provider can't accelerate HTSAT — reflect-pad, 5-D reshapes, relative-position-bias gather, and dynamic shapes shred the graph into CPU partitions, so the EP "registers" but every node runs on CPU. Loading this `.mlpackage` directly via Core ML (skipping ORT entirely) runs the full graph on the Apple GPU.
+
+## Inputs / Outputs
+
+| | name | shape | dtype | notes |
+|---|---|---|---|---|
+| Input | `audio` | `[1, 480000]` | float32 | 10 s mono @ 48 kHz, peak-normalized to `[-1, 1]` |
+| Output | `embedding` | `[1, 512]` | float32 | L2-normalized; cosine similarity == dot product |
+
+The mel-spectrogram extraction (STFT, Slaney mel filterbank, log) is **baked into the model graph** — you pass raw audio, not features.
+
+## Variable-length audio
+
+The graph has a fixed 10 s input shape. For arbitrary-length audio, recommended recipe:
+
+| Duration | Strategy |
+|---|---|
+| ≤ 10 s | Zero-pad to 480_000 samples, single forward pass. |
+| > 10 s | Sliding 10 s windows with 50 % overlap, embed each window, **mean-pool the embeddings, re-L2-normalize.** |
+
+For very long files cap window count to bound runtime — uniformly spacing N windows across `[0, T-10s]` gives full-file coverage without per-window blow-up.
+
+## Usage
+
+### Swift (Core ML)
+```swift
+import CoreML
+
+let config = MLModelConfiguration()
+config.computeUnits = .cpuAndGPU
+let model = try MLModel(contentsOf: compiledURL, configuration: config)
+
+let audio = try MLMultiArray(shape: [1, 480_000], dataType: .float32)
+// copy your normalized waveform into audio.dataPointer ...
+
+let provider = try MLDictionaryFeatureProvider(dictionary: ["audio": audio])
+let out = try model.prediction(from: provider)
+let embedding = out.featureValue(for: "embedding")!.multiArrayValue!
+```
+
+### Rust (objc2-core-ml)
+See [`scripts/convert-clap-to-coreml.py`](#how-it-was-built) reference repo for a Rust integration that loads this `.mlpackage`, runs sliding-window embedding for arbitrary-length audio, and mean-pools.
+
+### Python (coremltools)
+```python
+import coremltools as ct
+import numpy as np
+
+model = ct.models.MLModel("clap_audio_encoder.mlpackage")
+waveform = np.zeros((1, 480_000), dtype=np.float32)
+out = model.predict({"audio": waveform})
+embedding = out["embedding"]  # shape (1, 512)
+```
+
+## How it was built
+
+`coremltools` 8 + `torch.export` from `laion/larger_clap_general`'s PyTorch weights, then `convert_to="mlprogram"` + int8 linear weight quantization. The conversion is non-trivial — `ct.convert` rejects the model out of the box. Patches applied:
+
+1. **`F.interpolate(mode='bicubic')` → `'bilinear'`** — CoreML's MIL backend lacks bicubic upsampling. Used by HTSAT's positional-embedding resize. Accuracy delta is negligible.
+2. **`torch.jit.is_tracing()` → `True`** — forces HF's CLAP code onto the static-shape path during conversion.
+3. **`ClapAudioLayer.set_shift_and_window_size` → no-op** — the dynamic window adjustment hits a "data-dependent guard" error in `torch.export`. For our fixed `[1, 1, 1001, 64]` input the `__init__` values are already correct, so neutralizing is safe.
+4. **Custom STFT** — `torch.stft`'s op signature drifts across torch versions and the coremltools handler unpacks the wrong arity; implemented as strided conv1d with pre-baked cos/sin Hann bases instead.
+5. **Custom `fmod` MIL lowering** — HTSAT's relative-position arithmetic uses float modulo; coremltools has no built-in handler. Registered as `x - trunc(x/y) * y`.
+6. **`slice_scatter` override** — HTSAT's attention-mask builder generates empty-slice `slice_scatter` calls at deeper Swin stages (e.g. `slice(0, -window_size)` evaluates to `slice(0, 0)`). The built-in handler's shape check rejects these; registered override that no-ops empty slices and reduces non-empty ones to `slice_by_index + concat`.
+
+A full conversion script that applies all six patches is in the reference repo (link below).
+
+## Validation
+
+Cosine similarity vs the PyTorch reference, on random `[1, 480000]` peak-normalized inputs:
+
+| Trial | Cosine |
+|---|---|
+| 1 | 0.999393 |
+| 2 | 0.998725 |
+| 3 | 0.998992 |
+
+The drift is dominated by int8 weight quantization. For full fp32 weights you can re-run the conversion with the int8 step disabled (~3× larger file, ~1.001 cosine on the same inputs).
+
+## Performance
+
+Apple M-series, `MLComputeUnits.cpuAndGPU`:
+
+| | Latency per 10 s window |
+|---|---|
+| Cold start (first forward pass) | ~5 s (Core ML graph compile + GPU upload) |
+| Steady state | ~30 ms |
+
+Compared to running the original `.onnx` via `ort` on Apple Silicon CPU, that's a roughly 10× speedup for the steady state. ANE was not attempted (`MLComputeUnits.all`) — `CPUAndGPU` was the sweet spot during testing; the strictest backend often rejects whole-graph compilation for transformer audio models.
+
+## Limitations
+
+- **Audio encoder only.** The text encoder + logit-scale are not part of this package — use the original `laion/larger_clap_general` for the text path if you need joint-embedding search.
+- **Fixed input shape.** Audio shorter than 10 s must be zero-padded; longer requires the sliding-window recipe above.
+- **int8 quantization.** ~99.9 % cosine is sufficient for retrieval / search use cases; if you're using these embeddings as inputs to downstream training, prefer the fp32 variant.
+
+## Credits
+
+- [LAION](https://laion.ai) for [`larger_clap_general`](https://huggingface.co/laion/larger_clap_general).
+- [gridshiftstudio/clap-music-coreml](https://huggingface.co/gridshiftstudio/clap-music-coreml) for the first public proof that this conversion is viable + the two key patches.
+
+## Citation
+
+If you use this model in your work, please cite the original CLAP paper ([arXiv:2211.06687](https://arxiv.org/abs/2211.06687)):
+
+```bibtex
+@misc{https://doi.org/10.48550/arxiv.2211.06687,
+  doi = {10.48550/ARXIV.2211.06687},
+  url = {https://arxiv.org/abs/2211.06687},
+  author = {Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},
+  title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
+  publisher = {arXiv},
+  year = {2022},
+  copyright = {Creative Commons Attribution 4.0 International}
+}
+```
+
+## License
+
+This artifact inherits the source model's license: **Apache 2.0**.