README.md

---
license: cc-by-4.0
tags:
  - coreml
  - clap
  - audio-text
  - htsat
  - roberta
  - zero-shot-audio-classification
  - apple-neural-engine
library_name: coremlit
---

# clapkit-coreml — fp16-safe CoreML conversions of LAION CLAP (both encoders)

CoreML conversions of **[laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)**
— both towers of the CLAP audio↔text model — converted from the PyTorch source
(`transformers` `ClapModel`) with the campaign's **fp16-survivable numerical guard
discipline**, so they are correct on the Apple GPU and Neural Engine, not just CPU.

Consumed by the `clapkit` crate of the
[coremlit](https://github.com/findit-studio/coremlit) workspace (Rust, sync,
sans-I/O). Each graph emits the final **512-d joint-space embedding pre-L2-norm**;
the caller performs the L2 normalization (this keeps the fp16 `rsqrt` guard class
out of the graph entirely).

## Contents

| artifact | form | what it is |
|---|---|---|
| `clap_audio.{mlpackage,mlmodelc}` | fp16 | HTSAT audio tower + `audio_projection` (spectrogram-input) |
| `clap_text.{mlpackage,mlmodelc}` | fp16 | RoBERTa text tower + `text_projection` |
| `clap_audio_int8.{mlpackage,mlmodelc}` | int8 | audio tower — 8-bit k-means-palettized weights (weight-only; activations fp16) |
| `clap_text_int8.{mlpackage,mlmodelc}` | int8 | text tower — 8-bit k-means-palettized weights (weight-only; activations fp16) |

`.mlpackage` is the canonical distributable; `.mlmodelc` is the compiled form the
test suite loads directly. `CHECKSUMS.sha256` covers every file. The `_int8`
siblings are an **optional ~2× smaller** tier (see "int8 weight-only tier" below);
the fp16 graphs remain the default.

### I/O contract (pinned from the artifact metadata)

| encoder | input(s) | output |
|---|---|---|
| audio | `input_features` fp32 `[1, 1, 1001, 64]` (log-mel) | `audio_embeds` fp32 `[1, 512]` (pre-norm) |
| text | `input_ids` int32 `[1, 512]`, `attention_mask` int32 `[1, 512]` | `text_embeds` fp32 `[1, 512]` (pre-norm) |

- **Audio is 48 kHz mono**, a documented deviation from the workspace's 16 kHz
  convention (CLAP's native rate). One inference = one fixed 480,000-sample (10 s)
  window → the `[1, 1, 1001, 64]` mel.
- **Text length is fixed at 512** (the model's max). Padding a shorter prompt to
  512 with the attention mask reproduces the natural-length embedding exactly
  (verified cosine `1.0`), because RoBERTa derives positions from `input_ids` and
  the mask zeroes the padding.

## The mel frontend lives in Rust (spectrogram-input), by measurement

The audio graph takes the **log-mel spectrogram**, not raw audio. Riding the
STFT + Slaney-mel + `power_to_dB` frontend *inside* the graph was attempted and
rejected on measurement:

- a faithful in-graph STFT reproduces HF's `ClapFeatureExtractor` mel only in
  **float64** (the reference promotes to f64; an in-graph f32 STFT lands **0.55–0.90
  cosine** away from the correct path end-to-end) — f64 is hostile to an fp16 ANE
  graph;
- the `power_to_dB` floor `amin = 1e-10` is **1680× below fp16's smallest subnormal
  (`2^-24`)** — exactly the vanishing-guard class this campaign guards against.

So the mel is a **Rust port validated bit-for-bit against textclap's `mel.rs`**
(the frontend oracle). Its parameters: `n_fft = 1024`, `hop = 480`, `n_mels = 64`,
`fmin = 50`, `fmax = 14000`, periodic Hann, Slaney scale + Slaney norm,
`center=True` reflect padding, `10·log10(max(·, 1e-10))`, HTSAT input-norm `none`,
time-major `[1001, 64]` → reshaped to `[1, 1, 1001, 64]`.

## Verification (measured, this conversion)

- **MIL-level fp16-guard audit CLEAN** — 55 guard sites total (audio 30, text 25),
  every effective floor ≥ `2^-24`; no decomposed `softmax→log`; no unresolved
  guards. The text tower's LayerNorm `eps = 1e-12` (below fp16 subnormal) is raised
  to `2^-24` by the fp16 conversion; audio LayerNorm/BatchNorm `eps ≈ 1e-5` survive
  as-is. Normalization is out of the graph, so there is no `rsqrt`/`real_div` guard.
- **PyTorch fp32 vs CoreML fp32 (CPU)** on real inputs: worst cosine **1.00000000**
  (audio, 10 real clips: music / speech / SFX / ambient) and **1.00000000** (text,
  12 varied prompts). The graph conversion is faithful; the HTSAT `reshape_mel2img`
  bicubic resize is reproduced by an exact baked-constant matmul.
- **CoreML fp16 vs fp32**, worst cosine per compute unit:

  | encoder | ALL | CPU+GPU | CPU |
  |---|---|---|---|
  | audio | 0.99999573 | 0.99999390 | 0.99996626 |
  | text  | 0.99994979 | 0.99999733 | 0.99987553 |

  fp16 is clean on every placement — **fp16 is shipped** (no fp32 fallback needed).

## int8 weight-only tier (optional, ~2× smaller)

Alongside the fp16 graphs this repo ships an **8-bit weight-only** tier of both
towers — `clap_audio_int8` / `clap_text_int8`. Weights are stored 8-bit and
dequantized to fp16 at runtime; **activations stay fp16** (identical graph maths,
only the constant weights are compressed). Produced from the shipped fp16
`.mlpackage`s by coremltools post-training compression — no reconversion, same
source weights.

- **Method: 8-bit k-means palettization, per-tensor**
  (`coremltools.optimize.coreml.palettize_weights`, `nbits=8`, `mode="kmeans"`,
  `weight_threshold=2048`). Chosen over linear-symmetric int8
  (`linear_quantize_weights`) **by measurement**: on the text tower linear int8
  costs up to **0.43%** cosine (worst 0.99571 vs fp16) while k-means costs
  **0.13%** (worst 0.99873); on audio the two are close (0.99965 vs 0.99971).
  k-means wins on both towers and is decisive on text, so it is the shipped `_int8`
  tier. Tiny bias / LayerNorm-gain tensors (< 2048 elements) stay fp16; one RoBERTa
  attention-mask fill constant (±inf) is likewise left uncompressed.
- **int8-vs-fp16 embedding cosine** (identical inputs, CPU; the conversion
  verification set — 10 real clips + 12 varied prompts including CJK — plus the
  committed `golden_mel` fixture):

  | tower | worst | mean |
  |---|---|---|
  | audio | 0.99970584 | 0.99982787 |
  | text  | 0.99873250 | 0.99965990 |

  Well inside a 0.5% cosine budget on both towers. (int8-vs-fp32-**source** worst:
  audio 0.99967, text 0.99897 — barely further from the PyTorch source than fp16
  itself, which is 0.99997 / 0.99988.)
- **Zero-shot ranking unchanged.** The end-to-end gate (a ~192 s speech clip → 20
  windows → aggregate → 4-anchor zero-shot score) returns the **identical ranking**
  on int8 as on fp16: top label "This is a sound of a person speaking" at logit
  **8.88** (fp16 8.82), same order for music / dog / rain, same margins.
- **Placement unchanged.** Cross-compute-unit agreement holds (audio 0.99998, text
  0.99994 — ≥ the 0.9999 band on All / CPU+NE / CPU+GPU / CPU). Audio still fails
  `ANECCompile()` and falls back to GPU/CPU; text still compiles for the ANE — the
  palettized weights change neither.
- **Size** (`weight.bin` bytes, fp16 → int8):

  | tower | fp16 | int8 | ratio |
  |---|---|---|---|
  | audio | 60,562,432 | 30,409,920 | 1.99× |
  | text  | 251,665,792 | 125,746,688 | 2.00× |

  Compiled `.mlmodelc`: audio 58 MB → 29 MB, text 240 MB → 120 MB.

Same toolchain pin as the fp16 conversion (coremltools 9.0; scikit-learn provides
the k-means step). The int8 tier is a pure recompression of the pinned fp16
artifacts, so its provenance and licensing are exactly those of the fp16 graphs
below.

## Placement guidance (measured, never marketed)

- **text**: compiles for and runs on the ANE/GPU/CPU; fp16-clean on all.
- **audio**: the HTSAT graph currently **fails ANE compilation** (`ANECCompile()`)
  and falls back to GPU/CPU — still fp16-clean there. Consumers should not assume
  ANE placement for the audio tower; `clapkit` selects the compute unit and does
  not assert ANE.

## Toolchain (pinned)

coremltools 9.0 · torch 2.5.1 · transformers 5.14.0 · numpy 1.26.4 · python 3.11.15.
Source revision: `laion/clap-htsat-unfused@8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a`.

## Upstream provenance & licensing

These are **derivative conversions**: the same LAION CLAP weights, re-emitted as
CoreML graphs (projection heads in-graph, L2-norm out, mel frontend externalized).

| component | upstream | license |
|---|---|---|
| both encoders | [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused) | see note |
| tokenizer (used by `clapkit`, not redistributed here) | [Xenova/clap-htsat-unfused](https://huggingface.co/Xenova/clap-htsat-unfused) `@c28f2883…` (`tokenizer.json` sha `dc239041…`) | derived from the RoBERTa tokenizer |

> **License note.** Attribution to `laion/clap-htsat-unfused` is provided as
> required. The LAION CLAP checkpoints are commonly attributed as **CC-BY-4.0**
> (the position taken by the consuming project and reflected in the front-matter),
> while the current upstream HF model card declares **apache-2.0**. Both licenses
> require attribution, which this repository provides; downstream users should
> honor whichever the upstream author intends. If you are the upstream author and
> want changes to this redistribution, please open a discussion.