clapkit T1: LAION CLAP both encoders, fp16 (spectrogram-input audio + RoBERTa text), pre-norm

CHECKSUMS.sha256

@@ -0,0 +1,16 @@
+29fbf161ab063c891080f328de0ee4ed80fdbaef4b5d36c4086c38da582aa7c4  ./clap_audio.mlmodelc/analytics/coremldata.bin
+652c4652d19b4a7e926468e14582442f047389145576827068f6ae47f97ebb3e  ./clap_audio.mlmodelc/coremldata.bin
+44ea8733243b41a005a6aa25144fbde165c5e3f80ede79889f2d039da6a65ec4  ./clap_audio.mlmodelc/metadata.json
+1ecf76edf7846153623485a98c0a3d047e660cc68ee10a6d21a39664d309db52  ./clap_audio.mlmodelc/model.mil
+723fe6aab7c4af1c671a210a35c289c67763bc6a7532b9df155a0c3fc0c3c9d7  ./clap_audio.mlmodelc/weights/weight.bin
+60ee442fc798667c56c884b05de76b8e85bf192ee18f7faa09b866e05c32460c  ./clap_audio.mlpackage/Data/com.apple.CoreML/model.mlmodel
+723fe6aab7c4af1c671a210a35c289c67763bc6a7532b9df155a0c3fc0c3c9d7  ./clap_audio.mlpackage/Data/com.apple.CoreML/weights/weight.bin
+0ff7841663a6ffb424d0eebe85fcefbe2e0e5ed32ce70ba78a76e774c8d27fa6  ./clap_audio.mlpackage/Manifest.json
+6ce0a63b5bf13fc8f60fb1b8956373a2317d0a7349a0b6899ce5030eb9f8aef6  ./clap_text.mlmodelc/analytics/coremldata.bin
+edbf40e51518ad8f65a1d070f4c0e9b87fea644e6d69eb9eb55652e9fc697885  ./clap_text.mlmodelc/coremldata.bin
+8592b665a1e5a2d9b9078917b3dc9aedb0eb3fa8b8b4ae4630ee78b953082a7c  ./clap_text.mlmodelc/metadata.json
+0ec4d567c8d26a1aac4e161b1bdea6f9cd36441ec9e1da51fa5d28d79c22b744  ./clap_text.mlmodelc/model.mil
+7f4e15e9ccb0ffbc2341eec286e9d9934d3d3d8d6465dfddebed248bddc0e3dd  ./clap_text.mlmodelc/weights/weight.bin
+6c68d30cf088f657428175055e25e01552c5af56769b119dc25b7463d7135ade  ./clap_text.mlpackage/Data/com.apple.CoreML/model.mlmodel
+7f4e15e9ccb0ffbc2341eec286e9d9934d3d3d8d6465dfddebed248bddc0e3dd  ./clap_text.mlpackage/Data/com.apple.CoreML/weights/weight.bin
+ff9a1b297cd1593982aff0eb36cf867f0cbb9af67eac749c0fe7545cd13450dc  ./clap_text.mlpackage/Manifest.json

README.md

@@ -0,0 +1,120 @@
+---
+license: cc-by-4.0
+tags:
+  - coreml
+  - clap
+  - audio-text
+  - htsat
+  - roberta
+  - zero-shot-audio-classification
+  - apple-neural-engine
+library_name: coremlit
+---
+
+# clapkit-coreml — fp16-safe CoreML conversions of LAION CLAP (both encoders)
+
+CoreML conversions of **[laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)**
+— both towers of the CLAP audio↔text model — converted from the PyTorch source
+(`transformers` `ClapModel`) with the campaign's **fp16-survivable numerical guard
+discipline**, so they are correct on the Apple GPU and Neural Engine, not just CPU.
+
+Consumed by the `clapkit` crate of the
+[coremlit](https://github.com/findit-studio/coremlit) workspace (Rust, sync,
+sans-I/O). Each graph emits the final **512-d joint-space embedding pre-L2-norm**;
+the caller performs the L2 normalization (this keeps the fp16 `rsqrt` guard class
+out of the graph entirely).
+
+## Contents
+
+| artifact | form | what it is |
+|---|---|---|
+| `clap_audio.{mlpackage,mlmodelc}` | fp16 | HTSAT audio tower + `audio_projection` (spectrogram-input) |
+| `clap_text.{mlpackage,mlmodelc}` | fp16 | RoBERTa text tower + `text_projection` |
+
+`.mlpackage` is the canonical distributable; `.mlmodelc` is the compiled form the
+test suite loads directly. `CHECKSUMS.sha256` covers every file.
+
+### I/O contract (pinned from the artifact metadata)
+
+| encoder | input(s) | output |
+|---|---|---|
+| audio | `input_features` fp32 `[1, 1, 1001, 64]` (log-mel) | `audio_embeds` fp32 `[1, 512]` (pre-norm) |
+| text | `input_ids` int32 `[1, 512]`, `attention_mask` int32 `[1, 512]` | `text_embeds` fp32 `[1, 512]` (pre-norm) |
+
+- **Audio is 48 kHz mono**, a documented deviation from the workspace's 16 kHz
+  convention (CLAP's native rate). One inference = one fixed 480,000-sample (10 s)
+  window → the `[1, 1, 1001, 64]` mel.
+- **Text length is fixed at 512** (the model's max). Padding a shorter prompt to
+  512 with the attention mask reproduces the natural-length embedding exactly
+  (verified cosine `1.0`), because RoBERTa derives positions from `input_ids` and
+  the mask zeroes the padding.
+
+## The mel frontend lives in Rust (spectrogram-input), by measurement
+
+The audio graph takes the **log-mel spectrogram**, not raw audio. Riding the
+STFT + Slaney-mel + `power_to_dB` frontend *inside* the graph was attempted and
+rejected on measurement:
+
+- a faithful in-graph STFT reproduces HF's `ClapFeatureExtractor` mel only in
+  **float64** (the reference promotes to f64; an in-graph f32 STFT lands **0.55–0.90
+  cosine** away from the correct path end-to-end) — f64 is hostile to an fp16 ANE
+  graph;
+- the `power_to_dB` floor `amin = 1e-10` is **1680× below fp16's smallest subnormal
+  (`2^-24`)** — exactly the vanishing-guard class this campaign guards against.
+
+So the mel is a **Rust port validated bit-for-bit against textclap's `mel.rs`**
+(the frontend oracle). Its parameters: `n_fft = 1024`, `hop = 480`, `n_mels = 64`,
+`fmin = 50`, `fmax = 14000`, periodic Hann, Slaney scale + Slaney norm,
+`center=True` reflect padding, `10·log10(max(·, 1e-10))`, HTSAT input-norm `none`,
+time-major `[1001, 64]` → reshaped to `[1, 1, 1001, 64]`.
+
+## Verification (measured, this conversion)
+
+- **MIL-level fp16-guard audit CLEAN** — 55 guard sites total (audio 30, text 25),
+  every effective floor ≥ `2^-24`; no decomposed `softmax→log`; no unresolved
+  guards. The text tower's LayerNorm `eps = 1e-12` (below fp16 subnormal) is raised
+  to `2^-24` by the fp16 conversion; audio LayerNorm/BatchNorm `eps ≈ 1e-5` survive
+  as-is. Normalization is out of the graph, so there is no `rsqrt`/`real_div` guard.
+- **PyTorch fp32 vs CoreML fp32 (CPU)** on real inputs: worst cosine **1.00000000**
+  (audio, 10 real clips: music / speech / SFX / ambient) and **1.00000000** (text,
+  12 varied prompts). The graph conversion is faithful; the HTSAT `reshape_mel2img`
+  bicubic resize is reproduced by an exact baked-constant matmul.
+- **CoreML fp16 vs fp32**, worst cosine per compute unit:
+
+  | encoder | ALL | CPU+GPU | CPU |
+  |---|---|---|---|
+  | audio | 0.99999573 | 0.99999390 | 0.99996626 |
+  | text  | 0.99994979 | 0.99999733 | 0.99987553 |
+
+  fp16 is clean on every placement — **fp16 is shipped** (no fp32 fallback needed).
+
+## Placement guidance (measured, never marketed)
+
+- **text**: compiles for and runs on the ANE/GPU/CPU; fp16-clean on all.
+- **audio**: the HTSAT graph currently **fails ANE compilation** (`ANECCompile()`)
+  and falls back to GPU/CPU — still fp16-clean there. Consumers should not assume
+  ANE placement for the audio tower; `clapkit` selects the compute unit and does
+  not assert ANE.
+
+## Toolchain (pinned)
+
+coremltools 9.0 · torch 2.5.1 · transformers 5.14.0 · numpy 1.26.4 · python 3.11.15.
+Source revision: `laion/clap-htsat-unfused@8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a`.
+
+## Upstream provenance & licensing
+
+These are **derivative conversions**: the same LAION CLAP weights, re-emitted as
+CoreML graphs (projection heads in-graph, L2-norm out, mel frontend externalized).
+
+| component | upstream | license |
+|---|---|---|
+| both encoders | [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused) | see note |
+| tokenizer (used by `clapkit`, not redistributed here) | [Xenova/clap-htsat-unfused](https://huggingface.co/Xenova/clap-htsat-unfused) `@c28f2883…` (`tokenizer.json` sha `dc239041…`) | derived from the RoBERTa tokenizer |
+
+> **License note.** Attribution to `laion/clap-htsat-unfused` is provided as
+> required. The LAION CLAP checkpoints are commonly attributed as **CC-BY-4.0**
+> (the position taken by the consuming project and reflected in the front-matter),
+> while the current upstream HF model card declares **apache-2.0**. Both licenses
+> require attribution, which this repository provides; downstream users should
+> honor whichever the upstream author intends. If you are the upstream author and
+> want changes to this redistribution, please open a discussion.

clap_audio.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29fbf161ab063c891080f328de0ee4ed80fdbaef4b5d36c4086c38da582aa7c4
+size 243

clap_audio.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:652c4652d19b4a7e926468e14582442f047389145576827068f6ae47f97ebb3e
+size 589

clap_audio.mlmodelc/metadata.json

@@ -0,0 +1,79 @@
+[
+  {
+    "metadataOutputVersion" : "3.0",
+    "shortDescription" : "CLAP audio encoder (spectrogram-input): mel [1,1,1001,64] -> 512-d joint embedding, L2-norm applied by the caller",
+    "outputSchema" : [
+      {
+        "hasShapeFlexibility" : "0",
+        "isOptional" : "0",
+        "dataType" : "Float32",
+        "formattedType" : "MultiArray (Float32 1 × 512)",
+        "shortDescription" : "",
+        "shape" : "[1, 512]",
+        "name" : "audio_embeds",
+        "type" : "MultiArray"
+      }
+    ],
+    "storagePrecision" : "Float16",
+    "modelParameters" : [
+
+    ],
+    "author" : "clapkit T1: laion\/clap-htsat-unfused audio tower (HTSAT + audio_projection), pre-norm",
+    "specificationVersion" : 8,
+    "mlProgramOperationTypeHistogram" : {
+      "Ios17.mul" : 12,
+      "Ios17.linear" : 77,
+      "Ios17.transpose" : 79,
+      "Ios17.matmul" : 25,
+      "Ios17.conv" : 1,
+      "Ios17.gather" : 20,
+      "Ios17.add" : 41,
+      "Ios17.sliceByIndex" : 18,
+      "Ios17.batchNorm" : 1,
+      "Ios17.concat" : 4,
+      "Ios16.softmax" : 12,
+      "Ios16.gelu" : 12,
+      "Ios16.reduceMean" : 1,
+      "Ios16.relu" : 1,
+      "Ios17.reshape" : 130,
+      "Ios17.layerNorm" : 29,
+      "Ios17.cast" : 2
+    },
+    "computePrecision" : "Mixed (Float16, Float32, Int32, UInt16)",
+    "isUpdatable" : "0",
+    "stateSchema" : [
+
+    ],
+    "availability" : {
+      "macOS" : "14.0",
+      "tvOS" : "17.0",
+      "visionOS" : "1.0",
+      "watchOS" : "10.0",
+      "iOS" : "17.0",
+      "macCatalyst" : "17.0"
+    },
+    "modelType" : {
+      "name" : "MLModelType_mlProgram"
+    },
+    "inputSchema" : [
+      {
+        "hasShapeFlexibility" : "0",
+        "isOptional" : "0",
+        "dataType" : "Float32",
+        "formattedType" : "MultiArray (Float32 1 × 1 × 1001 × 64)",
+        "shortDescription" : "",
+        "shape" : "[1, 1, 1001, 64]",
+        "name" : "input_features",
+        "type" : "MultiArray"
+      }
+    ],
+    "userDefinedMetadata" : {
+      "com.github.apple.coremltools.conversion_date" : "2026-07-18",
+      "com.github.apple.coremltools.source" : "torch==2.5.1",
+      "com.github.apple.coremltools.version" : "9.0",
+      "com.github.apple.coremltools.source_dialect" : "TorchScript"
+    },
+    "generatedClassName" : "clap_audio",
+    "method" : "predict"
+  }
+]

clap_audio.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:723fe6aab7c4af1c671a210a35c289c67763bc6a7532b9df155a0c3fc0c3c9d7
+size 60562432

clap_audio.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60ee442fc798667c56c884b05de76b8e85bf192ee18f7faa09b866e05c32460c
+size 225224

clap_audio.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:723fe6aab7c4af1c671a210a35c289c67763bc6a7532b9df155a0c3fc0c3c9d7
+size 60562432

clap_audio.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "2B049E29-0466-4DD5-9DC7-673A6A97CE6F": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "DC7C3634-A1E3-42D3-9D0C-22CFE01170D0": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "2B049E29-0466-4DD5-9DC7-673A6A97CE6F"
+}

clap_text.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ce0a63b5bf13fc8f60fb1b8956373a2317d0a7349a0b6899ce5030eb9f8aef6
+size 243

clap_text.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edbf40e51518ad8f65a1d070f4c0e9b87fea644e6d69eb9eb55652e9fc697885
+size 606

clap_text.mlmodelc/metadata.json

@@ -0,0 +1,93 @@
+[
+  {
+    "metadataOutputVersion" : "3.0",
+    "shortDescription" : "CLAP text encoder: input_ids\/attention_mask [1,512] -> 512-d joint embedding, L2-norm applied by the caller",
+    "outputSchema" : [
+      {
+        "hasShapeFlexibility" : "0",
+        "isOptional" : "0",
+        "dataType" : "Float32",
+        "formattedType" : "MultiArray (Float32 1 × 512)",
+        "shortDescription" : "",
+        "shape" : "[1, 512]",
+        "name" : "text_embeds",
+        "type" : "MultiArray"
+      }
+    ],
+    "storagePrecision" : "Float16",
+    "modelParameters" : [
+
+    ],
+    "author" : "clapkit T1: laion\/clap-htsat-unfused text tower (RoBERTa + text_projection), pre-norm",
+    "specificationVersion" : 8,
+    "mlProgramOperationTypeHistogram" : {
+      "Ios17.notEqual" : 1,
+      "Ios17.reshape" : 48,
+      "Ios17.logicalAnd" : 1,
+      "Ios16.softmax" : 12,
+      "Ios17.matmul" : 24,
+      "Ios17.transpose" : 48,
+      "Select" : 2,
+      "Ios17.add" : 40,
+      "Ios17.sliceByIndex" : 1,
+      "Ios17.gatherAlongAxis" : 1,
+      "Ios17.gatherNd" : 1,
+      "Ios17.gather" : 3,
+      "Ios17.layerNorm" : 25,
+      "Ios17.tanh" : 1,
+      "Ios16.gelu" : 12,
+      "Ios17.cast" : 6,
+      "Ios16.relu" : 1,
+      "Ios17.linear" : 75,
+      "Ios17.greaterEqual" : 1,
+      "Ios16.cumsum" : 1,
+      "Ios17.mul" : 13
+    },
+    "computePrecision" : "Mixed (Float16, Float32, Int32, Int8, UInt16)",
+    "isUpdatable" : "0",
+    "stateSchema" : [
+
+    ],
+    "availability" : {
+      "macOS" : "14.0",
+      "tvOS" : "17.0",
+      "visionOS" : "1.0",
+      "watchOS" : "10.0",
+      "iOS" : "17.0",
+      "macCatalyst" : "17.0"
+    },
+    "modelType" : {
+      "name" : "MLModelType_mlProgram"
+    },
+    "inputSchema" : [
+      {
+        "hasShapeFlexibility" : "0",
+        "isOptional" : "0",
+        "dataType" : "Int32",
+        "formattedType" : "MultiArray (Int32 1 × 512)",
+        "shortDescription" : "",
+        "shape" : "[1, 512]",
+        "name" : "input_ids",
+        "type" : "MultiArray"
+      },
+      {
+        "hasShapeFlexibility" : "0",
+        "isOptional" : "0",
+        "dataType" : "Int32",
+        "formattedType" : "MultiArray (Int32 1 × 512)",
+        "shortDescription" : "",
+        "shape" : "[1, 512]",
+        "name" : "attention_mask",
+        "type" : "MultiArray"
+      }
+    ],
+    "userDefinedMetadata" : {
+      "com.github.apple.coremltools.conversion_date" : "2026-07-18",
+      "com.github.apple.coremltools.source" : "torch==2.5.1",
+      "com.github.apple.coremltools.version" : "9.0",
+      "com.github.apple.coremltools.source_dialect" : "TorchScript"
+    },
+    "generatedClassName" : "clap_text",
+    "method" : "predict"
+  }
+]

clap_text.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f4e15e9ccb0ffbc2341eec286e9d9934d3d3d8d6465dfddebed248bddc0e3dd
+size 251665792

clap_text.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c68d30cf088f657428175055e25e01552c5af56769b119dc25b7463d7135ade
+size 147601

clap_text.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f4e15e9ccb0ffbc2341eec286e9d9934d3d3d8d6465dfddebed248bddc0e3dd
+size 251665792

clap_text.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "294DDB9A-5C5F-4DEA-B14F-6614D82E2A56": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        },
+        "D6451709-22A7-4307-8C0F-C6BA31754263": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        }
+    },
+    "rootModelIdentifier": "D6451709-22A7-4307-8C0F-C6BA31754263"
+}