---
license: bsd-3-clause
tags:
- audio
- audio-classification
- sample-tagging
- clap
- htsat
- onnx
library_name: onnxruntime
---

# MAGDA Sample Tagger

ONNX exports of [LAION's CLAP HTSAT-unfused model](https://huggingface.co/laion/clap-htsat-unfused)
plus the RoBERTa tokenizer, packaged for the
[MAGDA DAW](https://github.com/Conceptual-Machines/magda-core)'s sample
library (issue #768).

## What's in this repo

| File | Size | SHA-256 |
|------|------|---------|
| `clap_audio.onnx` | 111.8 MB | `3f42f71e555b62709910b6efa66fa5879f00d9571874b12b0fa674f82dbfe332` |
| `clap_text.onnx` | 478.2 MB | `c07b27204836877d5b615c103685b66ea8f21bc6b5b70a572be356125423a8bf` |
| `tokenizer.json` | 3.4 MB | `4fd1d86b4f5b53f40a609fcd11c1f34024b735f870a07439d70202b98493661a` |

- `clap_audio.onnx` — audio encoder. Takes a mono 48 kHz waveform,
  produces a 512-d normalised embedding suitable for cosine similarity
  search.
- `clap_text.onnx` — text encoder. Takes RoBERTa token ids + attention
  mask, produces a 512-d normalised embedding in the same space as the
  audio encoder so a text query can rank audio files by similarity.
- `tokenizer.json` — the RoBERTa BPE tokenizer that pairs with the
  text encoder. MAGDA's C++ tokenizer reads this file directly.

## How MAGDA uses these

MAGDA's media database (a SQLite catalogue of audio samples) uses
these encoders to:

- Compute an embedding per indexed sample at index time, stored in the
  `media_embedding` table.
- Encode the user's free-text search query at query time and rank
  samples by cosine similarity to the query embedding.

Without these models MAGDA falls back to filename / tag full-text
search — still useful, just no semantic similarity.

## Export procedure

ONNX exports are generated from `laion/clap-htsat-unfused` via the
export script in MAGDA's prototype:

```
prototypes/media_db/src/media_db/embeddings/onnx_export.py
```

Notes:

- Run on CPU (MPS does not support float64 used by the audio encoder's
  mel filterbank).
- Requires `transformers >= 5.x`. The audio-feature accessor was renamed
  from `audios=` to `audio=` between 4.x and 5.x; passing the old kwarg
  silently returns wrong shapes.
- `tokenizer.json` is the unmodified file from the upstream HF repo,
  fetched via `AutoTokenizer.from_pretrained(...).save_pretrained(...)`.

## License

BSD-3-Clause — same as the upstream LAION CLAP weights. See the
upstream repo for the original notice and attribution.