| --- |
| license: bsd-3-clause |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - audio-classification |
| base_model: |
| - MIT/ast-finetuned-audioset-10-10-0.4593 |
| --- |
| # AST audio event classification — ExecuTorch |
|
|
| Ten seconds of sound in, 527 AudioSet labels out: speech, music, a dog, a door, an engine. |
| The shelf had speech recognition and speaker identity; this is the one that says what a |
| sound *is*. |
|
|
| - **Source**: MIT/ast-finetuned-audioset-10-10-0.4593 — 86M parameters, Audio Spectrogram Transformer |
| - **License**: bsd-3-clause |
| - **Input**: `input_values` `[1, 1024, 128]` fp32 — log-mel filterbank [1, 1024, 128] fp32 — `torchaudio.compliance.kaldi.fbank` at 16000 Hz with 128 mel bins, padded or trimmed to 1024 frames (10.24 s), then normalised with mean=-4.2677393 std=4.5689974. `ASTFeatureExtractor` does exactly this |
| - **Output**: logits [1, 527] — AudioSet labels, multi-label: apply sigmoid, not softmax |
|
|
| ## Variants |
|
|
| | build | file | size (MB) | Mac median (ms)* | top-1 vs eager | worst probability shift | |
| |---|---|---|---|---|---| |
| | fp32 | `audiocls_audioset_xnnpack_fp32.pte` | 346.6 | 285.0 | 8 of 8 | 0.0000 | |
| | fp16 | `audiocls_audioset_xnnpack_fp16.pte` | 173.9 | 545.1 | 8 of 8 | 0.0014 | |
| | int8 (dynamic) | `audiocls_audioset_xnnpack_int8.pte` | 90.9 | 267.3 | 8 of 8 | 0.0160 | |
| | Core ML (fp16, iOS) | `audiocls_audioset_coreml_all.pte` | 173.7 | 74.6 | 8 of 8 | 0.0033 | |
|
|
| \*Mac arm64, single process, median of 10, one 10.24 s clip. PyTorch eager fp32 on the same |
| machine is **122.7 ms**. Core ML at 74.6 ms is 1.6x that; int8 at 267.3 ms |
| is the fastest portable build and a quarter of the fp32 file. fp16 is **slower than fp32** |
| here (545.1 ms) — XNNPACK emulates it — and only earns its place by halving the file. |
| |
| ## What the classifier actually says |
| |
| The test clips are speech, and every build puts `Speech` in the top five on all |
| 8 of them. The top-1 label matches eager on 8 of 8, the top-five sets overlap |
| 40 of 40, and no sigmoid probability moves by more than the figure in the table. |
| |
| The distance the error has to cover is printed too: the gap between the winning logit and |
| the runner-up is at least **3.09** on these clips, which every build's shift is far |
| inside. Agreement alone would not show this — a build returning a constant vector would |
| agree with a broken reference on every clip — so the label check is there as well. |
| |
| ## The features are the caller's job, and the recipe is exact |
| |
| AST's front end is `torchaudio.compliance.kaldi.fbank`: a Kaldi-compatible filterbank with |
| its own windowing and edge handling. Reimplementing it inside the graph would be a second |
| model's worth of work for a transform `transformers` runs in two lines, so the graph starts |
| at the spectrogram. The recipe is read off the model's own preprocessor rather than written |
| from memory: |
| |
| ```python |
| from transformers import AutoFeatureExtractor |
| extractor = AutoFeatureExtractor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593") |
| inputs = extractor(waveform, sampling_rate=16000, return_tensors="pt")["input_values"] |
| ``` |
| |
| Getting it wrong does not throw. It shifts every probability. |
| |
| **The output is multi-label**: apply `sigmoid`, not `softmax`. A ten-second clip can be |
| speech *and* music *and* a car at once, which is the point of AudioSet. |
|
|
| ## Conversion |
|
|
| ```bash |
| python convert/export_audiocls.py audioset |
| python convert/check_audiocls.py audioset int8 |
| ``` |
|
|
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
|
|
