File size: 3,491 Bytes
3aa3702 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | ---
license: bsd-3-clause
tags:
- executorch
- xnnpack
- pte
- on-device
- audio-classification
base_model:
- MIT/ast-finetuned-audioset-10-10-0.4593
---
# AST audio event classification — ExecuTorch
Ten seconds of sound in, 527 AudioSet labels out: speech, music, a dog, a door, an engine.
The shelf had speech recognition and speaker identity; this is the one that says what a
sound *is*.
- **Source**: MIT/ast-finetuned-audioset-10-10-0.4593 — 86M parameters, Audio Spectrogram Transformer
- **License**: bsd-3-clause
- **Input**: `input_values` `[1, 1024, 128]` fp32 — log-mel filterbank [1, 1024, 128] fp32 — `torchaudio.compliance.kaldi.fbank` at 16000 Hz with 128 mel bins, padded or trimmed to 1024 frames (10.24 s), then normalised with mean=-4.2677393 std=4.5689974. `ASTFeatureExtractor` does exactly this
- **Output**: logits [1, 527] — AudioSet labels, multi-label: apply sigmoid, not softmax
## Variants
| build | file | size (MB) | Mac median (ms)* | top-1 vs eager | worst probability shift |
|---|---|---|---|---|---|
| fp32 | `audiocls_audioset_xnnpack_fp32.pte` | 346.6 | 285.0 | 8 of 8 | 0.0000 |
| fp16 | `audiocls_audioset_xnnpack_fp16.pte` | 173.9 | 545.1 | 8 of 8 | 0.0014 |
| int8 (dynamic) | `audiocls_audioset_xnnpack_int8.pte` | 90.9 | 267.3 | 8 of 8 | 0.0160 |
| Core ML (fp16, iOS) | `audiocls_audioset_coreml_all.pte` | 173.7 | 74.6 | 8 of 8 | 0.0033 |
\*Mac arm64, single process, median of 10, one 10.24 s clip. PyTorch eager fp32 on the same
machine is **122.7 ms**. Core ML at 74.6 ms is 1.6x that; int8 at 267.3 ms
is the fastest portable build and a quarter of the fp32 file. fp16 is **slower than fp32**
here (545.1 ms) — XNNPACK emulates it — and only earns its place by halving the file.
## What the classifier actually says
The test clips are speech, and every build puts `Speech` in the top five on all
8 of them. The top-1 label matches eager on 8 of 8, the top-five sets overlap
40 of 40, and no sigmoid probability moves by more than the figure in the table.
The distance the error has to cover is printed too: the gap between the winning logit and
the runner-up is at least **3.09** on these clips, which every build's shift is far
inside. Agreement alone would not show this — a build returning a constant vector would
agree with a broken reference on every clip — so the label check is there as well.
## The features are the caller's job, and the recipe is exact
AST's front end is `torchaudio.compliance.kaldi.fbank`: a Kaldi-compatible filterbank with
its own windowing and edge handling. Reimplementing it inside the graph would be a second
model's worth of work for a transform `transformers` runs in two lines, so the graph starts
at the spectrogram. The recipe is read off the model's own preprocessor rather than written
from memory:
```python
from transformers import AutoFeatureExtractor
extractor = AutoFeatureExtractor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
inputs = extractor(waveform, sampling_rate=16000, return_tensors="pt")["input_values"]
```
Getting it wrong does not throw. It shifts every probability.
**The output is multi-label**: apply `sigmoid`, not `softmax`. A ten-second clip can be
speech *and* music *and* a car at once, which is the point of AudioSet.
## Conversion
```bash
python convert/export_audiocls.py audioset
python convert/check_audiocls.py audioset int8
```
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))
|