litert-community/PANNs-CNN14-AudioSet-LiteRT

PANNs CNN14 — LiteRT (on-device AudioSet tagging, GPU CNN + host log-mel)

PANNs CNN14 (Cnn14_mAP=0.431) general sound-event tagging, converted to LiteRT with the CNN body running fully on the CompiledModel GPU (ML Drift) on Android. Given ~10 s of audio it predicts probabilities over the 527 AudioSet classes — speech, music, instruments, animals, vehicles, alarms, household sounds, and so on. AudioSet tagging is multi-label: several tags can be high at once.

waveform[320000] (32 kHz) →[host: log-mel]→ logmel[1,1,1001,64] →[GPU: CNN14]→ probs[1,527] (sigmoid)

On-device (Pixel 8a, Tensor G3 — verified)

nodes on GPU	45 / 45 LITERT_CL (full residency, single graph, 1 partition)
inference	~124 ms GPU + ~99 ms host log-mel ≈ 0.22 s per 10 s clip
size	162 MB (fp16)
accuracy	fp16 tflite-vs-PyTorch corr 1.000000; self-test top tag "Speech"

How it converts (litert-torch) — and why the log-mel is host-side

PANNs builds its spectrogram with torchlibrosa, whose STFT is a DFT-as-Conv1d — so there is no FFT op and the whole raw-audio→tags graph is almost GPU-clean; the only blocker is the STFT centering reflect-pad (one GATHER_ND, removable via pad_mode='constant', corr 1.0). But the converted spectral front-end is unusable: litert-torch lowers the giant 1024-tap DFT-conv incorrectly (fp32 tflite corr ≈ 0.19), and the power spectrum |STFT|² (~1e6) overflows fp16 on Mali → NaN.

So the spectral front-end is computed on the CPU (the Whisper/Kokoro pattern), matched to torchlibrosa exactly, and only the CNN body rides the GPU:

• log-mel (host) — reflect-pad center, periodic Hann, 1024-pt FFT, power, mel matmul (librosa.filters.mel, slaney), 10·log10(max(mel,1e-10)). Validated host-vs-torch corr 1.000000 (max|d| 0.0017). The mel basis is shipped here as mel_basis.bin [64, 513].
• CNN14 body (GPU) — bn0 + 6 conv blocks + mean/max time-pool + 2 FC + sigmoid. Pure CNN, converts at corr 1.000000 in fp32 and fp16, op-check banned NONE / >4D 0, one delegatable graph.

Files

File	What
cnn14_audioset_fp16.tflite	the CNN body, fp16, input logmel [1,1,1001,64] → probs [1,527]
mel_basis.bin	mel filterbank [64, 513] float32 for the host log-mel
audioset_labels.txt	the 527 AudioSet class display names (row index = class id)
build_panns.py	conversion + host-mel validation script

Preprocessing

Mono 32 kHz, padded/truncated to 10 s (320000 samples), values in [-1, 1]. Compute the log-mel as above → [1,1,1001,64]. The output 527 sigmoid probabilities are per-class (multi-label); take the top-K as tags.

Minimal usage

Android (Kotlin, CompiledModel GPU)

// staged into filesDir by an install script (162 MB — too big for assets)
val model = CompiledModel.create(File(ctx.filesDir, "cnn14_audioset_fp16.tflite").absolutePath,
    CompiledModel.Options(Accelerator.GPU), null)
val inputs = model.createInputBuffers(); val outputs = model.createOutputBuffers()
inputs[0].writeFloat(logmel)          // [1,1,1001,64] host log-mel (see Python below)
model.run(inputs, outputs)
val probs = outputs[0].readFloat()    // [527] sigmoid, multi-label -> top-K tags

Python (desktop verification)

import numpy as np, soundfile as sf
from ai_edge_litert.interpreter import Interpreter

SR, NFFT, HOP, NMEL, CLIP = 32000, 1024, 320, 64, 320000
wav, _ = sf.read("clip_32k.wav", dtype="float32")               # mono 32 kHz
x = np.zeros(CLIP, np.float32); n = min(len(wav), CLIP); x[:n] = wav[:n]

# torchlibrosa-exact log-mel: center reflect-pad, periodic Hann, |rFFT|^2, mel, 10*log10
pad = np.pad(x, NFFT // 2, mode="reflect")
win = 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(NFFT) / NFFT)
frames = 1 + CLIP // HOP                                        # 1001
power = np.stack([np.abs(np.fft.rfft(pad[t*HOP:t*HOP+NFFT] * win))**2 for t in range(frames)])
fb = np.fromfile("mel_basis.bin", np.float32).reshape(NMEL, 513)
logmel = (10.0 * np.log10(np.maximum(power @ fb.T, 1e-10))).astype(np.float32)

it = Interpreter(model_path="cnn14_audioset_fp16.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], logmel[None, None]); it.invoke()
probs = it.get_tensor(it.get_output_details()[0]["index"])[0]   # [527]

labels = open("audioset_labels.txt").read().splitlines()
for i in probs.argsort()[::-1][:5]:
    print(f"{probs[i]:.3f}  {labels[i]}")

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
TFLite benchmark_model (TfLiteGpuDelegateV2)	GPU (OpenCL)	45 / 45	111.1 ms
TFLite benchmark_model	CPU (XNNPACK, 4 threads)	—	XNNPACK declined the graph

Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.

XNNPACK declines these fp16 graphs — it reports failed to delegate DEPTHWISE_CONV_2D and then fails to allocate tensors — so there is no usable CPU number. Disabling XNNPACK falls back to reference kernels, which measured about 20× slower than the GPU on models of this size and would not represent CPU inference anyone would ship.

License

Code Apache-2.0; weights Cnn14_mAP=0.431.pth CC-BY-4.0 (Zenodo). AudioSet ontology © Google, CC-BY-4.0. Upstream: qiuqiangkong/audioset_tagging_cnn.

Citation

@article{kong2020panns,
  title={PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition},
  author={Kong, Qiuqiang and Cao, Yin and Iqbal, Turab and Wang, Yuxuan and Wang, Wenwu and Plumbley, Mark D},
  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  year={2020}
}