README.md

---
license: apache-2.0
library_name: litert
pipeline_tag: audio-classification
tags: [music-transcription, audio-to-midi, basic-pitch, cqt, litert, tflite, on-device, gpu]
base_model: spotify/basic-pitch
---

# Basic Pitch — LiteRT (CompiledModel GPU) music transcription


![on-device result](assets/hero.png)

*Audio input → note posteriorgram (MIDI pitch × time) from the on-device model.*

[Basic Pitch](https://github.com/spotify/basic-pitch) (Spotify, ICASSP 2022, Apache-2.0)
re-authored to a **GPU-native** LiteRT `.tflite` — **including the conv-based CQT front-end**
(9 octaves, shared 36×256 kernel banks, lowpass downsample chain, no FFT). Bit-exact torch
re-implementation of the official ONNX (corr 1.000000). FP32, **0.84 MB**.

Pixel 8a (Tensor G3): **241/241 nodes LITERT_CL** (1 partition), **~4.4 ms** per 2 s window;
note-event F1@0.5 **0.98** vs the official model, per-frame argmax agreement 98%.

## I/O
- Input `[1, 43844]` float32 — 2 s @ 22 050 Hz mono, [-1, 1] (official window; overlap windows by
  7 680 samples and keep center frames when stitching).
- Outputs: `contour [1,172,264]`, `note [1,172,88]`, `onset [1,172,88]` — sigmoid posteriorgrams
  (~11.6 ms frames; note/onset bins = MIDI 21–108).

## Minimal usage

```python
import numpy as np, soundfile as sf
from ai_edge_litert.interpreter import Interpreter

wav, _ = sf.read("audio_22050.wav", dtype="float32")   # mono 22.05 kHz
x = np.zeros(43844, np.float32); n = min(len(wav), 43844); x[:n] = wav[:n]
it = Interpreter(model_path="basicpitch.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], x[None]); it.invoke()
contour, note, onset = (it.get_tensor(o["index"])[0] for o in
                        sorted(it.get_output_details(), key=lambda o: o["index"]))
active = np.argwhere(note > 0.5)          # (frame, key); midi = key + 21, t = frame * 256/22050
```

### Kotlin (Android, LiteRT CompiledModel GPU)

```kotlin
// implementation("com.google.ai.edge.litert:litert:2.1.5")
val model = CompiledModel.create(File(ctx.filesDir, "basicpitch.tflite").absolutePath,
    CompiledModel.Options(Accelerator.GPU), null)
val inBuf = model.createInputBuffers(); val outBuf = model.createOutputBuffers()
inBuf[0].writeFloat(window43844)          // 2 s @ 22.05 kHz mono
model.run(inBuf, outBuf)
val note = outBuf[1].readFloat()          // [172 * 88], frame-major; midi = bin + 21
val onset = outBuf[2].readFloat()         // onset-triggered decoding -> note events
```

## Conversion
Extracted from the official `nmp.onnx` (102 constants; no TensorFlow). Reflect-pad →
anti-diagonal-constant `FULLY_CONNECTED`; PACK → concat + static slices. Two fp16-on-GPU fixes,
both exact: post-log clamp `clamp(10·log10(p+1e-10), min=-100)` (recovers `log(0)` from the
fp16-flushed floor; desktop no-op) and the per-bin CQT norm folded into per-octave kernel copies
(magnitude is linear in kernel scale). FP32 flatbuffer (fp16 weights cost ~0.005 corr on the tiny
CQT kernels).

## Upstream
[spotify/basic-pitch](https://github.com/spotify/basic-pitch) (Apache-2.0). Please cite
Bittner et al., *A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and
Multipitch Estimation* (ICASSP 2022).