| --- |
| license: gpl-3.0 |
| language: |
| - en |
| library_name: onnx |
| tags: |
| - audio |
| - music |
| - music-recommendation |
| - clap |
| - onnx |
| - on-device |
| - android |
| - mobile |
| pipeline_tag: feature-extraction |
| --- |
| |
| # LatentJam Models |
|
|
| ONNX model weights for [LatentJam](https://github.com/Nikita-sud/latentjam) — a privacy-first Android music player that recommends what to play next entirely on-device. These models live here because [`clap_audio.onnx`](./clap_audio.onnx) is 116 MB and exceeds GitHub's 100 MB per-file cap. |
|
|
| The Android app downloads these files at build time via [`scripts/download-models.sh`](https://github.com/Nikita-sud/latentjam/blob/main/scripts/download-models.sh) and bundles them into `app/src/main/assets/ml/`. Inference at runtime uses [ONNX Runtime](https://onnxruntime.ai/) with the [Qualcomm QNN execution provider](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html) for Hexagon NPU offload on Snapdragon devices, falling back to CPU on everything else. |
|
|
| ## Files |
|
|
| | File | Size | Role | |
| |---|---|---| |
| | [`clap_audio.onnx`](./clap_audio.onnx) | 116 MB | Audio encoder derived from CLAP. Consumes a 15 s mono PCM chunk at 48 kHz, produces a 512-d L2-normalized embedding per track. Runs once per track during library indexing, then the embedding is cached in the app's Room database. | |
| | [`predictor_state.onnx`](./predictor_state.onnx) | 32 MB | Transformer-style state encoder. Reads a sequence of recent listening events (skip / listen-through / replay, weighted by recency) and produces a user-state vector. | |
| | [`predictor_scorer_n100.onnx`](./predictor_scorer_n100.onnx) | 5 MB | Top-100 candidate scorer. Given the predictor state and 100 candidate embeddings (chosen by approximate-nearest-neighbor retrieval against the user state), scores each candidate. The highest score becomes the next track in smart-shuffle mode. | |
| | [`embedding_version.txt`](./embedding_version.txt) | 69 B | Bumps when the encoder changes. The app re-extracts all embeddings on mismatch. | |
| | [`predictor_version.txt`](./predictor_version.txt) | 20 B | Bumps when the predictor changes. The app drops the predictor cache on mismatch. | |
|
|
| ## Intended use |
|
|
| - Powering the **smart-shuffle** feature in the LatentJam Android app: cycling the shuffle button to `SMART` picks the next track using these models. |
| - Experimenting with on-device music recommendation on mobile. The encoder + predictor are deliberately small — the entire pipeline (audio decode → encoder → state encoder → scorer) runs end-to-end in under a second on a Snapdragon 8 Gen 3 with the Hexagon NPU enabled. |
|
|
| These models are **not** intended for: |
| - Server-side recommendation (use a bigger CLAP variant and a proper retrieval index) |
| - Music classification or tagging |
| - Generating audio |
|
|
| ## Pipeline overview |
|
|
| ``` |
| Library indexing (one-time, in background) |
| ┌──────────────────────────────────────┐ |
| mp3 / flac / opus / m4a / ogg ──┤ native C++ decoder (in LatentJam) │ |
| │ ↓ │ |
| │ 15 s mono PCM at 48 kHz │ |
| │ ↓ │ |
| │ clap_audio.onnx (this repo) │ |
| │ ↓ │ |
| │ 512-d embedding, L2-normalized │ |
| │ ↓ │ |
| │ Room (on-device cache) │ |
| └──────────────────────────────────────┘ |
| |
| Smart-shuffle inference (on demand) |
| ┌──────────────────────────────────────┐ |
| listening history (Room) ──┤ predictor_state.onnx (this repo) │ |
| │ ↓ │ |
| │ user-state vector │ |
| │ ↓ │ |
| │ ANN retrieval over cached embeddings │ |
| │ ↓ │ |
| │ 100 candidate tracks │ |
| │ ↓ │ |
| │ predictor_scorer_n100.onnx │ |
| │ ↓ │ |
| │ next track │ |
| └──────────────────────────────────────┘ |
| ``` |
|
|
| ## Privacy |
|
|
| - All inference is on-device. No audio, no embeddings, no listening history is ever transmitted anywhere. |
| - The LatentJam Android app does not request the `INTERNET` permission for the recommender. The only network access is the build-time download from this repo onto the developer's machine. |
|
|
| ## Limitations |
|
|
| - Smart mode requires that an embedding has been computed for every track. The first time you index a large library this takes a while — the encoder runs in the background only when the device is **charging + idle** (via WorkManager) to avoid thermal throttling and battery drain. |
| - The encoder is CLAP-derived but distilled to fit on-device. Genre/mood discrimination is good for popular Western genres and weaker for genres CLAP's training data underrepresented. |
| - The predictor was trained on a closed user-history dataset and may not generalize perfectly to your taste right away. On-device fine-tuning is planned but not yet shipped (see [`ml/retrain/RetrainWorker.kt`](https://github.com/Nikita-sud/latentjam/blob/main/app/src/main/java/io/github/nikitasud/latentjam/ml/retrain/RetrainWorker.kt) in the app repo — currently a stub). |
|
|
| ## License |
|
|
| GPL-3.0-or-later, matching the [LatentJam Android app](https://github.com/Nikita-sud/latentjam). |
|
|
| The CLAP audio encoder is derived from [LAION's CLAP](https://github.com/LAION-AI/CLAP) (CC0/MIT) and quantized + exported to ONNX for on-device use. The state encoder and scorer were trained from scratch for this project. |
|
|
| ## Links |
|
|
| - 📱 **Android app**: https://github.com/Nikita-sud/latentjam |
| - 📐 **Architecture notes**: https://github.com/Nikita-sud/latentjam/blob/main/ARCHITECTURE_NOTES.md |
| - 📜 **Fork notice & attribution**: https://github.com/Nikita-sud/latentjam/blob/main/FORK_NOTICE.md |
