Audio Classification
LiteRT
LiteRT
mediapipe
on-device
android
ios
web
python
flutter
audio
audioset
yamnet
sound-event-detection
fp32
int8
Instructions to use litert-community/MediaPipe-AudioClassifier-YAMNet with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/MediaPipe-AudioClassifier-YAMNet with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from litert-community/MediaPipe-AudioClassifier-YAMNet: direct link, hf CLI and curl.
- Browser
- Download file 7.72 kB
-
https://huggingface.co/litert-community/MediaPipe-AudioClassifier-YAMNet/resolve/main/README.md
- Command line
-
hf download hf://litert-community/MediaPipe-AudioClassifier-YAMNet/README.md
-
curl -L -o README.md https://huggingface.co/litert-community/MediaPipe-AudioClassifier-YAMNet/resolve/main/README.md
7.72 kB
| license: apache-2.0 | |
| library_name: litert | |
| pipeline_tag: audio-classification | |
| tags: | |
| - mediapipe | |
| - litert | |
| - tflite | |
| - audio-classification | |
| - on-device | |
| - android | |
| - ios | |
| - web | |
| - python | |
| - flutter | |
| - audio | |
| - audioset | |
| - yamnet | |
| - sound-event-detection | |
| - fp32 | |
| - int8 | |
| # MediaPipe YAMNet (Audio Classifier) | |
| Audio event classification model (`yamnet.tflite`, 4.1 MB, float32 PCM I/O with internal 8-bit weight quantization) trained on AudioSet to recognize **521 sound classes** (speech, musical instruments, animals, alarms, environmental sounds) in the [MediaPipe Audio Classifier (`AudioClassifier`)](https://developers.google.com/edge/mediapipe/solutions/audio/audio_classifier) task. | |
| <div style="border: 1px solid rgba(59, 130, 246, 0.28); background-color: rgba(59, 130, 246, 0.06); border-radius: 8px; padding: 12px 16px; margin: 16px 0;"> | |
| <p style="margin: 0 0 6px 0;"><strong>Documentation:</strong> <a href="https://developers.google.com/edge/mediapipe/solutions/audio/audio_classifier">MediaPipe Audio Classifier Docs</a></p> | |
| <p style="margin: 0 0 6px 0;"><strong>Samples:</strong> <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/audio_classifier/android">Android</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/audio_classifier/ios">iOS</a> · <a href="https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/audio_classifier/python/audio_classification.ipynb">Python Colab</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/audio-classifier.ts">Web</a> · <a href="https://github.com/google/flutter-mediapipe/tree/main/samples/audio/audio_classifier">Flutter</a></p> | |
| <p style="margin: 0;"><strong>Try live:</strong> <a href="https://google-ai-edge.github.io/mediapipe-samples-web/#/audio/audio_classifier">Web Demo</a></p> | |
| </div> | |
| Takes 16 kHz mono float32 waveform windows (`15600` samples = `0.975 s`, normalized in `[-1.0, 1.0]`), computes log-mel spectrograms internally, and outputs `[1, 521]` class scores aligned with [`yamnet_label_list.txt`](./yamnet_label_list.txt) (embedded inside `TFLITE_METADATA` and included as a companion file). | |
| ## Files | |
| | File | Precision | Size | SHA-256 | | |
| |---|---|---:|---| | |
| | [`yamnet.tflite`](https://huggingface.co/litert-community/MediaPipe-AudioClassifier-YAMNet/blob/main/yamnet.tflite) | `float32` | 4.13 MB (4,126,810 B) | `4d8b4a53282dc83ef04e3e7dbc4fbc98082e34e44ed798e16c3a0cdd4c584faf` | | |
| ### Companion Files in Repo Root | |
| | File | Role | Size | SHA-256 | | |
| |---|---|---:|---| | |
| | [`yamnet_label_list.txt`](./yamnet_label_list.txt) | Label map | 6.2 KB (6,230 B) | `8e1267a120c1932b7273c0d0e0c5529edbb9a35512b437b1c8982baa59047051` | | |
| ### Download | |
| ```bash | |
| pip install -U huggingface_hub | |
| hf download litert-community/MediaPipe-AudioClassifier-YAMNet yamnet.tflite --local-dir . | |
| ``` | |
| ## Inputs, Outputs, and Model Specifications | |
| ### `yamnet.tflite` | |
| - Embedded `TFLITE_METADATA`: name `yamnet/classification` · version `1` · author `MediaPipe` · packed associated files: `yamnet_label_list.txt` | |
| | Direction | Tensor | Shape | Type | Notes | | |
| |---|---|---|---|---| | |
| | Input | `waveform_binary` | `[15600]` | `float32` | audio_clip — Input audio clip to be classified (16 kHz float32 waveform samples in range -1.0..1.0). | | |
| | Output | `tower0/network/layer32/final_output` | `[1, 521]` | `float32` | scores — Scores in range 0..1.0 for each of the 521 output classes. — Labels: `yamnet_label_list.txt` | | |
| ## Use with MediaPipe Tasks | |
| ### Python Quickstart | |
| ```bash | |
| pip install -U mediapipe huggingface_hub | |
| ``` | |
| ```python | |
| import numpy as np | |
| import mediapipe as mp | |
| from huggingface_hub import hf_hub_download | |
| model_path = hf_hub_download(repo_id="litert-community/MediaPipe-AudioClassifier-YAMNet", filename="yamnet.tflite") | |
| BaseOptions = mp.tasks.BaseOptions | |
| AudioClassifier = mp.tasks.audio.AudioClassifier | |
| AudioClassifierOptions = mp.tasks.audio.AudioClassifierOptions | |
| AudioRunningMode = mp.tasks.audio.RunningMode | |
| AudioData = mp.tasks.components.containers.AudioData | |
| options = AudioClassifierOptions( | |
| base_options=BaseOptions(model_asset_path=model_path), | |
| running_mode=AudioRunningMode.AUDIO_CLIPS, | |
| max_results=5, | |
| ) | |
| clip_samples = np.zeros(16000, dtype=np.float32) # replace with 16 kHz mono PCM in [-1.0, 1.0] | |
| audio_data = AudioData.create_from_array(clip_samples, sample_rate=16000) | |
| with AudioClassifier.create_from_options(options) as classifier: | |
| for window in classifier.classify(audio_data): | |
| print([(c.category_name, round(c.score, 3)) for c in window.classifications[0].categories]) | |
| ``` | |
| ### Cross-Platform Guides & Samples | |
| - **Task overview & options:** [Audio Classifier documentation](https://developers.google.com/edge/mediapipe/solutions/audio/audio_classifier) | |
| - **Platform setup guides:** [Python](https://developers.google.com/edge/mediapipe/solutions/setup_python) · [Web / JavaScript](https://developers.google.com/edge/mediapipe/solutions/setup_web) · [Android (Kotlin / Java)](https://developers.google.com/edge/mediapipe/solutions/setup_android) · [iOS (Swift)](https://developers.google.com/edge/mediapipe/solutions/setup_ios) · [Flutter (Dart)](https://developers.google.com/edge/mediapipe/solutions/setup_flutter) | |
| - **Runnable sample apps:** [Web Demo](https://google-ai-edge.github.io/mediapipe-samples-web/#/audio/audio_classifier) · [Web Source](https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/audio-classifier.ts) · [Android](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/audio_classifier/android) · [iOS](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/audio_classifier/ios) · [Python Notebook](https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/audio_classifier/python/audio_classification.ipynb) · [Flutter](https://github.com/google/flutter-mediapipe/tree/main/samples/audio/audio_classifier) | |
| ## Latency Benchmarks | |
| End-to-end MediaPipe Task pipeline average latency reported by the [Audio Classifier documentation](https://developers.google.com/edge/mediapipe/solutions/audio/audio_classifier#benchmarks): | |
| | Model Variant | Device | CPU Latency | GPU Latency | | |
| |---|---|---:|---:| | |
| | YamNet | Pixel 6 | 12.29 ms | — | | |
| ## Intended Use, Limitations, and Fairness | |
| - **Intended applications:** Real-time audio event tagging and acoustic scene recognition across the 521 AudioSet categories on mobile, web, desktop, and embedded devices. | |
| - **Limitations & considerations:** Multi-label sound event classifier (not a speech-to-text ASR model or biometric speaker recognizer); expects 16 kHz mono audio normalized to `[-1.0, 1.0]`. No standalone Google fairness PDF is published on the MediaPipe documentation page. | |
| ## Provenance | |
| Every model binary in this repository is byte-identical (unmodified FlatBuffer weights and embedded `TFLITE_METADATA` bytes) to the official artifact served by Google from `storage.googleapis.com/mediapipe-models`. | |
| | Repo File | Upstream URL | GCS Generation | Upstream `Last-Modified` | Pinned Bucket Version | | |
| |---|---|---|---|---| | |
| | `yamnet.tflite` | [`yamnet.tflite`](https://storage.googleapis.com/mediapipe-models/audio_classifier/yamnet/float32/latest/yamnet.tflite) | `1682545011390892` | Wed, 26 Apr 2023 21:36:51 GMT | `/1/` | | |
| ## License and Citation | |
| - License: `apache-2.0` — Stated as Apache License, Version 2.0 (`http://www.apache.org/licenses/LICENSE-2.0`) in the embedded `TFLITE_METADATA` block of the `.tflite` model file(s). | |
| ### Citation | |
| Trained on [AudioSet](https://research.google.com/audioset/) (`TFLITE_METADATA` author: MediaPipe, Apache License 2.0). | |