|
Download README.md from audiojs/yamnet: direct link, hf CLI and curl.
- Browser
- Download file 5.37 kB
-
https://huggingface.co/audiojs/yamnet/resolve/main/README.md
- Command line
-
hf download hf://audiojs/yamnet/README.md
-
curl -L -o README.md https://huggingface.co/audiojs/yamnet/resolve/main/README.md
5.37 kB
| license: apache-2.0 | |
| library_name: onnx | |
| pipeline_tag: audio-classification | |
| tags: | |
| - audio | |
| - audio-classification | |
| - audio-tagging | |
| - audioset | |
| - yamnet | |
| - onnx | |
| # YAMNet, ONNX (float16 weights) | |
| YAMNet (tensorflow/models, [`research/audioset/yamnet`](https://github.com/tensorflow/models/tree/c14bf9ad91962cf189f9f58db2132c06247fcd53/research/audioset/yamnet), | |
| by Manoj Plakal and Dan Ellis, Google): a MobileNetV1 trained on AudioSet that scores each 0.96 s log-mel patch for 521 | |
| AudioSet classes, a sigmoid each. This is its network over patches, converted for | |
| [@audio/neural-tag](https://github.com/audiojs/neural/tree/main/packages/neural-tag), which computes the patches in | |
| JavaScript as upstream's `features.py` does and runs this file through onnxruntime (Node, or a browser's | |
| onnxruntime-web). | |
| | File | Size | SHA-256 | | |
| |---|---|---| | |
| | `yamnet.fp16.onnx` | 7.5 MB (7,495,175 bytes) | `0a1f3b050ebfdfa651e78fb12714c3c1fe750ceebd58529818a1ddddec515688` | | |
| ## Source | |
| - Code: [tensorflow/models](https://github.com/tensorflow/models) at `c14bf9ad91962cf189f9f58db2132c06247fcd53`, the | |
| last commit to touch `research/audioset/yamnet`: `yamnet.py`, `features.py`, `params.py`, `yamnet_class_map.csv`. | |
| - Weights: [`yamnet.h5`](https://storage.googleapis.com/audioset/yamnet.h5), as upstream's README links it | |
| (15,296,092 bytes, SHA-256 `13c3308955bbfaef262f175ac9c40e47b134573a93984f009220dd7cc12a1744`). | |
| ## Graph | |
| | | name | shape | | | |
| |---|---|---|---| | |
| | input | `patches` | [n, 96, 64] float32 | log-mel patches: 96 frames of 10 ms × 64 mel bands, as `features.py` frames them | | |
| | output | `scores` | [n, 521] float32 | a sigmoid per class, in `yamnet_class_map.csv`'s order | | |
| The patches, at 16 kHz mono (`features.py`, `params.py`): the waveform padded with zeros to at least 15,600 samples and | |
| then to a whole hop; frames of 400 samples every 160, periodic Hann, |FFT 512|; 64 HTK-mel bands 125 to 7500 Hz | |
| (`tf.signal.linear_to_mel_weight_matrix`); ln(mel + 0.001); 96 frames every 48 (0.96 s every 0.48 s). | |
| ```js | |
| import { tag } from '@audio/neural-tag' | |
| let tags = await tag(samples, 44100) // [{ name: 'Speech', id: '/m/09x0r', index: 0, score }, …] | |
| ``` | |
| ## Conversion and verification | |
| `scripts/export.py` in @audio/neural-tag builds the graph from `yamnet.h5`'s arrays (h5py, onnx; opset 17): the 14 | |
| convolution layers of `yamnet.py`, each batch norm (center, no scale, epsilon 1e-4) folded into its convolution in | |
| float64, TensorFlow's 'same' padding written out per layer, global average pooling, the dense layer and the sigmoid; | |
| 3,729,481 parameters. The weights of over 1024 values are stored as float16 and Cast to float32 in the graph (biases | |
| and the first three kernels stay float32): computed in float32, the file half the float32 export's 14.9 MB. | |
| Against the Keras model (upstream's `yamnet.py` with `yamnet.h5` loaded, TensorFlow 2.21, tf-keras 2.21) on the same 49 | |
| patches of seven test signals (speech, plucked notes, a swept tone, white noise, a beep, silence, nothing): | |
| | | max \|score difference\| | mean | top class | | |
| |---|---|---|---| | |
| | float32 export | 1.6e-6 | 1.5e-8 | the same in 49 of 49 | | |
| | this file | 3.6e-3 | 3.0e-6 | the same in 49 of 49 | | |
| End to end, @audio/neural-tag's JavaScript features through this file against upstream's `yamnet_frames_model` on the | |
| same waveform: 2.3e-3 at most on 12.3 s of speech, 3.6e-3 on plucked notes, within 1.4e-3 on the rest; the same top | |
| classes. The features themselves are within 2.4e-7 of upstream's ops computed in float64. | |
| ## Accuracy, as upstream states it | |
| On AudioSet's 20,366-segment evaluation set, over the 521 classes: balanced average d-prime 2.318, balanced mAP 0.306, | |
| balanced average lwlrap 0.393 (upstream's README). Six of AudioSet's 527 classes were dropped "on the recommendation of | |
| our Fairness reviewers": the gendered Speech and Singing, Battle cry, Funny music. "YAMNet's classifier outputs have not | |
| been calibrated across classes, so you cannot directly treat the outputs as probabilities" (the TF Hub model card, | |
| Limitations). | |
| ## Licence and attribution | |
| Apache License 2.0 ([LICENSE](LICENSE): tensorflow/models' licence at the commit above, which covers `/research`). | |
| YAMNet's code and weights are Google's, "Copyright 2019 The TensorFlow Authors"; the same model on Kaggle (formerly TF | |
| Hub `google/yamnet/1`) states Apache 2.0. Modified from the original weights: converted to ONNX, batch norms folded, the | |
| weights stored as float16. The class names and ids are the AudioSet ontology's, which Google makes available under | |
| CC BY-SA 4.0; AudioSet's labels are CC BY 4.0. | |
| ```bibtex | |
| @inproceedings{gemmeke2017audioset, | |
| title = {Audio Set: An ontology and human-labeled dataset for audio events}, | |
| author = {Gemmeke, Jort F. and Ellis, Daniel P. W. and Freedman, Dylan and Jansen, Aren and Lawrence, Wade and Moore, R. Channing and Plakal, Manoj and Ritter, Marvin}, | |
| booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, | |
| year = {2017} | |
| } | |
| @article{howard2017mobilenets, | |
| title = {MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications}, | |
| author = {Howard, Andrew G. and Zhu, Menglong and Chen, Bo and Kalenichenko, Dmitry and Wang, Weijun and Weyand, Tobias and Andreetto, Marco and Adam, Hartwig}, | |
| journal = {arXiv:1704.04861}, | |
| year = {2017} | |
| } | |
| ``` | |