|
|
--- |
|
|
license: cc-by-nc-4.0 |
|
|
datasets: |
|
|
- DBD-research-group/BirdSet |
|
|
base_model: |
|
|
- facebook/convnext-base-224-22k |
|
|
pipeline_tag: audio-classification |
|
|
library_name: transformers |
|
|
tags: |
|
|
- audio-classification |
|
|
- audio |
|
|
--- |
|
|
# AudioProtoPNet: An Interpretable Deep Learning Model for Bird Sound Classification |
|
|
|
|
|
|
|
|
|
|
|
## Abstract |
|
|
|
|
|
Deep learning models have significantly advanced acoustic bird monitoring by recognizing numerous bird species |
|
|
based on their vocalizations. However, traditional deep learning models are often "black boxes," providing |
|
|
limited insight into their underlying computations, which restricts their utility for ornithologists and machine |
|
|
learning engineers. Explainable models, on the other hand, can facilitate debugging, knowledge discovery, |
|
|
trust, and interdisciplinary collaboration. |
|
|
|
|
|
This work introduces **AudioProtoPNet**, an adaptation of the Prototypical Part Network (ProtoPNet) |
|
|
designed for multi-label bird sound classification. AudioProtoPNet is inherently interpretable, leveraging a |
|
|
ConvNeXt backbone to extract embeddings and a prototype learning classifier trained on these embeddings. |
|
|
The classifier learns prototypical patterns of each bird species' vocalizations from spectrograms of |
|
|
instances in the training data. |
|
|
|
|
|
During inference, recordings are classified by comparing them to learned prototypes in the embedding space, |
|
|
providing explanations for the model's decisions and insights into the most informative embeddings of each |
|
|
bird species. |
|
|
|
|
|
- **Paper**: www.sciencedirect.com/science/article/pii/S1574954125000901 |
|
|
|
|
|
## Model Description |
|
|
|
|
|
### Training Data |
|
|
|
|
|
The model was trained on the **BirdSet training dataset**, which comprises 9734 bird species and over 6800 |
|
|
hours of recordings. |
|
|
|
|
|
### Evaluation |
|
|
|
|
|
AudioProtoPNet's performance was evaluated on seven BirdSet test datasets, covering diverse geographical |
|
|
regions. The model demonstrated superior performance compared to state-of-the-art bird sound classification |
|
|
models like Perch (which itself outperforms BirdNet). AudioProtoPNet achieved an average AUROC of 0.90 |
|
|
and a cmAP of 0.42, representing relative improvements of 7.1% and 16.7% over Perch, respectively. |
|
|
|
|
|
These results highlight the feasibility of developing powerful yet interpretable deep learning models for the |
|
|
challenging task of multi-label bird sound classification, offering valuable insights for professionals in |
|
|
ornithology and machine learning. |
|
|
|
|
|
### Evaluation Results |
|
|
|
|
|
**Table 1: Mean Performance of AudioProtoPNet Models with Varying Prototypes** |
|
|
|
|
|
Mean performance of AudioProtoPNet models with one, five, ten, and twenty prototypes per class for the |
|
|
validation dataset POW and the seven test datasets, averaged over five different random seeds. The 'Score' |
|
|
column represents the average of the respective metric across all test datasets. Best values for each metric are |
|
|
**bolded**. While models with five, ten, and twenty prototypes performed |
|
|
similarly, the model with only one prototype per class showed slightly lower performance. |
|
|
|
|
|
| | Metric | POW | PER | NES | UHH | HSN | NBP | SSW | SNE | Score | |
|
|
|----------------------|---------|-------|-------|-------|-------|-------|-------|-------|-------|-------| |
|
|
| AudioProtoPNet-1 | cmAP | 0.49 | **0.30** | 0.36 | 0.28 | 0.50 | 0.66 | 0.40 | 0.32 | 0.40 | |
|
|
| | AUROC | 0.88 | 0.79 | 0.92 | 0.85 | 0.91 | 0.92 | 0.96 | 0.84 | 0.88 | |
|
|
| | T1-Acc | **0.87** | 0.59 | 0.49 | 0.42 | 0.64 | 0.71 | 0.64 | 0.70 | 0.60 | |
|
|
| AudioProtoPNet-5 | cmAP | **0.50** | **0.30** | **0.38** | **0.31** | **0.54** | **0.68** | 0.42 | 0.33 | **0.42** | |
|
|
| | AUROC | 0.88 | 0.79 | 0.93 | **0.87** | **0.92** | **0.93** | **0.97** | **0.88** | **0.90** | |
|
|
| | T1-Acc | 0.84 | 0.59 | **0.52** | **0.49** | **0.65** | 0.71 | 0.66 | 0.74 | **0.62** | |
|
|
| AudioProtoPNet-10 | cmAP | **0.50** | **0.30** | **0.38** | 0.30 | **0.54** | **0.68** | 0.42 | **0.34** | **0.42** | |
|
|
| | AUROC | 0.88 | **0.80** | **0.94** | 0.86 | **0.92** | **0.93** | **0.97** | 0.86 | **0.90** | |
|
|
| | T1-Acc | 0.85 | 0.59 | **0.52** | 0.47 | 0.64 | **0.72** | 0.67 | 0.74 | **0.62** | |
|
|
| AudioProtoPNet-20 | cmAP | **0.50** | **0.30** | **0.38** | **0.31** | **0.54** | **0.68** | **0.43** | 0.33 | **0.42** | |
|
|
| | AUROC | **0.89** | **0.80** | **0.94** | 0.86 | **0.92** | **0.93** | **0.97** | 0.87 | **0.90** | |
|
|
| | T1-Acc | **0.87** | **0.60** | **0.52** | 0.42 | **0.65** | **0.72** | **0.68** | **0.75** | **0.62** | |
|
|
|
|
|
**Table 2: Comparative Performance of AudioProtoPNet, ConvNeXt, and Perch** |
|
|
|
|
|
Mean performance of AudioProtoPNet-5, ConvNeXt, and Perch for the validation dataset POW and the seven |
|
|
test datasets, averaged over five different random seeds. The 'Score' column represents the average of the |
|
|
respective metric across all test datasets. Best values for each metric are **bolded**. AudioProtoPNet-5 notably outperformed both Perch and ConvNeXt in terms of cmAP, AUROC, |
|
|
and top-1 accuracy scores. |
|
|
|
|
|
| Model | Metric | POW | PER | NES | UHH | HSN | NBP | SSW | SNE | Score | |
|
|
| :---------------- | :------ | :----- | :----- | :----- | :----- | :----- | :----- | :----- | :----- | :----- | |
|
|
| AudioProtoPNet-5 | cmAP | 0.50 | **0.30** | 0.38 | **0.31** | **0.54** | **0.68** | **0.42** | **0.33** | **0.42** | |
|
|
| | AUROC | 0.88 | **0.79** | **0.93** | **0.87** | **0.92** | **0.93** | **0.97** | **0.86** | **0.90** | |
|
|
| | T1-Acc | 0.84 | **0.59** | 0.52 | 0.49 | **0.65** | **0.71** | **0.66** | **0.74** | **0.62** | |
|
|
| ConvNeXt | cmAP | 0.41 | 0.21 | 0.35 | 0.25 | 0.49 | 0.66 | 0.38 | 0.31 | 0.38 | |
|
|
| | AUROC | 0.83 | 0.73 | 0.89 | 0.72 | 0.88 | 0.92 | 0.93 | 0.83 | 0.84 | |
|
|
| | T1-Acc | 0.75 | 0.43 | 0.49 | 0.43 | 0.60 | 0.69 | 0.58 | 0.62 | 0.56 | |
|
|
| Perch | cmAP | 0.30 | 0.18 | **0.39** | 0.27 | 0.45 | 0.63 | 0.28 | 0.29 | 0.36 | |
|
|
| | AUROC | 0.84 | 0.70 | 0.90 | 0.76 | 0.86 | 0.91 | 0.91 | 0.83 | 0.84 | |
|
|
| | T1-Acc | 0.85 | 0.48 | **0.66** | **0.57** | 0.58 | 0.69 | 0.62 | 0.69 | 0.61 | |
|
|
|
|
|
|
|
|
|
|
|
## Example |
|
|
|
|
|
This model can be easily loaded and used for inference with the `transformers` library. |
|
|
|
|
|
```python |
|
|
from transformers import AutoFeatureExtractor, AutoModelForSequenceClassification |
|
|
import librosa |
|
|
import torch |
|
|
|
|
|
# Load the model and feature extractor |
|
|
model = AutoModelForSequenceClassification.from_pretrained("DBD-research-group/AudioProtoPNet-10-BirdSet-XCL",trust_remote_code=True) |
|
|
feature_extractor = AutoFeatureExtractor.from_pretrained("DBD-research-group/AudioProtoPNet-10-BirdSet-XCL", trust_remote_code=True) |
|
|
model.eval() |
|
|
|
|
|
# Load an example audio file |
|
|
audio_path = librosa.ex('robin') |
|
|
label = "eurrob1" # The eBird label for the European Robin. |
|
|
|
|
|
# The model is trained on audio sampled at 32,000 Hz |
|
|
audio, sample_rate = librosa.load(audio_path, sr=32_000) |
|
|
|
|
|
mel_spectrogram = feature_extractor(audio) |
|
|
|
|
|
outputs = model(mel_spectrogram) |
|
|
probabilities = torch.sigmoid(outputs[0]).detach() |
|
|
|
|
|
# Get the top 5 predictions by confidence |
|
|
top_n_probs, top_n_indices = torch.topk(probabilities, k=5, dim=-1) |
|
|
|
|
|
label2id = model.config.label2id |
|
|
id2label = model.config.id2label |
|
|
|
|
|
print(f'Selected species with confidence:') |
|
|
print(f"{label:<7} - {probabilities[:, label2id[label]].item():.2%}") |
|
|
print("\nTop 5 Predictions with confidence:") |
|
|
for idx, conf in zip(top_n_indices.squeeze(), top_n_probs.squeeze()): |
|
|
print(f"{id2label[idx.item()]:<7} - {conf:.2%}") |
|
|
``` |
|
|
|
|
|
**Expected output** |
|
|
``` |
|
|
Selected species with confidence: |
|
|
eurrob1 - 26.81% |
|
|
Top 5 Predictions with confidence: |
|
|
coatit2 - 49.99% |
|
|
sablar2 - 48.29% |
|
|
palwar5 - 41.58% |
|
|
gretit1 - 37.51% |
|
|
verdin - 34.72% |
|
|
``` |
|
|
|
|
|
## More Details |
|
|
For more details refer to our paper at: https://www.sciencedirect.com/science/article/pii/S1574954125000901 |
|
|
|
|
|
## Citation |
|
|
``` |
|
|
@misc{heinrich2024audioprotopnet, |
|
|
title={AudioProtoPNet: An interpretable deep learning model for bird sound classification}, |
|
|
author={René Heinrich and Lukas Rauch and Bernhard Sick and Christoph Scholz}, |
|
|
year={2024}, |
|
|
url={https://www.sciencedirect.com/science/article/pii/S1574954125000901}, |
|
|
} |
|
|
``` |
