---
library_name: transformers
tags:
- arxiv:2609.34839
- bioacoustics
- audio
- wav2vec2
- self-supervised-learning
- dolphin
- bottlenose-dolphin
- whistle
- openwhistle
pipeline_tag: feature-extraction
datasets:
- dolphinteam/OpenWhistle-Pretraining
---

# OpenWhistle BioClust

`dolphinteam/OpenWhistle-BioClust` provides a Wav2Vec2 audio backbone pretrained from scratch on bottlenose dolphin whistle recordings using native Fairseq on three NVIDIA L40S GPUs at Bioclust.

It follows the repository layout and frontend geometry of [OpenWhistle-Wav2Vec2.0](https://huggingface.co/dolphinteam/OpenWhistle-Wav2Vec2.0). The weights come from the completed Bioclust run at **400,000 optimizer updates**.

## Model Details

- Model type: `wav2vec2`
- Transformers architecture: `Wav2Vec2ForPreTraining`
- Hidden size: 768; Transformer layers: 12; attention heads: 8
- Seven convolutional layers, with strides `[15, 2, 2, 2, 2, 2, 2]`
- Total stride: 960 audio samples, approximately 21.77 ms at 44,100 Hz
- Sampling rate: 44,100 Hz; mono audio
- Weights: `model.safetensors`, exported in float32

The export maps every learned Fairseq tensor strictly and checks encoder embeddings against the native checkpoint. Conversion details and the checkpoint SHA-256 are recorded in `export_provenance.json`.

## Training and Evaluation Data

Pretraining uses [dolphinteam/OpenWhistle-Pretraining](https://huggingface.co/datasets/dolphinteam/OpenWhistle-Pretraining), pinned to revision `eea37423c7841b0f99a7c0380f945999c0a2c489`.

The manifests contain 28,410 training examples and 3,370 validation examples. Fairseq excludes two training examples shorter than five seconds. Existing 44.1 kHz mono WAVs were reused after checking their headers and metadata; 55 missing or invalid examples were reconstructed from the matching Arrow cache as PCM_24 FLAC files.

Training completed on 12 October 2026 with native Fairseq 0.12.2, PyTorch 2.5.1/CUDA 12.4, FP16, seed 42, 32,000 warmup updates and a peak learning rate of 0.0005. Each GPU has a maximum audio budget of 4,704,000 samples per update, with accumulation 1 and crops capped at 20 seconds. Validation and checkpointing occur every 5,000 updates.

The final native Fairseq validation loss is **0.853** at 400,000 updates. The best observed validation loss is **0.852** at 395,000 updates. This repository publishes the **final 400,000-update checkpoint**. These self-supervised losses do not measure downstream classification or detection performance.

The OpenWhistle supervised datasets are available for subsequent fine-tuning and evaluation:

- [OpenWhistle-Classification-Finetuning](https://huggingface.co/datasets/dolphinteam/OpenWhistle-Classification-Finetuning)
- [OpenWhistle-Detection-Finetuning](https://huggingface.co/datasets/dolphinteam/OpenWhistle-Detection-Finetuning)

## Intended Use

Use the backbone to extract audio representations or fine-tune a downstream model for bottlenose dolphin whistle classification and detection. This checkpoint has no supervised classification or detection head.

## Input Audio

Provide mono waveforms sampled at **44,100 Hz**. The feature extractor normalizes the waveform and uses right padding with value 0.0 and `return_attention_mask=False`.

For comparisons with the unpadded native encoder, process waveforms individually or batch equal-length waveforms. Resample audio recorded at other sampling rates before processing.

## Loading

```python
import torch
from transformers import AutoFeatureExtractor, AutoModel

repo_id = "dolphinteam/OpenWhistle-BioClust"
processor = AutoFeatureExtractor.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id).eval()

# waveform is a one-dimensional mono array sampled at 44,100 Hz.
inputs = processor(waveform, sampling_rate=44100, return_tensors="pt")
with torch.no_grad():
    embeddings = model(**inputs).last_hidden_state
```

To load the quantizer and projection heads as well:

```python
from transformers import Wav2Vec2ForPreTraining
model = Wav2Vec2ForPreTraining.from_pretrained(repo_id)
```

Transformers pretraining uses its own masking and loss implementation. The original training recipe and metrics come from native Fairseq.

## Files

- `config.json`: model architecture and Transformers configuration
- `model.safetensors`: converted encoder and pretraining-head weights
- `preprocessor_config.json`: audio preprocessing settings
- `trainer_state.json`: native Fairseq metrics reconstructed in the reference Trainer-state layout
- `export_provenance.json`: checkpoint identity, tensor mapping and embedding parity
- `training_provenance.json`: dataset revision, training settings and validation history

The native optimizer, scheduler and RNG states are retained in the Bioclust Fairseq checkpoint and are not included in this inference repository. `trainer_state.json` is a metrics record, not a Transformers training-resume artifact; training FLOPs were not tracked.

## Limitations

- Training focuses on bottlenose dolphin whistles; transfer to other species or recording settings requires evaluation.
- Downstream results for this checkpoint have not yet been established.
- Biological interpretations of embeddings require independent validation.

## License

No model-weight license has been specified, following the reference repository. Contact the OpenWhistle maintainers for licensing terms.

## Paper, collection, and authors

- Paper: [OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations](https://arxiv.org/abs/2609.34839)
- Hugging Face collection: [NeurIPS 2026 OpenWhistle](https://huggingface.co/collections/dolphinteam/neurips26-openwhistle)
- This repository: [https://huggingface.co/dolphinteam/OpenWhistle-BioClust](https://huggingface.co/dolphinteam/OpenWhistle-BioClust)

Authors: Faadil Mustun, Chiara Semenzin, Roberto Dessi, Pablo Robin Guerrero, Pierre Orhan, Alexis Emanuelli, Emanuele Rossi, Yair Lakretz, Gonzalo de Polavieja, and German Sumbre.

## Citation

```bibtex
@misc{mustun2026openwhistle,
  title         = {OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations},
  author        = {Mustun, Faadil and Semenzin, Chiara and Dessi, Roberto and Robin Guerrero, Pablo and Orhan, Pierre and Emanuelli, Alexis and Rossi, Emanuele and Lakretz, Yair and de Polavieja, Gonzalo and Sumbre, German},
  year          = {2026},
  eprint        = {2609.34839},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.34839}
}
```