DAS-DeepAudioSegmenter/whisperseg-aer

WhisperSeg AER checkpoints

WhisperSeg AER segments animal vocalizations with a choice of model sizes. Compact vocabulary and FP16 weights reduce storage and memory use; smaller architectures and shorter decoders also reduce computation. All release checkpoints bundle the model and tokenizer in one DAS-compatible .ckpt file.

The models use three approaches:

• Compact Base and Large: remove unused vocabulary rows and retain the numeric and special tokens needed for segmentation, then convert weights to FP16. Encoder and decoder depths stay unchanged. These conversions require no retraining; paired fitted-weight controls below quantify their accuracy cost.
• Pruned Base: keep all six encoder layers and train a decoder with six, four, two or one layers on the vad-animals training split, with the encoder frozen. Fewer decoder layers reduce size and inference cost, with some loss of accuracy or transfer performance.
• Tiny and Small: start from the multilingual Whisper Tiny and Whisper Small speech models, adapt their token vocabulary and audio positions for segmentation, and train their decoders on the same multispecies data with the encoder frozen. Their full encoder and decoder depths are retained. These are newly trained segmenters; their scores do not isolate the effect of compression against an original Tiny or Small WhisperSeg model.

All checkpoints contain FP16 weights and a compact vocabulary. cmp denotes vocabulary compaction, ms denotes the Tiny/Small multispecies-trained models, and 6e4d, for example, denotes six encoder and four decoder layers. The Base 6E+6D checkpoint is decoder-retrained; the plain compact Animal VAD Base checkpoint is converted without retraining. The released Tiny and Small models are multispecies-trained and have no nightingale-specific adaptation.

Model	Encoder / decoder	Parameters (M)	File (MiB)	Checkpoint
Tiny multispecies	4 / 4	17.89	34.5	Tiny
Animal VAD Base · original reference	6 / 6	72.1	275.1	Upstream reference
Animal VAD Base · compact	6 / 6	46.1	89.5	Base compact
Base 6E+6D · retrained	6 / 6	46.1	88.3	Base 6E+6D
Base 6E+4D	6 / 4	37.7	72.3	Base 6E+4D
Base 6E+2D	6 / 2	29.3	56.2	Base 6E+2D
Base 6E+1D	6 / 1	25.1	48.2	Base 6E+1D
Small multispecies	12 / 12	202.00	385.8	Small
Animal VAD Large · original reference	32 / 32	1,542.0	5,882.8	Upstream reference
Animal VAD Large · compact	32 / 32	1,477.1	2,820.9	Animal VAD Large
Large multispecies · original reference	32 / 32	1,542.0	5,882.8	Upstream reference
Large multispecies · compact	32 / 32	1,477.1	2,820.9	Large multispecies

Original references are linked for comparison and are not release files. Their file sizes refer to the original pytorch_model.bin; release sizes refer to .ckpt files.

Tiny is the smallest released model. Small improves accuracy over Tiny on their shared tests at the cost of more memory and computation. The pruned Base models offer intermediate sizes; their shared-input accuracy evaluation is being refreshed. Performance on unfamiliar sounds varies more, so choose using a test recording representative of your application.

Use with DAS

Install DAS, download a checkpoint, and pass its path to WhisperSegmenter. This example runs inference on one second of zero-valued audio; replace it with your recording:

import numpy as np
from das.whisperseg.model import WhisperSegmenter

segmenter = WhisperSegmenter("whisperseg-tiny-ms-cmp-fp16.ckpt", device="cpu")
prediction = segmenter.segment(
    np.zeros(16_000, dtype=np.float32),
    sr=16_000,
    num_trials=1,
    batch_size=1,
    num_beams=1,
    max_length=128,
)

Use a current DAS release that includes the Whisper generation memory fix. SHA256 checksums, source revisions, vocabulary sizes and weight precision are provided in manifest.json.

Evaluation and comparability

We distinguish transfer to an unseen species (before any training on that species) from fine-tuning (performance after learning from labeled development audio of that species). Nightingale is absent from the multispecies training data. Events use one-to-one matching at IoU ≥ 0.5. All current evaluations use batch 1, three offset decoding trials, four beams, max_length=128, and the generation fix force_unique_generate_call=True in both generation paths. The fix changes inference, not training or model weights.

Release models on shared inputs

The earlier Base/Large scores and resource measurements used pre-fix inference. They are superseded, and are being replaced rather than combined with newer Tiny/Small evaluations. All 12 release/reference checkpoints are queued on one M2 Max host (32 GiB, PyTorch 2.12.0, Transformers 4.57.1) using identical inputs:

• Multispecies: the same fixed 65-file, 920-event subset of the vad-animals test split; unweighted macro event F1 across four species.
• Unseen-species transfer: the same continuous 944.937-second recording, with 2,870 events; pooled event F1. This full recording is used only for multispecies checkpoints. It includes development audio and must not be used to test species-adapted checkpoints.

Completed shared-input evaluation: all 12 existing release/reference models, 24 evaluations. Training histories differ; legacy 6D/1D rows are not matched-training controls.

Model	Multispecies macro F1 ↑	Unseen-species F1, 945 s ↑	Peak MPS GiB ↓	Inference s/audio s ↓
Tiny multispecies	0.9280	0.7462	0.122	0.330
Small multispecies	0.9589	0.7809	1.027	0.572
Animal VAD Base · original FP32 reference	0.7041	0.5948	1.311	0.353
Animal VAD Base · compact FP16	0.7041	0.5950	0.277	0.316
Base 6E+6D · legacy 1,612-recording fit	0.9721	0.6967	0.230	0.312
Base 6E+4D	0.9713	0.7177	0.168	0.260
Base 6E+2D	0.9708	0.6842	0.129	0.199
Base 6E+1D · legacy 1,612-recording fit	0.9621	0.6597	0.129	0.178
Animal VAD Large · original FP32 reference	0.7004	0.7413	10.297	2.302
Animal VAD Large · compact FP16	0.7015	0.7409	6.360	1.757
Large multispecies · original FP32 reference	0.6997	0.5456	10.484	2.120
Large multispecies · compact FP16	0.7007	0.5445	6.555	1.620

Peak MPS driver allocation and inference time in this table refer to the shared full recording. Runtime excludes loading, reading and scoring and is a single timed pass, not a repeated throughput estimate. Memory includes allocator/framework overhead and is host-specific. Queues wait for other GPU work to finish before measuring resources.

Training provenance: what the comparisons isolate

The plain compact Base/Large release checkpoints are conversions of upstream segmentation-trained models, without additional fitting. The decoder-pruned Base models received additional multispecies fitting; Tiny/Small were segmentation-trained from speech checkpoints. These are useful off-the-shelf choices, but different prior training confounds comparisons across these groups. Identical test inputs do not make this a controlled architecture-only experiment.

For the separately fitted comparison in the paper, additional training uses the same 3,121 recordings (81,442 events), temporal 10% validation split, frozen encoder, effective batch 4, learning rate 3e-6, seed 0, plateau reduction and early stopping. The audit confirmed that the older 2D/4D Base fits use those exact data and split. The historical retrained 6D/1D release models instead used 1,612 recordings, despite summary metadata claiming 3,121. Their controlled accuracy comparisons are excluded pending replacement fits and adapted descendants on 3,121 recordings. Their release-model evaluations describe the existing files, with this training-history caveat; no model weights are replaced by this card update. Backbone initializations still differ. Large uses gradient accumulation to obtain effective batch 4; floating-point trajectories can differ from a physical batch of 4. Species adaptation uses the same development segment and temporal validation split for all models. Test audio is a disjoint 189-second held-out segment (530 events) and the continuous first 600 seconds from a separate bird (1,387 events). Those adapted results must not be substituted into the full-recording transfer column.

Paired conversion controls after fitting

These controlled tests use the same fitted Base weights, converted without retraining. They are experimental fitted checkpoints, not the unmodified upstream Base release reference. Multispecies scores use the 65-file test; the two species-specific scores use adapted weights on the held-out 189-second and separate-bird 600-second tests. Resources below refer to the latter test on M1 Pro, 32 GiB, PyTorch 2.13.0 and Transformers 4.57.1, and must remain separate from M2 Max or CUDA results.

Same fitted Base weights	Multispecies macro F1	Held-out after tuning F1	Across-bird after tuning F1	Peak MPS MiB	Inference s/audio s
Full vocabulary, FP32	0.9706	0.9397	0.9053	1317.7	0.495
Reduced vocabulary, FP32	0.9706	0.9397	0.9053	520.2	0.474
Reduced vocabulary, FP16	0.9695	0.9397	0.9029	260.1	0.456

Vocabulary removal alone loses 0.000 F1 percentage points on these tests; the subsequent FP16 conversion loses at most 0.242 points. Vocabulary reduction plus FP16 reduces Base weight storage by 68.0%; FP16 halves weight bytes, not FLOPs. Runtime is one pass including cold kernels. Pruning comparisons additionally involve fitting and do not isolate conversion alone.

CUDA and remaining controls

CUDA resource tests use a Tesla T4, one common 60-second input, one full warmup and five timed repeats. All source bundles include the generation fix. A replacement run uses a shared benchmark adapter and records generated-token traces; valid completed rows are reused. GPU allocated/reserved memory excludes driver/context; process RAM includes loading. CUDA resources are not CUDA accuracy scores, and cannot be paired with accuracy of different fitted/adapted weights.

Untrained Tiny/Small speech controls were excluded from the paper's resource comparison because their zero-event outputs made timing ratios misleading. Full-vocabulary Tiny/Small segmentation fits are running with the same additional training recipe, followed by paired vocabulary-only/FP16 tests and local/CUDA benchmarks. Pending results are not presented as measured. Only one new species and one separate-bird recording are tested; there are no repeated training seeds or population-level confidence intervals.

Citation

Please cite both papers when using any model in this repository:

• Gu et al. (2024), Positive Transfer of the Whisper Speech Transformer to Human and Animal Voice Activity Detection, ICASSP 2024 (WhisperSeg).
• Steinfath et al. (2021), Fast and accurate annotation of acoustic signals with deep neural networks, eLife 10:e68837 (DAS).