--- license: apache-2.0 library_name: pytorch pipeline_tag: feature-extraction base_model: - DAS-DeepAudioSegmenter/perch2-torch tags: - audio - bioacoustics - embeddings - torch-export - das --- # PERCH 2 embeddings (PyTorch, embedding only) The PERCH 2 embedding network from [perch2-torch](https://huggingface.co/DAS-DeepAudioSegmenter/perch2-torch) as a single self-contained `torch.export` file. The 14,795-class label head is left out, so the file holds just the waveform frontend and EfficientNet-B3 backbone: 10.7 M instead of 101.7 M parameters, and 47.6 MB instead of 408 MB. For class logits, use [perch2-torch](https://huggingface.co/DAS-DeepAudioSegmenter/perch2-torch). ## Use the model file `perch_v2_embed.pt2` takes a batch of 5 s mono windows at 32 kHz, `(batch, 160000)` with batch 1-4096, and returns the unpooled spatial embedding `(batch, 16, 4, 1536)`: 16 time frames and 4 frequency positions per window. ```python import torch from huggingface_hub import hf_hub_download from torch.export.passes import move_to_device_pass path = hf_hub_download("DAS-DeepAudioSegmenter/perch2-torch-embed", "perch_v2_embed.pt2") core = move_to_device_pass(torch.export.load(path), "cpu").module() # or "cuda" / "mps" windows = torch.zeros(2, 160_000) # two 5 s windows of 32 kHz audio spatial = core(windows) # (2, 16, 4, 1536) embedding = spatial.mean(dim=(1, 2)) # (2, 1536), one embedding per window ``` Requires PyTorch >= 2.11. `torch.export` files load in the version they were exported with or newer; tested with 2.11 and 2.14. ## Limitations of the model file on its own - **Fixed 5 s input.** Each window must be exactly 160,000 samples. Shorter audio has to be zero-padded, and longer audio cut into windows by you. - **32 kHz assumed.** The model does not resample; whatever you pass is treated as 32 kHz audio. - **Per-window output only.** You get unpooled frames for each window. There is no song-level average or maximum, and no continuous time axis across overlapping windows, which needs the overlap frames trimmed and averaged. - **Memory.** All windows passed in one call (up to 4,096) are processed at once, so long recordings have to be split into batches. ## Alternative: `perch2_embed.py` for audio of any length `perch2_embed.py` handles all of the above: windowing, padding, batching, overlap pooling and truncation. It needs only torch and huggingface_hub. ```bash pip install torch huggingface_hub soundfile hf download DAS-DeepAudioSegmenter/perch2-torch-embed perch2_embed.py --local-dir . ``` ```python import soundfile as sf from perch2_embed import load, embed core = load("auto") # downloads and caches perch_v2_embed.pt2 wave, sr = sf.read("song.wav", dtype="float32") # the model assumes 32 kHz: resample first for normal speed out = embed(core, wave if wave.ndim == 1 else wave.mean(axis=1), sample_rate=sr) out["avg"].shape, out["max"].shape, out["temporal"].shape # (1536,), (1536,), (frames, 1536) ``` ### Options `load(device="cpu", path=None, revision=None)` - `device`: `"cpu"`, `"cuda"`, `"mps"`, or `"auto"` (CUDA, then MPS, then CPU). - `path`: a local copy of `perch_v2_embed.pt2`; otherwise it is downloaded from this repo and cached. - `revision`: a repo revision to pin the downloaded file. `embed(core, wave, hop_frames=1, max_seconds=None, sample_rate=32000, batch_size=32)` - `wave`: mono 1-D float array or tensor of any length, fed to the model unchanged (treated as 32 kHz). - `hop_frames`: window hop in 0.32 s frames, 1-15. `1` is the model's own hop and reproduces the full perch2-torch model; larger values compute fewer windows (see below). - `max_seconds`: keep only the first `max_seconds` of audio. `None` embeds everything. - `sample_rate`: the rate at which `wave` was sampled. Nothing is resampled: it only converts `max_seconds` to samples and frame positions to seconds in `times`. - `batch_size`: 5 s windows per forward pass, which sets memory use. ### Returns A dict with: - `avg` `(1536,)`: the same as the full model's `embedding` output; - `max` `(1536,)`: the maximum over the same overlap-trimmed frames (the full model does not output this); - `temporal` `(frames, 1536)`: stitched frames 0.32 s apart in model time, averaged over the 4 frequency positions; - `spatial` `(frames, 4, 1536)`: the same frames with the frequency positions kept; - `times` `(frames,)`: frame centres in seconds; - `n_windows`: the number of 5 s windows computed. ### Notes - The model always treats its input as 32,000 samples per second, so the sample rate of `wave` sets the playback speed the model hears. Audio at 32 kHz is heard at normal speed. Audio at a higher rate fed unchanged is slowed down and lowered in pitch by `sample_rate / 32000`. For example, a recording resampled to 64 kHz (e.g. `librosa.load(path, sr=64000)`) and fed unchanged is heard at half speed and half pitch, and a 44.1 kHz file read as is plays at 0.73x speed. Windows, hops and frames are fixed in samples: one 5 s window covers 160,000 / sample_rate seconds of your audio and frames are 10,240 / sample_rate seconds apart (0.32 s at 32 kHz, 0.16 s at 64 kHz). Pass the true `sample_rate` so that `max_seconds` and `times` are in seconds of the original audio. - `hop_frames > 1` computes fewer windows, but the embeddings change. On 75 bird songs of 8 s or longer (audio loaded at 64 kHz), cosine similarity to the default hop was: | hop | windows computed | avg: median (worst) | max: median (worst) | |---|---|---|---| | 2 (0.64 s) | 52% | 0.993 (0.889) | 0.978 (0.930) | | 4 (1.28 s) | 28% | 0.989 (0.888) | 0.982 (0.937) | | 8 (2.56 s) | 15% | 0.977 (0.879) | 0.983 (0.938) | | 15 (4.8 s) | 10% | 0.955 (0.840) | 0.969 (0.862) | Use the same hop for everything you compare. ## Verification Against perch2-torch (revision `d5d3bb6a`): - the core matches the full model's frontend and backbone: CPU difference <= 4e-6, MPS relative difference 2.4e-6; - `embed()` at the default hop matches the full model's windowing and pooling for inputs of 1 to 361 windows (avg within 3e-7, max and temporal within 1e-5), and `avg` equals the full model's `embedding` within 3e-7; - truncation equals embedding the pre-cut audio; - PyTorch 2.14 loads the file with timm unavailable and gives identical output. ## Attribution and license Exported from [perch2-torch](https://huggingface.co/DAS-DeepAudioSegmenter/perch2-torch), which is derived from the Apache-2.0 [PERCH ONNX model](https://huggingface.co/justinchuby/Perch-onnx) and the Apache-2.0 [wrice EfficientNet-B3 checkpoint](https://huggingface.co/wrice/perch-v2-efficientnet-b3). Apache-2.0.