Bearings

Bearings is a self-supervised encoder for first-order Ambisonic (FOA) audio. It learns the spatial attributes of a sound field, direction and diffuseness, by reconstructing masked intensity-vector and diffuseness targets from a 7-channel log-mel + active-intensity representation. The encoder needs its tokens fused with a content encoder such as GRAM-T to solve sound event localisation and detection (SELD).

• Repository: https://github.com/root-goksenin/Embisonics
• Developed by: Goksenin Yuksel, goksenin.yuksel@ru.nl
• License: MIT

Usage

With transformers (no clone needed, pip install transformers torchaudio):

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("labhamlet/bearings", trust_remote_code=True).eval()
wav = torch.randn(1, 4, int(48000 * 5.3))     # (batch, [W, Y, Z, X], samples), any sample rate

with torch.no_grad():
    out = model(wav, sampling_rate=48000)
out.last_hidden_state.shape   # (1, 600, 384): B x (T * F) x E, T = 75 time steps, F = 8 mel bands
out.timestamps.shape          # (600,) seconds, one per token, window centres: 0.04, ..., 5.96
out.n_valid_time              # tensor([67]): time steps that end before the zero padding

strategy="frames" returns (B, T, 8 * 384) per-frame vectors, strategy="mean" and strategy="cls" return one (B, 384) vector per clip. AutoFeatureExtractor.from_pretrained("labhamlet/bearings", trust_remote_code=True) does the resampling and padding separately if you want to cache waveforms.

With the research code, clone the repository and install requirements.txt, then:

import torch
from src.hub import load_bearings

model = load_bearings("labhamlet/bearings", "bearings-base.ckpt")
wav = torch.randn(1, 4, int(48000 * 5.3))     # (batch, [W, Y, Z, X], samples), any sample rate

out = model.encode_waveform(wav, sr=48000)
out["embeddings"].shape    # (1, 600, 384): B x (T * F) x E, T = 75 time steps, F = 8 mel bands
out["timestamps"].shape    # (600,) seconds, one per token, window centres: 0.04, ..., 5.96
out["n_valid_time"]        # 67: time steps that end before the zero padding

encode_waveform resamples to 32 kHz, applies per-clip W-RMS normalisation, zero-pads to a whole number of 2 s windows, encodes each window and concatenates along time. The AutoModel forward does exactly the same and the two paths give identical embeddings.

The token axis stacks the T x F grid time-major: token i is time step i // 8 and mel band i % 8. The 8 bands of a time step are adjacent and share a timestamp. Each time step covers 80 ms. embeddings.view(B, n_time, n_freq, E) restores the grid and .reshape(B, n_time, 8 * 384) gives the per-frame vectors the SELD probes consume.

Evaluation

Used as a frozen spatial stream next to a frozen content encoder in SELD probes on TAU-NIGENS 2021 and STARSS23 (DCASE 2023 metrics: ER20, F20, LE, LR). Scripts and task ids are in seld-dcase2023-gram/ of the repository.