README.md

---
library_name: transformers
tags:
- audio
- spatial
- speech
license: mit
datasets:
- agkphysics/AudioSet
metrics:
- accuracy
pipeline_tag: feature-extraction
---

# Model Card for Model ID

GRAM (General Purpose Audio Representation Model) is trained on AudioSet with newly proposed naturalistic training methadology.
GRAMs utilize MWMAE (Multi-window multi-head attention), and RIR augmentations to achieve state-of-the-art results on downstream tasks such as FSD50K, ESC50, VL even when conditions are very adverse. What is different about GRAMs is the spatial reasoning capabilities, and the robustness to noise and reverberation.
GRAMs natively support binaural localization, and ambisonics localization tasks unlike other models in the literature. Therefore, GRAMs can be used for understanding spatial scenes as well as classifying sounds, or recognizing speech in very noisy and reverberant environments.


## Model Details

The GRAM learns spatial audio representation by reconstructing multi-channel masked spectrogram
patches. First, a patch extractor consisting of a single convolutional layer with 2D convolutional
filters divides each multi-channel spectrogram into n non-overlapping patches. Nonmasked patch embeddings are input to the encoder, for which we selected the 12-layer ViT-Base
(ViT-B) Transformer (Dosovitskiy et al., 2021) similar to Huang et al. (2022); Yadav et al. (2024).
The encoder outputs patch representations that can be further used for fine-tuning.
### Model Description

GRAM is a self-supervised, multi-channel masked auto-encoder model that efficiently learns spatial general-purpose audio representations
from simulated real-world sound scenes. To train GRAM, we developed a custom pipeline which
makes use of the Soundspace 2.0 platform (Chen et al., 2022a) to simulate high-quality real-world
sound scenes from AudioSet (Gemmeke et al., 2017), and of WHAMR! (Maciejewski et al., 2020)
for adding background noise. We present two versions of GRAM to ensure flexible application across audio formats:
GRAM-Binaural for two-channel audio clips, and GRAM-Ambisonics for four-channel audio clips in
the first-order Ambisonics format.


- **Developed by:** Goksenin Yuksel, goksenin.yuksel@ru.nl
- **Model type:** Transformers, Audio Foundation Models
- **Language(s) (NLP):** GRAMs support all languages, but mainly English.
- **License:** MIT

### Model Sources

- **Repository:** https://github.com/labhamlet/GRAM-T
- **Paper:** https://arxiv.org/abs/2506.00934

## Uses

GRAMs can be used as a powerful feature extractor for downstream tasks such as enviromental sound classification, speech recognition, speaker counting, sound localization. 
Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model.


## How to Get Started with the Model


GRAMs have three strategies to choose from; "raw", "mean" or "cls". We advise to use "raw" strategy as this prooduces embeddings over the time frames, and is more robust to silent parts.

Raw : Break audio clips into non-overlapping 2 second chunks, concatenating the features in time and finally taking a mean over the time axis to generate a fixed vector representation independent of the input audio duration
Mean : Break audio clips into non-overlapping 2 second chunks, and taking a mean over all the patches to generate a fixed vector representation independent of the input audio duration
Cls : Break audio clips into non-overlapping 2 second chunks, and return the "[CLS]" token representation.
~~~python
from transformers import AutoModel, AutoFeatureExtractor

model = AutoModel.from_pretrained("labhamlet/gramt-mono", trust_remote_code=True)
extractor = AutoFeatureExtractor.from_pretrained("labhamlet/gramt-mono", trust_remote_code=True)

audio = torch.zeros([1,320000])
extracted = extractor(audio, return_tensors="pt")
log_mel = extracted['input_values']
print(model(log_mel, strategy = "raw").shape)
~~~

## Training Details


### Training Data

The 85,000 naturalistic scenes were split into a train set of 70,000 scenes (corresponding to 70 Matterport3D houses), and a test set of 15,000 scenes (15
Matterport3D houses) for down-stream evaluation (see Section 3.4). We used the 70,000 naturalistic
scenes in the train set to generate naturalistic scenes for all audio clips in the unbalanced training
set of AudioSet (10-second sound tracks of 1.74 million YouTube videos (Gemmeke et al., 2017)).
Specifically, during training we randomly paired an AudioSet clip with a noise sound clip from the
WHAMR! background noise database (Maciejewski et al., 2020). WHAMR! noise clips longer than
10 s were trimmed to 10 s duration and a linear fade-in/fade-out of 200 ms was applied to every noise
clip prior to mixing of the sound scene.
To create a naturalistic sound scene, we then convolved the AudioSet clip either with BRIR(s, r, θ)
for GRAM-Binaural, or with a ARIR(s, r, θ) for GRAM-Ambisonics, to obtain T. Similarly, we
convolved the WHAMR! noise clip with the to obtain naturalistic scenes.


### Training Procedure

We transformed the channels of each sound scene (i.e., the waveforms) into logscale mel spectrograms using 128 mel filters in the frequency range of 50-16000 Hz with a 25 ms
Hanning window and 10 ms hop length, resulting in spectrograms of dimension 1024 × 128. For
GRAM-Ambisonic, we extracted normalized active Intensity Vectors (IVs) from the spectrograms
as additional input features encoding spatial information. We concatenated mel
spectrograms and intensity vectors, resulting in input x = [xmel, IV s] for each naturalistic scene
generated from an AudioSet clip. In-batch sampling: As the online mixing of naturalistic acoustic scenes is computationally expensive
due to multiple long convolutions, we used a random in-batch sampling procedure to increase the
effective batch size in a computationally efficient manner. We randomly sampled 16 partially
overlapping segments of 2 seconds to create 16 samples of dimension 200 × 128. This increases the
original batch size of 96 to an effective batch size of 1536. 

For pre-training, we divided the binaural spectrogram into 2×8×16,ambisonics spectrograms into 7×8×16 patches. We used an adapted version
of the mask-based framework of MW-MAE (Yadav et al., 2024), randomly selecting a subset of
n patches M1, . . . , Mn for i = 1, . . . , n for masking (masking ratio = 0.8) and replacing their
embedding with a learnable mask token. Finally, we added fixed sinusoidal positional embeddings to
all embedded patches.


We trained all GRAMs for 500 K steps on an H100 92 GB GPU machine
with 16 CPU cores. We used the AdamW optimizer (Loshchilov & Hutter, 2017) with weight decay
rate of 0.01, gradient clipping, and a cosine learning rate scheduler with 10 K steps warm-up. The
initial learning rate was set to 0.0002, and decayed to 0. We optimize the mean squared error (MSE)
loss function between the predicted masked patches and their corresponding input spectrogram
patches.

#### Preprocessing

Firstly, RMS Normalization was applied to audio clips to get all of them in the same loudness levels. Later, instance normalization was applied to the convolved scenes


#### Training Hyperparameters

- **Training regime:**: GRAMs were trained with mixed precision, torch.compile and flash attention.

## Evaluation

We evaluate GRAM and other state-of-the-art models on the HEAR benchmark
task suite, which presents a wide range of tasks to evaluate the downstream performance of audio
representation models (Turian et al., 2022). We additionally evaluated performance on simulated real-worldsound scenes using Nat-HEAR.
### Testing Data, Factors & Metrics

#### Testing Data

**HEAR**:  The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. 
HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.

**NatHEAR**: Orovides a naturalistic version of all selected datasets in the HEAR benchmark suite in
two audio formats: a two-channel, binaural format and a four-channel, first-order Ambisonics
format. Weincluded sound localization tasks for two different domains which we generated using
HEAR benchmark datasets: A speech localization task based on SC-5, and an environmental sound
localization task based on ESC-50. The localization tasks are modeled as a multi-output regression
task in which model outputs represent the estimated 3D Cartesian coordinates [x, y, z] on the unit
sphere (Adavanne et al., 2018). Finally, to assess the transferability of GRAM to real-world sound
scenes, we evaluate also on the sound event detection and localization tasks in TUT Sound Events
2018 REAL (Adavanne et al., 2019)


### Results


| Model | DCASE | FSD50K | LC | ESC-50 | CD | VL | SC-5 | NS | BO | Mri-S | Mri-T | s(m) |
|-------|-------|--------|-------|--------|-------|-------|------|-----|-------|-------|-------|------|
| HEAR-Naive | 8.8 | 13.2 | 43.5 ± 1.6 | 28.6 ± 3.1 | 38.0 ± 2.3 | 14.8 ± 3.0 | 13.3 | 87.6 | 98.7 ± 1.9 | 94.1 ± 0.5 | 87.6 ± 6.4 | 0.0 |
| Wav2Vec 2.0 | 23.5 | 29.4 | 69.9 ± 2.1 | 46.4 ± 1.8 | 57.3 ± 1.1 | 34.9 ± 2.4 | 85.3 | 17.4 | 81.4 ± 4.8 | 90.7 ± 0.8 | 77.0 ± 0.9 | 31.5 |
| HuBERT | 78.3 | 32.8 | 63.3 ± 1.2 | 58.6 ± 2.8 | 71.2 ± 1.2 | 65.2 ± 2.9 | 94.0 | 19.8 | 93.2 ± 5.9 | 94.6 ± 0.4 | 85.0 ± 2.5 | 44.7 |
| WavLM | 27.0 | 25.7 | 61.3 ± 2.3 | 49.5 ± 3.8 | 64.3 ± 1.3 | 60.1 ± 3.2 | 93.8 | 18.2 | 84.3 ± 6.3 | 88.8 ± 1.0 | 76.8 ± 0.5 | 36.8 |
| MAE | – | 33.4 | 62.3 ± 1.1 | 72.9 ± 2.1 | 60.8 ± 1.8 | 21.3 ± 5.8 | 66.6 | 63.6 | 94.5 ± 5.6 | 94.8 ± 0.6 | 85.1 ± 10.4 | 32.7 |
| SSAST* | – | 21.4 | 57.8 ± 3.3 | 58.3 ± 2.6 | 48.0 ± 2.1 | 15.4 ± 2.6 | 22.0 | 64.2 | 95.8 ± 4.3 | 90.2 ± 5.9 | 89.1 ± 8.0 | 15.8 |
| BEATs | – | 54.1 | 77.8 ± 1.2 | 85.8 ± 2.9 | 66.9 ± 2.5 | 39.7 ± 4.3 | 86.9 | 68.6 | 94.1 ± 3.5 | 95.5 ± 0.4 | 96.6 ± 0.5 | 61.4 |
| MW-MAE | 94.2 | 51.8 | 80.3 ± 1.9 | 82.2 ± 3.2 | 74.4 ± 1.5 | 45.5 ± 1.7 | 91.6 | 69.4 | 95.8 ± 4.3 | 97.5 ± 0.4 | 97.6 ± 0.6 | 71.0 |
| SSAM | 87.3 | 53.5 | 75.5 ± 1.4 | 82.9 ± 3.6 | 70.2 ± 0.4 | 56.4 ± 5.2 | 89.3 | 72.6 | 93.2 ± 3.5 | 97.8 ± 0.5 | 96.9 ± 0.5 | 71.2 |
| **GRAM-Binaural** | 95.6 | 56.1 | 81.0 ± 1.1 | 86.7 ± 2.4 | 75.0 ± 1.4 | 53.2 ± 3.0 | 92.5 | 77.0 | 94.9 ± 3.2 | 97.3 ± 0.3 | 98.1 ± 0.2 | 74.6 |
| **GRAM-Ambisonics** | 94.3 | 53.0 | 79.4 ± 1.5 | 85.9 ± 1.5 | 71.9 ± 1.9 | 53.7 ± 1.2 | 89.6 | 73.8 | 94.9 ± 4.9 | 97.6 ± 0.5 | 98.5 ± 0.4 | 73.4 |
| **GRAM-Mono** | 95.3 | 56.8 | 81.3 ± 1.8 | 87.5 ± 2.3 | 75.1 ± 0.6 | 57.3 ± 3.4 | 93.5 | 75.8 | 95.8 ± 3.7 | 97.4 ± 0.3 | 98.0 ± 0.2 | 76.1 |

#### Summary

We present a General-purpose, Real-world Audio representation Model (GRAM), which learns spatial
audio representations using a multi-channel masked auto-encoder approach. GRAM demonstrates
remarkable performance in naturalistic sound scenes as well as clean sound scenes, surpassing all
state-of-the-art self-supervised spectrogram-based audio foundation models while requiring only a
fraction of the training data. Moreover, GRAM is the first audio foundation model that is available
in both a two-channel, binaural format and a four-channel, first-order ambisonics format

## Model Card Contact

Goksenin Yuksel; goksenin.yuksel@ru.nl