Update README.md

README.md

@@ -1,199 +1,179 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- audio
+- speech
+- waveform
+license: mit
+datasets:
+- agkphysics/AudioSet
+metrics:
+- accuracy
+pipeline_tag: feature-extraction
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
+WavJEPA, a waveform-based version of the Joint-Embedding Predictive Architecture. WavJEPA leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level. We show that
+this approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources. Additionally, to overcome the
+performance drop that time-domain models typically exhibit in noisy and reverberant real-world acoustic environments, we present WavJEPA-Nat
 ## Model Details
 
-### Model Description
+The WavJEPA framework comprises a waveform encoder, context encoder, target encoder and a predictor. WavJEPA’s objective is to predict latent
+representation of various targets blocks based on a single context block extracted from the same
+sound wave. As waveform encoder, we use the feature encoder of Wav2Vec 2.0, which is composed
+of stacked temporal convolution layers (Baevski et al., 2020). Similar to the original I-JEPA architecture (Assran et al., 2023), a Vision Transformer (ViT) (Dosovitskiy et al., 2021) is used for the
+target encoder, context encoder and predictor. 
 
-<!-- Provide a longer summary of what this model is. -->
+### Model Description
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+WavJEPA is the first framework applying semantic learning to general-purpose audio representations in the time domain, surpassing state-of-the-art time-domain approaches on the HEAR (Turian
+et al., 2022) and ARCH (La Quatra et al., 2024) benchmark suites while requiring only a fraction
+of the computational resources.  WavJEPA leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level. We show that
+this approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources.
+Additionally, we address the degraded performance of time-domain
+models in real-world sound scenes with WavJEPA-Nat, a multi-channel extension of the WavJEPA
+framework trained on simulated real-world sound scenes. Evaluation on Nat-HEAR (Yuksel et al.,
+2025), a naturalistic version of the HEAR benchmark suite, demonstrates that WavJEPA-Nat exceeds the robustness of other time-domain foundation models to noise and reverberation. 
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
 
-### Model Sources [optional]
+- **Developed by:** Goksenin Yuksel, goksenin.yuksel@ru.nl
+- **Model type:** Transformers, Audio Foundation Models, Raw Waveform Models
+- **Language(s) (NLP):** WavJEPA and WavJEPA-Nat support all languages, but mainly English.
+- **License:** MIT
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/labhamlet/wavjepa
+- **Paper:** https://arxiv.org/abs/2509.23238
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+WavJEPA can be used as a powerful feature extractor for downstream tasks such as enviromental sound classification, speech recognition, speaker counting etc.
+Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model.
 
-[More Information Needed]
 
-### Recommendations
+## How to Get Started with the Model
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-## How to Get Started with the Model
+~~~python
+from transformers import AutoModel, AutoFeatureExtractor
 
-Use the code below to get started with the model.
+model = AutoModel.from_pretrained("labhamlet/wavjepa-base", trust_remote_code=True)
+extractor = AutoFeatureExtractor.from_pretrained("labhamlet/wavjepa-base", trust_remote_code=True)
 
-[More Information Needed]
+audio = torch.zeros([1,160000])
+extracted = extractor(audio, return_tensors="pt")
+audio_feature = extracted['input_values']
+print(model(audio_feature).shape)
+~~~
 
 ## Training Details
 
-### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+### Training Data
 
-[More Information Needed]
+We train WavJEPA on the unbalanced training set of AudioSet, which consists of 1.74 million 10-second sound clips scraped from YouTube (Gemmeke
+et al., 2017). 
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Each sound clip was resampled to 16 kHz and mean centered to enforce equal loudness
+across sound clips. We then randomly sampled 8 sections of 2 s from each sound clip, effectively increasing the batch size by a factor of 8 in a computationally efficient manner. Finally, each instance
+is instance normalized (Ulyanov et al., 2017). The waveform encoder converts each 2 s instance into
+an embedding w
+200×768, effectively resampling the audio to 100 Hz with a stride of 10 ms and a
+receptive field size of 12.5 ms
 
-#### Preprocessing [optional]
+We sampled starting indices for the context block with p = 0.065 and for target blocks
+with p = 0.025. We set M to 10 for both context block and target block . To update the target encoder
+parameters ∆, we linearly increased τ from τ0 = 0.999 to τe = 0.99999 over the first 100,000 steps,
+after which τ was kept constant. We used K = 8 for the top K averaging.
+We trained WavJEPA for 375,000 steps using a batch size of 32 on two NVIDIA H100 94 GB
+GPUs. Given our in-batch sampling factor of 8, we boost our effective batch size to 256. We use
+the AdamW optimizer (Loshchilov & Hutter, 2019) with a weight decay coefficient λw = 0.04. The
+learning rate schedule follows a cosine decay with linear warm-up over 100,000 steps, reaching a
+peak learning rate of 2 × 10−4 before decaying to zero
 
-[More Information Needed]
+#### Preprocessing
 
+RMS Normalization was applied to audio clips to get all of them in the same loudness levels, and later instance normalization is applied.
 
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Training Hyperparameters
 
-[More Information Needed]
+- **Training regime:**: WavJEPA and WavJEPA-Nat were trained with mixed precision, torch.compile and flash attention.
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
+We evaluate WavJEPA and other state-of-the-art models on the HEAR and ARCH benchmark
+task suite, which presents a wide range of tasks to evaluate the downstream performance of audio
+representation models (Turian et al., 2022). 
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+**HEAR**:  The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
+HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. 
+HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.
 
-[More Information Needed]
+**ARCH**: ARCH, a comprehensive benchmark for evaluating ARL methods on diverse audio classification domains, covering acoustic events, music, and speech. ARCH comprises 12 datasets, that allow us to thoroughly assess pre-trained SSL models of different sizes. 
+ARCH streamlines benchmarking of ARL techniques through its unified access to a wide range of domains and its ability to readily incorporate new datasets and models.
 
 ### Results
 
-[More Information Needed]
 
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
+**HEAR**
+
+| Model | Size | DCASE | FSD50K | LC | ESC-50 | CD | VL | SC-5 | NS | BO | Mri-S | Mri-T | s(m) |
+|-------|------|-------|--------|-----|--------|-----|-----|------|-----|-----|-------|-------|------|
+| **Baseline** |
+| HEAR-Naive | N/A | 7.6 | 12.5 | 40.3 ± 1.2 | 27.4 ± 3.3 | 36.7 ± 2.5 | 16.0 ± 3.4 | 13.3 | 89.2 | 97.1 ± 3.2 | 94.2 ± 1.1 | 93.7 ± 0.3 | 0.0 |
+| **Speech pre-training** |
+| Wav2Vec2.0 | B | 23.5 | 29.4 | 69.9 ± 2.1 | 46.4 ± 1.8 | 57.3 ± 1.1 | 34.9 ± 2.4 | 85.3 | 17.4 | 81.4 ± 4.8 | 90.7 ± 0.8 | 77.0 ± 0.9 | 30.9 |
+| HuBERT | B | 78.0 | 32.8 | 63.3 ± 1.2 | 58.6 ± 2.8 | 71.2 ± 1.2 | 65.2 ± 2.9 | 94.0 | 19.8 | 93.2 ± 5.9 | 94.6 ± 0.4 | 85.0 ± 2.5 | 47.3 |
+| WavLM | B | 27.0 | 25.7 | 61.3 ± 2.3 | 49.5 ± 3.8 | 64.3 ± 1.3 | 60.1 ± 3.2 | 93.6 | 16.0 | 84.3 ± 6.3 | 88.8 ± 1.0 | 76.8 ± 0.5 | 35.1 |
+| Data2Vec | B | 46.5 | 15.2 | 47.9 ± 1.2 | 28.0 ± 2.8 | 55.7 ± 1.0 | 44.9 ± 3.1 | 88.5 | 14.0 | 78.4 ± 4.1 | 85.1 ± 0.7 | 70.5 ± 3.3 | 23.6 |
+| Wav2Vec2.0 | L | 66.0 | 34.8 | 64.6 ± 1.9 | 59.8 ± 1.5 | 65.7 ± 0.8 | 53.3 ± 6.3 | 75.8 | 40.6 | 93.6 ± 2.6 | 94.8 ± 0.5 | 82.4 ± 3.0 | 42.5 |
+| HuBERT | L | 34.8 | 31.4 | 63.8 ± 1.3 | 60.4 ± 3.0 | 71.0 ± 1.2 | 69.0 ± 2.8 | 84.8 | 20.4 | 93.6 ± 3.0 | 95.3 ± 0.8 | 82.5 ± 2.0 | 44.3 |
+| WavLM | L | 77.4 | 40.1 | 69.4 ± 2.1 | 66.6 ± 2.5 | 76.3 ± 2.2 | 79.2 ± 3.9 | 93.8 | 18.2 | 93.6 ± 5.4 | 95.8 ± 0.8 | 90.1 ± 1.0 | 58.1 |
+| Data2Vec | L | 40.8 | 18.7 | 50.9 ± 1.7 | 34.4 ± 2.5 | 62.8 ± 1.6 | 60.0 ± 4.9 | 86.1 | 14.4 | 80.1 ± 8.5 | 84.7 ± 2.6 | 65.6 ± 3.1 | 29.0 |
+| **AudioSet pre-training** |
+| Wav2Vec2.0 | B | 52.0 | 34.7 | 60.4 ± 1.7 | 58.9 ± 1.9 | 56.3 ± 1.3 | 27.9 ± 4.6 | 72.1 | 42.0 | 86.0 ± 9.6 | 92.9 ± 1.4 | 77.3 ± 0.5 | 31.9 |
+| HuBERT | B | 86.2 | 41.1 | 63.5 ± 3.4 | 69.1 ± 1.6 | 69.5 ± 1.2 | 53.3 ± 3.1 | 83.5 | 38.8 | 91.5 ± 8.8 | 95.6 ± 0.5 | 90.4 ± 0.8 | 51.1 |
+| Wav2Vec2.0 | L | 82.6 | 47.8 | 73.6 ± 1.2 | 72.6 ± 2.1 | 68.2 ± 1.7 | 42.2 ± 6.0 | 83.9 | 30.8 | 91.5 ± 5.0 | 96.5 ± 0.3 | 88.7 ± 2.5 | 55.9 |
+| HuBERT | L | 86.2 | 45.4 | 75.2 ± 1.4 | 66.3 ± 4.6 | 70.1 ± 0.8 | 39.6 ± 3.6 | 85.7 | 38.6 | 91.6 ± 9.6 | 97.3 ± 0.5 | 89.6 ± 2.3 | 57.7 |
+| **WavJEPA** | B | 93.9 | 54.4 | 76.7 ± 2.4 | 86.5 ± 3.3 | 71.0 ± 0.8 | 49.8 ± 3.4 | 90.0 | 34.4 | 89.4 ± 5.4 | 97.3 ± 0.4 | 88.5 ± 0.5 | 66.0 |
+
+**ARCH**
+
+| Model | Size | ESC-50 | US8K | FSD50K | VIVAE | FMA | MTT | IRMAS | MS-DB | RAVDESS | AM | SLURP | EMOVO | s(m) |
+|-------|------|--------|------|--------|-------|-----|-----|-------|-------|---------|-----|-------|-------|------|
+| **Baseline** |
+| HEAR-Naive | N/A | 13.0 | 36.0 | 2.2 | 22.0 | 39.0 | 9.9 | 19.9 | 35.2 | 22.6 | 45.7 | 5.4 | 18.4 | 0.0 |
+| **Speech pre-training** |
+| Wav2Vec2.0 | B | 45.7 | 55.5 | 19.4 | 31.5 | 50.5 | 37.6 | 35.1 | 66.1 | 55.3 | 86.4 | 14.4 | 31.8 | 49.7 |
+| WavLM | B | 49.9 | 61.8 | 17.6 | 36.3 | 48.7 | 34.9 | 32.6 | 54.2 | 67.9 | 99.5 | 31.0 | 43.1 | 68.0 |
+| HuBERT | B | 58.9 | 67.3 | 24.5 | 40.5 | 54.6 | 38.8 | 36.7 | 58.5 | 65.3 | 99.6 | 33.8 | 40.5 | 59.7 |
+| Data2Vec | B | 23.6 | 45.6 | 10.1 | 30.2 | 40.6 | 27.6 | 25.9 | 50.7 | 48.0 | 99.1 | 43.6 | 27.3 | 38.8 |
+| Wav2Vec2.0 | L | 13.1 | 42.7 | 5.8 | 22.0 | 41.7 | 21.0 | 19.9 | 50.2 | 11.6 | 45.7 | 7.3 | 19.3 | 8.6 |
+| WavLM | L | 67.2 | 70.9 | 32.2 | 42.5 | 61.1 | 41.3 | 42.5 | 68.0 | 71.8 | 99.8 | 42.3 | 45.3 | 75.8 |
+| HuBERT | L | 64.0 | 70.0 | 29.5 | 41.0 | 54.8 | 38.4 | 36.8 | 64.1 | 72.6 | 99.9 | 45.3 | 43.8 | 81.5 |
+| Data2Vec | L | 25.4 | 49.2 | 10.8 | 30.6 | 43.5 | 28.5 | 27.1 | 44.2 | 45.1 | 99.2 | 28.6 | 23.1 | 35.1 |
+| **AudioSet pre-training** |
+| W2V2 | B | 52.6 | 70.5 | 21.3 | 31.3 | 59.5 | 37.9 | 35.9 | 64.6 | 45.9 | 88.1 | 11.0 | 30.8 | 53.8 |
+| HuBERT | B | 68.8 | 79.1 | 31.1 | 40.1 | 65.9 | 43.4 | 47.7 | 67.8 | 63.5 | 98.8 | 20.5 | 33.4 | 75.5 |
+| Wav2Vec 2.0 | L | 74.4 | 79.0 | 37.6 | 39.7 | 66.6 | 44.5 | 49.9 | 76.9 | 59.5 | 99.4 | 17.7 | 38.2 | 80.0 |
+| HuBERT | L | 71.5 | 75.6 | 37.4 | 44.3 | 67.5 | 43.4 | 50.5 | 77.8 | 73.3 | 99.6 | 20.5 | 38.6 | 83.9 |
+| **WavJEPA** | B | 83.9 | 83.5 | 48.0 | 44.06 | 68.2 | 46.0 | 59.0 | 79.5 | 62.5 | 99.5 | 23.3 | 46.6 | 92.3 |
 
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+#### Summary
 
-[More Information Needed]
+We presented WavJEPA, a state-of-the-art audio foundation model that leverages self-supervised semantic learning to obtain robust general-purpose audio representations from raw waveforms.
+WavJEPA’s results highlight the superior performance of semantic audio representation learning in comparison with representation learning at the speech unit or token level, as is common in existing
+time-domain speech representation learning approaches.
 
 ## Model Card Contact
 
-[More Information Needed]
+Goksenin Yuksel; goksenin.yuksel@ru.nl