Instructions to use chungimungi/WavLate with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chungimungi/WavLate with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="chungimungi/WavLate", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("chungimungi/WavLate", trust_remote_code=True, device_map="auto") - sentence-transformers
How to use chungimungi/WavLate with sentence-transformers:
from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("chungimungi/WavLate", trust_remote_code=True) queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
WavLate
WavLate is a late-interaction (multi-vector) model for retrieval between audio clips and text captions. It keeps one vector for every 0.31 seconds of audio and one vector for every caption token, and scores a pair with MaxSim: each caption token finds its best-matching moment in the clip, and the score is the average of those best matches. Clips and captions are encoded independently, so every clip can be indexed once and searched with any caption. To our knowledge it is the first open late-interaction model trained for audio and caption retrieval.
One general model, with no per-dataset fine-tuning, it reaches a mean R@1 of 37.9 over the four AudioCaps and Clotho retrieval tasks, ahead of the eight other open audio and text models we evaluated on the same test pools.
Quick start
Using Sentence Transformers
pip install -U "sentence_transformers[audio]>=6.0.0" soundfile torchaudio
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("chungimungi/WavLate", trust_remote_code=True)
captions = ["A dog barks while birds chirp", "Rain falls on a tin roof"]
clips = [
"https://raw.githubusercontent.com/karolpiczak/ESC-50/master/audio/1-100032-A-0.wav",
"https://raw.githubusercontent.com/karolpiczak/ESC-50/master/audio/1-17367-A-10.wav",
]
text = model.encode_query(captions)
audio = model.encode_document(clips, batch_size=1)
print([tuple(embedding.shape) for embedding in text])
# [(7, 128), (6, 128)]
print([tuple(embedding.shape) for embedding in audio])
# [(16, 128), (16, 128)]
scores = model.similarity(text, audio)
print(scores.cpu())
# tensor([[0.3718, 0.1570],
# [0.1569, 0.4258]])
To use your own recordings, replace the ESC-50 URLs
with local audio paths or URLs. Files are automatically converted to mono and resampled to 32 kHz.
When passing raw waveforms instead, include their sample rate:
{"audio": {"array": wave, "sampling_rate": rate}}.
Captions receive the [Q] prefix automatically and can then be compared with audio using
model.similarity(text, audio). Always pass text embeddings first, even when retrieving
captions for audio.
Using Transformers
import soundfile as sf
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained("chungimungi/WavLate", trust_remote_code=True).eval()
captions = ["A dog barks while birds chirp", "Rain falls on a tin roof"]
wave, rate = sf.read("clip.wav", dtype="float32") # any length; mono or stereo
if wave.ndim > 1:
wave = wave.mean(axis=1)
with torch.inference_mode():
text, text_mask = model.encode_text(captions) # [2, tokens, 128]
audio, audio_mask = model.encode_audio([wave], sampling_rate=rate) # [1, frames, 128]
scores = model.score(text, text_mask, audio, audio_mask) # [2 captions, 1 clip]
print(scores)
encode_textreturns one 128-dimensional unit vector per caption token and a mask of the tokens that count (padding and special tokens are excluded).encode_audioexpects mono audio at 32 kHz. Passsampling_rateto resample other rates (this needstorchaudio). Every 10 second window gives 32 frame vectors. Clips longer than 10 seconds are split into consecutive windows and all their frames are kept, up to 30 seconds.scorereturns mean MaxSim scores of shape[number of captions, number of clips]. Higher means a better match. The same score ranks clips for a caption (text to audio) and captions for a clip (audio to text).- Tested with
transformers5.23 andtorch2.11. The text tower needstransformers4.48 or newer (ModernBERT).
Model
| Part | Details |
|---|---|
| Audio tower | HTS-AT (Swin transformer on log-mel spectrograms, 31M parameters) with MGA-CLAP's locality-aware output, initialised from MGA-CLAP |
| Audio vectors | 32 per 10 s window, MGA-CLAP frame projection (768 to 1024), then a linear projection to 128 dimensions, L2-normalised |
| Text tower | mLateOn: ModernBERT backbone with ColBERT projection layers (768 to 1536 to 768 to 128) |
| Text vectors | one 128-dimensional L2-normalised vector per token; captions are prefixed with the [Q] marker |
| Score | mean MaxSim from caption tokens to audio frames |
| Parameters | 344M in total, of which 197M are the text tower's token embedding table |
Results
R@1 / R@10 (%) on AudioCaps test (964 clips, 4,820 captions) and Clotho evaluation (1,045 clips, 5,225 captions). A2T means audio to text (a clip finds its captions), T2A means text to audio (a caption finds its clip). All baselines except the last row were run by us on the same test pools with their released checkpoints and their own preprocessing. Bold marks the best value in each column among the models we ran.
| Model | AudioCaps A2T | AudioCaps T2A | Clotho A2T | Clotho T2A | Mean |
|---|---|---|---|---|---|
| WavLate | 53.6 / 93.5 | 43.6 / 87.9 | 30.0 / 71.0 | 24.3 / 65.0 | 37.9 / 79.4 |
| M2D-CLAP 2025 | 58.6 / 92.6 | 41.8 / 88.4 | 25.1 / 64.6 | 20.1 / 59.5 | 36.4 / 76.3 |
| MGA-CLAP | 54.7 / 91.5 | 41.6 / 87.1 | 26.1 / 65.0 | 21.2 / 60.9 | 35.9 / 76.1 |
| GLAP (librosa resampling) | 53.8 / 91.7 | 38.9 / 82.0 | 21.4 / 62.1 | 19.3 / 57.9 | 33.4 / 73.4 |
| WavCaps HTSAT-BERT-PT | 51.9 / 89.6 | 38.5 / 85.4 | 23.1 / 62.3 | 19.1 / 57.5 | 33.2 / 73.7 |
| Cacophony | 42.7 / 86.1 | 33.5 / 82.4 | 26.2 / 64.4 | 18.5 / 55.1 | 30.2 / 72.0 |
| LAION-CLAP larger_clap_general | 43.6 / 89.5 | 34.3 / 83.8 | 22.0 / 58.5 | 14.2 / 50.1 | 28.5 / 70.5 |
| LAION-CLAP clap-htsat-fused | 42.1 / 89.4 | 33.8 / 80.7 | 20.1 / 62.0 | 15.0 / 49.8 | 27.8 / 70.5 |
| MS-CLAP 2023 | 27.8 / 75.0 | 16.9 / 64.8 | 24.0 / 60.9 | 15.8 / 51.4 | 21.1 / 63.0 |
| MultiScale Matching, as reported | 52.2 / 91.3 | 40.6 / 86.0 | 23.0 / 61.2 | 19.1 / 56.3 | 33.7 / 73.7 |
The last row copies the numbers from the paper (weights not released; AudioCaps test set of 957 clips).
The WavLate row is computed from this exact release (AutoModel), one clip at a time.
Training
- Data: 441,198 clips with 456,546 captions from WavCaps (FreeSound, BBC Sound Effects, SoundBible, AudioSet strongly labelled), AudioCaps train and Clotho development, with test-set overlaps removed. See chungimungi/WavLate-training-data.
- Objective: multi-positive InfoNCE on the MaxSim scores of the batch, plus a KL distillation term toward the similarity ranking of a frozen pooled MGA-CLAP teacher.
- Hard negatives: 10 per clip, mined with pooled MGA-CLAP and kept only when a cross-encoder
(
cross-encoder/ettin-reranker-32m-v1) judges them different from the clip's own captions. Batches are built so each clip meets its mined negatives. One-word swap captions (for example "dog" to "cat") add text negatives. - Schedule: 5 epochs, 4,310 steps, global batch 512 clips with GradCache, AdamW, 2x A100 40 GB, 7.4 hours.
Licences
Audio tower code and initial weights: MGA-CLAP (MIT) built on HTS-AT (MIT) and Swin Transformer (MIT). Text tower: lightonai/mLateOn (Apache 2.0). Training captions: WavCaps, AudioCaps, Clotho.
- Downloads last month
- -
Model tree for chungimungi/WavLate
Base model
jhu-clsp/mmBERT-base