WavLate

image

WavLate is a late-interaction (multi-vector) model for retrieval between audio clips and text captions. It keeps one vector for every 0.31 seconds of audio and one vector for every caption token, and scores a pair with MaxSim: each caption token finds its best-matching moment in the clip, and the score is the average of those best matches. Clips and captions are encoded independently, so every clip can be indexed once and searched with any caption. To our knowledge it is the first open late-interaction model trained for audio and caption retrieval.

One general model, with no per-dataset fine-tuning, it reaches a mean R@1 of 37.9 over the four AudioCaps and Clotho retrieval tasks, ahead of the eight other open audio and text models we evaluated on the same test pools.

Quick start

Using Sentence Transformers

pip install -U "sentence_transformers[audio]>=6.0.0" soundfile torchaudio
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("chungimungi/WavLate", trust_remote_code=True)

captions = ["A dog barks while birds chirp", "Rain falls on a tin roof"]
clips = [
    "https://raw.githubusercontent.com/karolpiczak/ESC-50/master/audio/1-100032-A-0.wav",
    "https://raw.githubusercontent.com/karolpiczak/ESC-50/master/audio/1-17367-A-10.wav",
]

text = model.encode_query(captions)
audio = model.encode_document(clips, batch_size=1)
print([tuple(embedding.shape) for embedding in text])
# [(7, 128), (6, 128)]
print([tuple(embedding.shape) for embedding in audio])
# [(16, 128), (16, 128)]

scores = model.similarity(text, audio)
print(scores.cpu())
# tensor([[0.3718, 0.1570],
#         [0.1569, 0.4258]])

To use your own recordings, replace the ESC-50 URLs with local audio paths or URLs. Files are automatically converted to mono and resampled to 32 kHz. When passing raw waveforms instead, include their sample rate: {"audio": {"array": wave, "sampling_rate": rate}}.

Captions receive the [Q] prefix automatically and can then be compared with audio using model.similarity(text, audio). Always pass text embeddings first, even when retrieving captions for audio.

Using Transformers

import soundfile as sf
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("chungimungi/WavLate", trust_remote_code=True).eval()

captions = ["A dog barks while birds chirp", "Rain falls on a tin roof"]
wave, rate = sf.read("clip.wav", dtype="float32")    # any length; mono or stereo
if wave.ndim > 1:
    wave = wave.mean(axis=1)

with torch.inference_mode():
    text, text_mask = model.encode_text(captions)                     # [2, tokens, 128]
    audio, audio_mask = model.encode_audio([wave], sampling_rate=rate) # [1, frames, 128]
    scores = model.score(text, text_mask, audio, audio_mask)          # [2 captions, 1 clip]
print(scores)
  • encode_text returns one 128-dimensional unit vector per caption token and a mask of the tokens that count (padding and special tokens are excluded).
  • encode_audio expects mono audio at 32 kHz. Pass sampling_rate to resample other rates (this needs torchaudio). Every 10 second window gives 32 frame vectors. Clips longer than 10 seconds are split into consecutive windows and all their frames are kept, up to 30 seconds.
  • score returns mean MaxSim scores of shape [number of captions, number of clips]. Higher means a better match. The same score ranks clips for a caption (text to audio) and captions for a clip (audio to text).
  • Tested with transformers 5.23 and torch 2.11. The text tower needs transformers 4.48 or newer (ModernBERT).

Model

Part Details
Audio tower HTS-AT (Swin transformer on log-mel spectrograms, 31M parameters) with MGA-CLAP's locality-aware output, initialised from MGA-CLAP
Audio vectors 32 per 10 s window, MGA-CLAP frame projection (768 to 1024), then a linear projection to 128 dimensions, L2-normalised
Text tower mLateOn: ModernBERT backbone with ColBERT projection layers (768 to 1536 to 768 to 128)
Text vectors one 128-dimensional L2-normalised vector per token; captions are prefixed with the [Q] marker
Score mean MaxSim from caption tokens to audio frames
Parameters 344M in total, of which 197M are the text tower's token embedding table

Results

R@1 / R@10 (%) on AudioCaps test (964 clips, 4,820 captions) and Clotho evaluation (1,045 clips, 5,225 captions). A2T means audio to text (a clip finds its captions), T2A means text to audio (a caption finds its clip). All baselines except the last row were run by us on the same test pools with their released checkpoints and their own preprocessing. Bold marks the best value in each column among the models we ran.

Model AudioCaps A2T AudioCaps T2A Clotho A2T Clotho T2A Mean
WavLate 53.6 / 93.5 43.6 / 87.9 30.0 / 71.0 24.3 / 65.0 37.9 / 79.4
M2D-CLAP 2025 58.6 / 92.6 41.8 / 88.4 25.1 / 64.6 20.1 / 59.5 36.4 / 76.3
MGA-CLAP 54.7 / 91.5 41.6 / 87.1 26.1 / 65.0 21.2 / 60.9 35.9 / 76.1
GLAP (librosa resampling) 53.8 / 91.7 38.9 / 82.0 21.4 / 62.1 19.3 / 57.9 33.4 / 73.4
WavCaps HTSAT-BERT-PT 51.9 / 89.6 38.5 / 85.4 23.1 / 62.3 19.1 / 57.5 33.2 / 73.7
Cacophony 42.7 / 86.1 33.5 / 82.4 26.2 / 64.4 18.5 / 55.1 30.2 / 72.0
LAION-CLAP larger_clap_general 43.6 / 89.5 34.3 / 83.8 22.0 / 58.5 14.2 / 50.1 28.5 / 70.5
LAION-CLAP clap-htsat-fused 42.1 / 89.4 33.8 / 80.7 20.1 / 62.0 15.0 / 49.8 27.8 / 70.5
MS-CLAP 2023 27.8 / 75.0 16.9 / 64.8 24.0 / 60.9 15.8 / 51.4 21.1 / 63.0
MultiScale Matching, as reported 52.2 / 91.3 40.6 / 86.0 23.0 / 61.2 19.1 / 56.3 33.7 / 73.7

The last row copies the numbers from the paper (weights not released; AudioCaps test set of 957 clips). The WavLate row is computed from this exact release (AutoModel), one clip at a time.

Training

  • Data: 441,198 clips with 456,546 captions from WavCaps (FreeSound, BBC Sound Effects, SoundBible, AudioSet strongly labelled), AudioCaps train and Clotho development, with test-set overlaps removed. See chungimungi/WavLate-training-data.
  • Objective: multi-positive InfoNCE on the MaxSim scores of the batch, plus a KL distillation term toward the similarity ranking of a frozen pooled MGA-CLAP teacher.
  • Hard negatives: 10 per clip, mined with pooled MGA-CLAP and kept only when a cross-encoder (cross-encoder/ettin-reranker-32m-v1) judges them different from the clip's own captions. Batches are built so each clip meets its mined negatives. One-word swap captions (for example "dog" to "cat") add text negatives.
  • Schedule: 5 epochs, 4,310 steps, global batch 512 clips with GradCache, AdamW, 2x A100 40 GB, 7.4 hours.

Licences

Audio tower code and initial weights: MGA-CLAP (MIT) built on HTS-AT (MIT) and Swin Transformer (MIT). Text tower: lightonai/mLateOn (Apache 2.0). Training captions: WavCaps, AudioCaps, Clotho.

Downloads last month
-
Safetensors
Model size
0.3B params
Tensor type
I64
·
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for chungimungi/WavLate

Finetuned
(4)
this model

Dataset used to train chungimungi/WavLate

Paper for chungimungi/WavLate

Article mentioning chungimungi/WavLate