mtg-upf/allmusiccaps_baseline_struct

AllMusicCaps: Baseline + Structured captions

Baseline plus LLM-structured captions.

Audio-text contrastive (CLAP) model over the OMAR-RQ audio encoder and an all-mpnet-base-v2 text encoder, trained on music with album-review derived captions. Embeddings are 512-dimensional; audio is expected at 24 kHz.

Usage

from amclap import get_model

model = get_model(model_id="mtg-upf/allmusiccaps_baseline_struct", device="cpu")

audio_emb = model.forward_audio(audio)   # (B, 512), audio at 24 kHz
text_emb = model.forward_text(["warm analog synth pads"])  # (B, 512)

Training

Text encoder	frozen (stock all-mpnet-base-v2)
Reported checkpoint	step 149,796
Audio encoder layers	see config.gin

The text encoder was frozen during training, so stock all-mpnet-base-v2 weights are reloaded at construction.

License

CC BY-NC-SA 4.0. See LICENSE.

Citation

@inproceedings{alonso2026allmusiccaps,
  title = {{AllMusicCaps}: Album Reviews as Complementary Supervision for Music {CLAP}},
  author = {Alonso-Jim{\'e}nez, Pablo and Lizarraga-Seijas, Xavier and Serra, Xavier and Bogdanov, Dmitry},
  booktitle = {International Society for Music Information Retrieval Conference (ISMIR)},
  year = {2026},
}