metadata
license: mit
base_model: laion/clap-htsat-unfused
library_name: peft
tags:
- clap
- audio-retrieval
- music-information-retrieval
- lora
- contrastive
pipeline_tag: feature-extraction
Crate — CLAP LoRA for producer vocabulary
LoRA adapter over laion/clap-htsat-unfused fine-tuned so producer terms base CLAP
barely knows — boom-bap, tape-saturated, rimshot, reese bass — pull the right
audio. Part of Crate, sound-native
search for music producers (hum / drop a track / describe it → one embedding space).
Results — producer-vocab retrieval (held-out)
| metric | base CLAP | fine-tuned |
|---|---|---|
| recall@1 | 0.210 | 0.405 |
| recall@10 | 0.746 | 0.951 |
| recall@5 | 0.580 | 0.868 |
Usage
from transformers import ClapModel, ClapProcessor
from peft import PeftModel
base = "laion/clap-htsat-unfused"
model = PeftModel.from_pretrained(ClapModel.from_pretrained(base), "jahnaviym/crate-clap-lora").merge_and_unload()
proc = ClapProcessor.from_pretrained(base)
# proc(text=[...]) / proc(audios=[...], sampling_rate=48000) → get_text/audio_features
Training
- Symmetric InfoNCE on (audio, text) pairs from Freesound (CC0/CC-BY) + free packs.
- LoRA on the audio/text projection heads (r=16, alpha=32).
- Label-preserving augmentation (pitch/stretch/noise/EQ) for positive pairs.
See the repo for the full pipeline and eval.