Leitmotif — checkpoints

Weights for Leitmotif: reading a song two ways, as a graph of its own structure and through the words people use to describe it. One-second segments become graph nodes, edges join segments that follow each other or sound alike, a GNN reads that structure and BERT reads the caption.

Try it: 404mahdi/leitmotif Space

Files

File	Stage	What it is
stage4_contrastive_pretrained.pt	4	InfoNCE dual encoder: graph encoder + BERT, shared embedding space
stage4_contrastive_pretrained_index.pt	4	Precomputed graph and text embeddings for the 2,555 test clips, with their YouTube ids
stage3_cross_attention_masked_pretrained.pt	3	Cross-attention fusion tagger over 50 context tags
*_args.json	—	The exact run configuration each checkpoint was trained with
*_node_stats.pt	—	Node feature normalisation statistics; inference needs these to match training

Usage

These load through the project's own inference wrapper, which expects them in checkpoints/:

from huggingface_hub import hf_hub_download
from src.inference import Leitmotif

for name in ["stage4_contrastive_pretrained.pt", "stage4_contrastive_pretrained_args.json",
             "stage4_contrastive_pretrained_node_stats.pt", "stage4_contrastive_pretrained_index.pt"]:
    hf_hub_download("404mahdi/leitmotif", name, local_dir="checkpoints")

lm = Leitmotif()
lm.search("a sad piano ballad with strings")   # closest MusicCaps clips
lm.tags("song.wav", "a mellow piano ballad")   # predicted context tags
lm.explain("song.wav", "a mellow piano ballad")  # words x segments attention

Results

Retrieval over 2,555 MusicCaps test clips, chance R@10 = 0.004:

Direction	R@1	R@5	R@10	Median rank
Caption → audio	0.013	0.051	0.087	151
Audio → caption	0.013	0.056	0.095	140

Tagging on the same clips, captions with tag words masked, 50 tags:

Model	Macro-F1	Micro-F1	AUC-PR
Graph only (audio)	0.227	0.304	0.170
BERT only (masked caption)	0.468	0.556	0.459
Early concatenation	0.448	0.535	0.438
Cross-attention (this checkpoint)	0.440	0.528	0.428

Limitations, stated plainly

On aggregate tagging metrics the audio graph does not beat text alone — the cross-attention model shipped here scores below the BERT-only baseline. It is published because the per-tag picture is the actual finding: audio helps where a masked caption goes quiet, on who is singing and which instruments play, and costs accuracy on tags the remaining words still describe. The full ablation is in the paper.

Trained on one RTX 5060 Laptop GPU (8 GB) on 10-second clips at 16 kHz. Retrieval quality is roughly 20× chance, not production search. English captions only, and it inherits MusicCaps' genre and cultural skew.

Citation

Solo project by Mahdi Hasan for CSE425 (Neural Networks), BRAC University. Code MIT licensed; MusicCaps is CC BY-SA 4.0 and keeps its own terms.