YirongSun/glap_ft

SPEAR-GLAP fine-tuned checkpoint

This is the HF-native best checkpoint produced by audio-side fine-tuning of GLAP on 12,718 SPEAR audio-query pairs. The text encoder and text projection were frozen; the audio encoder and audio projection were optimized in FP32.

The implementation is released from commit cbf5599c5adc95c4d9f4128a0e9c6773aa2bc0d4 of SUAT-AIRI/SPEAR-GLAP. This repository contains custom model code. Review it and pin an immutable Hugging Face revision before setting trust_remote_code=True in a production or reproducible workflow.

Important evaluation status

The supplied 5,451-row test manifest was evaluated after every epoch and used to choose this checkpoint. Therefore all numbers below are test-tuned and must not be presented as unbiased holdout results. Retrieval uses one paired target audio per query row, including nine rows whose query text is repeated.

Metric	Result
text-to-audio R@1	11.0989%
text-to-audio R@5	26.8391%
text-to-audio R@10	36.9107%
MRR	19.5750%
MeanRank	101.81
MedianRank	22

The selected checkpoint is epoch 4 at optimizer step 1,592. Its model.safetensors SHA-256 is cb97916b295ded7b78fe0b74561e2db09ee79948dd5bb98b822771e4148273f9.

Loading

The validated runtime uses Python 3.10, PyTorch/Torchaudio 2.6.0 with CUDA 12.4, Transformers 4.40.2, SentencePiece, Einops, and Safetensors. Pinned revisions support fully offline text and audio loading from a custom Hugging Face cache.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "YirongSun/glap_ft",
    trust_remote_code=True,
)
model.eval()

with torch.inference_mode():
    text_embeddings = model.get_text_features(
        ["A calm adult speaker in a quiet room."],
        source_lang="eng_Latn",
    )

# Audio input is a float waveform batch shaped [batch, samples] at 16 kHz.
audio = torch.zeros(1, 16000)
with torch.inference_mode():
    audio_embeddings = model.get_audio_features(audio)

similarity = text_embeddings @ audio_embeddings.T

Both feature methods return L2-normalized 1,024-dimensional embeddings. Audio must be decoded, converted to mono, and resampled to 16 kHz by the caller.

Training recipe and provenance

The release includes training_recipe.yaml, training_metadata.json, evaluation_summary.json, and provenance.json. They record the frozen input hashes, exact executable source fingerprint, checkpoint-selection rule, and machine-readable metrics without publishing host-specific data paths.

Limitations

• Test-set checkpoint selection makes the reported results test-tuned.
• Repeated text is evaluated per manifest row against that row's paired audio, not as a multi-positive target.
• The supplied split does not guarantee speaker, dialogue, transcript, or identity isolation.
• The maintained fine-tuning recipe is single-GPU FP32 and performs no hyperparameter search.

Attribution

SPEAR-GLAP derives from Xiaomi Research's Dasheng-GLAP. The retained model implementation and this checkpoint are distributed under Apache-2.0; see LICENSE and the source repository for details.