| --- |
| license: mit |
| language: |
| - ru |
| tags: |
| - clap |
| - audio-text |
| - speech |
| - russian |
| - contrastive-learning |
| pipeline_tag: feature-extraction |
| --- |
| |
| # Best model: weights + inference script |
|
|
| This package contains the trained weights of the best-performing CLAP model from the paper's |
| multi-seed encoder comparison, plus a standalone script to compute an audio-text similarity score |
| with it — without needing the training pipeline or dataset. |
|
|
| ## Model |
|
|
| - **Text encoder**: `Qwen/Qwen3-Embedding-4B` (frozen, downloaded from HuggingFace). |
| - **Audio encoder**: `openai/whisper-large-v3`, last encoder layer (layer 32/32; frozen). |
| - **Trained parameters** (what's actually in the checkpoint, ~11MB): two projection heads |
| (`mlp3_residual` architecture — Linear → LayerNorm → GELU → Dropout → residual block → Linear, |
| hidden_dim=512, joint_dim=256) mapping the frozen text/audio embeddings into a shared 256-d |
| space, plus a small learned gender-embedding table (male/female/unknown, dim 16, concatenated |
| onto the text embedding before projection) and the InfoNCE logit scale/bias. |
|
|
| ## Checkpoints |
|
|
| All **5 trained seeds** are published (`checkpoints/qwen3_whisper_last_seed{0-4}_best.pt`), matching |
| the paper's mean±std-over-5-seeds methodology — any of them can be used with `infer.py`; they are |
| equivalent for practical use. To reproduce the exact number reported for a specific seed in the |
| paper's tables, use the corresponding `seedN` checkpoint. |
|
|
| The frozen pretrained encoders (Qwen3-Embedding-4B, ~8GB; Whisper-large-v3, ~3GB) are **not** |
| included here — `infer.py` downloads them automatically from HuggingFace on first run. |
|
|
| ## Usage |
|
|
| ```sh |
| pip install -r requirements.txt |
| python infer.py --audio clip.wav --text "тихо прошептала, едва сдерживая слёзы" |
| ``` |
|
|
| Optional flags: |
|
|
| ```sh |
| python infer.py \ |
| --audio clip.wav \ |
| --text "тихо прошептала, едва сдерживая слёзы" \ |
| --gender unknown \ |
| --checkpoint checkpoints/qwen3_whisper_last_seed0_best.pt \ |
| --device cuda |
| ``` |
|
|
| Output: a single number, the cosine similarity between the audio and text joint embeddings, |
| in `[-1, 1]`: |
|
|
| ``` |
| cosine similarity: 0.1823 |
| ``` |
|
|
| `--gender` defaults to `unknown` — this matches the setup used for zero-shot evaluation on |
| external datasets in the paper (reader gender is generally not known/applicable at inference |
| time); pass `male`/`female` explicitly if you want to condition on a specific target voice. |
|
|
| ## Limitations |
|
|
| - `infer.py` is intentionally self-contained: it does not import anything from the main sdclap |
| repository, only reimplements the minimal inference-time logic (see inline comments) — this |
| keeps it runnable with just `pip install -r requirements.txt` and a checkpoint file. |
|
|
| ## License |
|
|
| Code (`infer.py`) — MIT. Checkpoint weights — MIT (trained parameters only, no third-party |
| copyrighted content embedded). |
|
|
