Hugging Face's logo

jaisawhney
/
SonicEmbed

Sentence Similarity
sentence-transformers
Safetensors
English
embedding_gemma2
audio
music
music-information-retrieval
text-to-audio-retrieval
embeddings
Model card Files
xet
 Community
jaisawhney commited on
Commit
16c7399
·
verified ·
1 Parent(s): 413610a

Update README

Browse files
Files changed (3) hide show
  1. .gitattributes +36 -36
  2. README.md +4 -2
  3. eval/song_describer_overlap.csv +51 -0
.gitattributes CHANGED
@@ -1,36 +1,36 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
- *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
- *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
36
- tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -59,6 +59,8 @@ print(query @ track)
59
 
60
  - **Prompts:** use `prompt_name="query"` for search text, and `prompt_name="document"` for captions you search from a clip. Audio takes no prompt.
61
  - **Links count as audio:** a text input that is only a link to an `.mp3`, `.wav`, `.ogg`, `.flac` or `.aac` file is fetched as audio.
 
 
62
 
63
  ## Results
64
 
@@ -92,9 +94,9 @@ SonicEmbed was trained on English-only captions, and other languages are unteste
92
 
93
  ## Training
94
 
95
- Fine-tuned with Sentence Transformers on 1,234,845 caption and clip pairs from [IAMD](https://huggingface.co/datasets/Telecom-Paris/iamd_v0), minus its non-commercial recordings, with [MusicCaps](https://huggingface.co/datasets/google/MusicCaps) making up 10 % of the pairs. The result was blended with the original weights to recover some general text quality.
96
 
97
- IAMD and MusicCaps were checked for Song Describer tracks, and any items found were excluded from training.
98
 
99
  ## Reproducing the numbers
100
 
 
59
 
60
  - **Prompts:** use `prompt_name="query"` for search text, and `prompt_name="document"` for captions you search from a clip. Audio takes no prompt.
61
  - **Links count as audio:** a text input that is only a link to an `.mp3`, `.wav`, `.ogg`, `.flac` or `.aac` file is fetched as audio.
62
+ - **In-memory audio must already be 16 kHz:** `sampling_rate` is ignored, so resample first.
63
+ - **Only the first 30 s of audio is used:** anything longer is truncated. To cover a whole track, average clips as above.
64
 
65
  ## Results
66
 
 
94
 
95
  ## Training
96
 
97
+ Fine-tuned with Sentence Transformers on 1,234,845 caption and clip pairs from [IAMD](https://huggingface.co/datasets/Telecom-Paris/iamd_v0), minus its non-commercial recordings, with [MusicCaps](https://huggingface.co/datasets/google/MusicCaps) making up 10 % of the pairs. The result was blended with the original weights to recover some general text quality, with the ratio chosen on a held-out split of the training data.
98
 
99
+ IAMD and MusicCaps were checked for Song Describer tracks, and any items found were excluded from training ([list](eval/song_describer_overlap.csv)).
100
 
101
  ## Reproducing the numbers
102
 
eval/song_describer_overlap.csv ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ track_id,iamd_items
2
+ 5345,ThePluteusTrip
3
+ 43886,OpenfmSummerHits2012
4
+ 146587,AOC14R onclassical-OC16C
5
+ 146588,onclassical-OC16C
6
+ 172015,Jonay_The_Darker_The_Sky_Is_The_Brighter_The_Stars_Shine
7
+ 172017,Jonay_The_Darker_The_Sky_Is_The_Brighter_The_Stars_Shine
8
+ 172018,Jonay_The_Darker_The_Sky_Is_The_Brighter_The_Stars_Shine
9
+ 172021,Jonay_The_Darker_The_Sky_Is_The_Brighter_The_Stars_Shine
10
+ 172023,Jonay_The_Darker_The_Sky_Is_The_Brighter_The_Stars_Shine
11
+ 172031,Jonay_The_Darker_The_Sky_Is_The_Brighter_The_Stars_Shine
12
+ 177570,cc-by_music
13
+ 219026,AdvertisingForMetaphysics
14
+ 242349,jamendo-035540_
15
+ 242384,jamendo-035540_
16
+ 242410,OpenfmSummerHits2012 jamendo-035540_
17
+ 242640,jamendo-035540_
18
+ 243172,jamendo-035540_
19
+ 243734,jamendo-035540_
20
+ 244237,jamendo-035540_
21
+ 271119,AOC24R
22
+ 285434,AOC20R
23
+ 285485,AOC20R
24
+ 285486,AOC20R
25
+ 285504,AOC20R
26
+ 347879,LorenzosMusicSelection
27
+ 356441,LorenzosMusicSelection
28
+ 617329,jamendo-132386
29
+ 617330,jamendo-132386
30
+ 832634,LorenzosMusic-JustHadToLetYouKnowEp LorenzosMusicSelection Roulette-LorenzosMusic
31
+ 956244,jamendo-112462
32
+ 1007274,jamendo-118548
33
+ 1033252,jamendo-121612
34
+ 1033253,jamendo-121612
35
+ 1060600,jamendo-125122
36
+ 1062831,jamendo-125504
37
+ 1067070,jamendo-126135
38
+ 1069500,jamendo-126482
39
+ 1086667,jamendo-129531
40
+ 1126719,jamendo-134851
41
+ 1134446,jamendo-136140
42
+ 1163315,jamendo-139821
43
+ 1163322,jamendo-139821
44
+ 1176656,Nutshell_201704
45
+ 1176657,Nutshell_201704
46
+ 1176716,Nutshell_201704
47
+ 1178762,jamendo-141440
48
+ 1178795,jamendo-141440
49
+ 1336413,jamendo-157930
50
+ 1336429,jamendo-157930
51
+ 1336433,jamendo-157930