radoslavralev commited on
Commit
9fbaebd
·
verified ·
1 Parent(s): c402a93

Add new SentenceTransformer model

Browse files
Files changed (1) hide show
  1. README.md +184 -138
README.md CHANGED
@@ -5,42 +5,88 @@ tags:
5
  - feature-extraction
6
  - dense
7
  - generated_from_trainer
8
- - dataset_size:111468
9
  - loss:MultipleNegativesRankingLoss
10
  base_model: thenlper/gte-small
11
  widget:
12
- - source_sentence: What is something you do (or don’t do), even though you feel conflicted
13
- about it?
14
  sentences:
15
- - What is something you do (or don’t do), even though you feel conflicted about
16
- it?
17
- - Is it worth buying the iPhone 7?
18
- - 'Hypothetical scenarios: King Henry VIII loses his battle with James IV in 1513
19
- & dies; Pope Julius II doesn''t die in 1513. How''s the world different?'
20
- - source_sentence: Exams for a mechanical engineer?
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  sentences:
22
- - Exams for a mechanical engineer?
23
- - Can you prefer any website or ideas by which I can understand antenna subject
24
- practically in b.tech?
25
- - Mackenzie is a writer-in-residence at the 2B Theatre in Halifax and teaches at
26
- the National Theatre School of Canada in Montreal .
27
- - source_sentence: What will a Christian wife do if her husband left her for years?
 
 
 
 
 
28
  sentences:
29
- - How many United States Presidents have there been?
30
- - What is planning without words?
31
- - What will a Christian wife do if her husband left her for years?
32
- - source_sentence: How do I research for MUN?
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  sentences:
34
- - How do I research for MUN?
35
- - What is the best way to be an investment banker?
36
- - What is the best way to do an MUN research?
37
- - source_sentence: I am poor, ugly, untalented, 20 years old, and have big dreams.
38
- How can I succeed in life?
 
 
 
 
 
 
 
 
 
39
  sentences:
40
- - What app can I use taking notes?
41
- - Am I too old to succeed in my life at age 32?
42
- - I am poor, ugly, untalented, 20 years old, and have big dreams. How can I succeed
43
- in life?
 
 
44
  pipeline_tag: sentence-similarity
45
  library_name: sentence-transformers
46
  metrics:
@@ -73,46 +119,46 @@ model-index:
73
  value: 0.3
74
  name: Cosine Accuracy@1
75
  - type: cosine_accuracy@3
76
- value: 0.58
77
  name: Cosine Accuracy@3
78
  - type: cosine_accuracy@5
79
- value: 0.6
80
  name: Cosine Accuracy@5
81
  - type: cosine_accuracy@10
82
- value: 0.68
83
  name: Cosine Accuracy@10
84
  - type: cosine_precision@1
85
  value: 0.3
86
  name: Cosine Precision@1
87
  - type: cosine_precision@3
88
- value: 0.19333333333333333
89
  name: Cosine Precision@3
90
  - type: cosine_precision@5
91
- value: 0.12000000000000002
92
  name: Cosine Precision@5
93
  - type: cosine_precision@10
94
- value: 0.068
95
  name: Cosine Precision@10
96
  - type: cosine_recall@1
97
  value: 0.3
98
  name: Cosine Recall@1
99
  - type: cosine_recall@3
100
- value: 0.58
101
  name: Cosine Recall@3
102
  - type: cosine_recall@5
103
- value: 0.6
104
  name: Cosine Recall@5
105
  - type: cosine_recall@10
106
- value: 0.68
107
  name: Cosine Recall@10
108
  - type: cosine_ndcg@10
109
- value: 0.4950369328373354
110
  name: Cosine Ndcg@10
111
  - type: cosine_mrr@10
112
- value: 0.43527777777777776
113
  name: Cosine Mrr@10
114
  - type: cosine_map@100
115
- value: 0.4475531768839056
116
  name: Cosine Map@100
117
  - task:
118
  type: information-retrieval
@@ -122,49 +168,49 @@ model-index:
122
  type: NanoNQ
123
  metrics:
124
  - type: cosine_accuracy@1
125
- value: 0.26
126
  name: Cosine Accuracy@1
127
  - type: cosine_accuracy@3
128
- value: 0.48
129
  name: Cosine Accuracy@3
130
  - type: cosine_accuracy@5
131
- value: 0.52
132
  name: Cosine Accuracy@5
133
  - type: cosine_accuracy@10
134
- value: 0.64
135
  name: Cosine Accuracy@10
136
  - type: cosine_precision@1
137
- value: 0.26
138
  name: Cosine Precision@1
139
  - type: cosine_precision@3
140
- value: 0.16666666666666663
141
  name: Cosine Precision@3
142
  - type: cosine_precision@5
143
- value: 0.10800000000000001
144
  name: Cosine Precision@5
145
  - type: cosine_precision@10
146
- value: 0.066
147
  name: Cosine Precision@10
148
  - type: cosine_recall@1
149
- value: 0.24
150
  name: Cosine Recall@1
151
  - type: cosine_recall@3
152
- value: 0.45
153
  name: Cosine Recall@3
154
  - type: cosine_recall@5
155
- value: 0.49
156
  name: Cosine Recall@5
157
  - type: cosine_recall@10
158
- value: 0.6
159
  name: Cosine Recall@10
160
  - type: cosine_ndcg@10
161
- value: 0.4279054208986469
162
  name: Cosine Ndcg@10
163
  - type: cosine_mrr@10
164
- value: 0.3892142857142856
165
  name: Cosine Mrr@10
166
  - type: cosine_map@100
167
- value: 0.3750113241088494
168
  name: Cosine Map@100
169
  - task:
170
  type: nano-beir
@@ -174,49 +220,49 @@ model-index:
174
  type: NanoBEIR_mean
175
  metrics:
176
  - type: cosine_accuracy@1
177
- value: 0.28
178
  name: Cosine Accuracy@1
179
  - type: cosine_accuracy@3
180
  value: 0.53
181
  name: Cosine Accuracy@3
182
  - type: cosine_accuracy@5
183
- value: 0.56
184
  name: Cosine Accuracy@5
185
  - type: cosine_accuracy@10
186
- value: 0.66
187
  name: Cosine Accuracy@10
188
  - type: cosine_precision@1
189
- value: 0.28
190
  name: Cosine Precision@1
191
  - type: cosine_precision@3
192
- value: 0.18
193
  name: Cosine Precision@3
194
  - type: cosine_precision@5
195
- value: 0.11400000000000002
196
  name: Cosine Precision@5
197
  - type: cosine_precision@10
198
- value: 0.067
199
  name: Cosine Precision@10
200
  - type: cosine_recall@1
201
- value: 0.27
202
  name: Cosine Recall@1
203
  - type: cosine_recall@3
204
  value: 0.515
205
  name: Cosine Recall@3
206
  - type: cosine_recall@5
207
- value: 0.5449999999999999
208
  name: Cosine Recall@5
209
  - type: cosine_recall@10
210
- value: 0.64
211
  name: Cosine Recall@10
212
  - type: cosine_ndcg@10
213
- value: 0.46147117686799116
214
  name: Cosine Ndcg@10
215
  - type: cosine_mrr@10
216
- value: 0.4122460317460317
217
  name: Cosine Mrr@10
218
  - type: cosine_map@100
219
- value: 0.4112822504963775
220
  name: Cosine Map@100
221
  ---
222
 
@@ -270,9 +316,9 @@ from sentence_transformers import SentenceTransformer
270
  model = SentenceTransformer("redis/model-b-structured")
271
  # Run inference
272
  sentences = [
273
- 'I am poor, ugly, untalented, 20 years old, and have big dreams. How can I succeed in life?',
274
- 'I am poor, ugly, untalented, 20 years old, and have big dreams. How can I succeed in life?',
275
- 'Am I too old to succeed in my life at age 32?',
276
  ]
277
  embeddings = model.encode(sentences)
278
  print(embeddings.shape)
@@ -281,9 +327,9 @@ print(embeddings.shape)
281
  # Get the similarity scores for the embeddings
282
  similarities = model.similarity(embeddings, embeddings)
283
  print(similarities)
284
- # tensor([[1.0000, 1.0000, 0.3917],
285
- # [1.0000, 1.0000, 0.3917],
286
- # [0.3917, 0.3917, 1.0000]])
287
  ```
288
 
289
  <!--
@@ -321,21 +367,21 @@ You can finetune this model on your own dataset.
321
 
322
  | Metric | NanoMSMARCO | NanoNQ |
323
  |:--------------------|:------------|:-----------|
324
- | cosine_accuracy@1 | 0.3 | 0.26 |
325
- | cosine_accuracy@3 | 0.58 | 0.48 |
326
- | cosine_accuracy@5 | 0.6 | 0.52 |
327
- | cosine_accuracy@10 | 0.68 | 0.64 |
328
- | cosine_precision@1 | 0.3 | 0.26 |
329
- | cosine_precision@3 | 0.1933 | 0.1667 |
330
- | cosine_precision@5 | 0.12 | 0.108 |
331
- | cosine_precision@10 | 0.068 | 0.066 |
332
- | cosine_recall@1 | 0.3 | 0.24 |
333
- | cosine_recall@3 | 0.58 | 0.45 |
334
- | cosine_recall@5 | 0.6 | 0.49 |
335
- | cosine_recall@10 | 0.68 | 0.6 |
336
- | **cosine_ndcg@10** | **0.495** | **0.4279** |
337
- | cosine_mrr@10 | 0.4353 | 0.3892 |
338
- | cosine_map@100 | 0.4476 | 0.375 |
339
 
340
  #### Nano BEIR
341
 
@@ -353,21 +399,21 @@ You can finetune this model on your own dataset.
353
 
354
  | Metric | Value |
355
  |:--------------------|:-----------|
356
- | cosine_accuracy@1 | 0.28 |
357
  | cosine_accuracy@3 | 0.53 |
358
- | cosine_accuracy@5 | 0.56 |
359
- | cosine_accuracy@10 | 0.66 |
360
- | cosine_precision@1 | 0.28 |
361
- | cosine_precision@3 | 0.18 |
362
- | cosine_precision@5 | 0.114 |
363
- | cosine_precision@10 | 0.067 |
364
- | cosine_recall@1 | 0.27 |
365
  | cosine_recall@3 | 0.515 |
366
- | cosine_recall@5 | 0.545 |
367
- | cosine_recall@10 | 0.64 |
368
- | **cosine_ndcg@10** | **0.4615** |
369
- | cosine_mrr@10 | 0.4122 |
370
- | cosine_map@100 | 0.4113 |
371
 
372
  <!--
373
  ## Bias, Risks and Limitations
@@ -387,19 +433,19 @@ You can finetune this model on your own dataset.
387
 
388
  #### Unnamed Dataset
389
 
390
- * Size: 111,468 training samples
391
  * Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
392
  * Approximate statistics based on the first 1000 samples:
393
- | | anchor | positive | negative |
394
- |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
395
- | type | string | string | string |
396
- | details | <ul><li>min: 6 tokens</li><li>mean: 16.11 tokens</li><li>max: 71 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 16.16 tokens</li><li>max: 71 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 17.35 tokens</li><li>max: 76 tokens</li></ul> |
397
  * Samples:
398
- | anchor | positive | negative |
399
- |:--------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------|
400
- | <code>How many grams of protein should I eat a day?</code> | <code>How much protein should I eat per day?</code> | <code>How does hypokalemia lead to polyuria in primary aldosteronism?</code> |
401
- | <code>Who said to get out of economic crisis we need to buy more?</code> | <code>Who said to get out of economic crisis we need to buy more?</code> | <code>What are some good IT certifications that don't require programming skills?</code> |
402
- | <code>What is the difference between Chinese and western culture within China?</code> | <code>What is the difference between Chinese and western culture within China?</code> | <code>What is the difference between Chinese and western culture outside China?</code> |
403
  * Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
404
  ```json
405
  {
@@ -416,16 +462,16 @@ You can finetune this model on your own dataset.
416
  * Size: 12,386 evaluation samples
417
  * Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
418
  * Approximate statistics based on the first 1000 samples:
419
- | | anchor | positive | negative |
420
- |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
421
- | type | string | string | string |
422
- | details | <ul><li>min: 6 tokens</li><li>mean: 16.22 tokens</li><li>max: 62 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 16.28 tokens</li><li>max: 62 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 17.39 tokens</li><li>max: 66 tokens</li></ul> |
423
  * Samples:
424
- | anchor | positive | negative |
425
- |:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------|
426
- | <code>What is it about novels that allow them to deal with deep themes that short stories, drama, and poetry cannot achieve?</code> | <code>What is it about novels that allow them to deal with deep themes that short stories, drama, and poetry cannot achieve?</code> | <code>What are films that deal with themes like death and letting go?</code> |
427
- | <code>If alien civilizations are thought to be much more advanced than us, why haven't they made contact with us yet?</code> | <code>If there are super intelligent alien beings somewhere in the Galaxy why haven't they tried to contact us yet?</code> | <code>What's not so good about Aston Martin cars?</code> |
428
- | <code>How can you determine the Lewis dot structure for sulfur trioxide?</code> | <code>How can you determine the Lewis dot structure for sulfur trioxide?</code> | <code>How can you determine the Lewis dot structure for sulfur?</code> |
429
  * Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
430
  ```json
431
  {
@@ -441,8 +487,8 @@ You can finetune this model on your own dataset.
441
  - `eval_strategy`: steps
442
  - `per_device_train_batch_size`: 128
443
  - `per_device_eval_batch_size`: 128
444
- - `learning_rate`: 2e-05
445
- - `weight_decay`: 0.0001
446
  - `max_steps`: 3000
447
  - `warmup_ratio`: 0.1
448
  - `fp16`: True
@@ -470,8 +516,8 @@ You can finetune this model on your own dataset.
470
  - `gradient_accumulation_steps`: 1
471
  - `eval_accumulation_steps`: None
472
  - `torch_empty_cache_steps`: None
473
- - `learning_rate`: 2e-05
474
- - `weight_decay`: 0.0001
475
  - `adam_beta1`: 0.9
476
  - `adam_beta2`: 0.999
477
  - `adam_epsilon`: 1e-08
@@ -584,19 +630,19 @@ You can finetune this model on your own dataset.
584
  ### Training Logs
585
  | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_cosine_ndcg@10 | NanoNQ_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 |
586
  |:------:|:----:|:-------------:|:---------------:|:--------------------------:|:---------------------:|:----------------------------:|
587
- | 0 | 0 | - | 3.6560 | 0.6259 | 0.6583 | 0.6421 |
588
- | 0.2874 | 250 | 2.1436 | 0.4823 | 0.5264 | 0.5634 | 0.5449 |
589
- | 0.5747 | 500 | 0.5891 | 0.4299 | 0.5280 | 0.5051 | 0.5165 |
590
- | 0.8621 | 750 | 0.5393 | 0.4123 | 0.5246 | 0.4755 | 0.5001 |
591
- | 1.1494 | 1000 | 0.5173 | 0.4027 | 0.5068 | 0.4549 | 0.4809 |
592
- | 1.4368 | 1250 | 0.5022 | 0.3954 | 0.5055 | 0.4513 | 0.4784 |
593
- | 1.7241 | 1500 | 0.4958 | 0.3909 | 0.5033 | 0.4466 | 0.4749 |
594
- | 2.0115 | 1750 | 0.4908 | 0.3890 | 0.4897 | 0.4416 | 0.4656 |
595
- | 2.2989 | 2000 | 0.4824 | 0.3859 | 0.4912 | 0.4359 | 0.4636 |
596
- | 2.5862 | 2250 | 0.4797 | 0.3847 | 0.4987 | 0.4387 | 0.4687 |
597
- | 2.8736 | 2500 | 0.4728 | 0.3834 | 0.4969 | 0.4256 | 0.4613 |
598
- | 3.1609 | 2750 | 0.4721 | 0.3824 | 0.4863 | 0.4279 | 0.4571 |
599
- | 3.4483 | 3000 | 0.4694 | 0.3822 | 0.4950 | 0.4279 | 0.4615 |
600
 
601
 
602
  ### Framework Versions
 
5
  - feature-extraction
6
  - dense
7
  - generated_from_trainer
8
+ - dataset_size:111470
9
  - loss:MultipleNegativesRankingLoss
10
  base_model: thenlper/gte-small
11
  widget:
12
+ - source_sentence: why are some rocks radioactive
 
13
  sentences:
14
+ - Radioactive accessory minerals such as zircon may contribute to the radioactivity
15
+ of a mineral which is otherwise non-radioactive by calculation. Many granites
16
+ or other igneous rocks contain some radioactivity because of minor, but highly
17
+ radioactive, accessory minerals.re = mineral density (S Atomic number / Molecular
18
+ Weight) where re is the electron density in grams/cc.efinition. Radioactivity
19
+ in minerals are caused by the inclusion of naturally-occurring radioactive elements
20
+ in the mineral's composition. The degree of radioactivity is dependent on the
21
+ concentration and isotope present in the mineral.
22
+ - Taking B-complex vitamins, which include vitamin B12, can cause urine to have
23
+ a bright yellow or even orange color, but check with your doctor to be sure that's
24
+ what is going on in your case. B vitamins are water-soluble vitamins, which means
25
+ that what your body doesn't use is excreted in your urine. Riboflavin (vitamin
26
+ B2) is especially likely to cause this color change in urine. Several medications
27
+ can also turn urine a bright yellow or orange color. Changes in urine color may
28
+ also signal certain health problems.
29
+ - Radioactive material is just another name for a group of unstable atoms that emit
30
+ ionizing radiation. These groups of unstable atoms emit radiation because they
31
+ try to become stable. Radioactive materials emit radiation in a process called
32
+ radioactive decay.
33
+ - source_sentence: How was your experience of Lucid dreaming at home?
34
  sentences:
35
+ - How was your experience of Lucid dreaming at home?
36
+ - How was your experience of Lucid dreaming outside the home?
37
+ - "Bournemouth /Ë\x88bÉ\x94É\x99rnmÉ\x99θ/ is a large coastal resort town on the\
38
+ \ south coast of England directly to the east of the Jurassic Coast, a 96-mile\
39
+ \ (155 km) World Heritage Site. According to the 2011 census, the town has a population\
40
+ \ of 183,491 making it the largest settlement in Dorset.he Bournemouth Eye is\
41
+ \ a helium-filled balloon attached to a steel cable in the town's lower gardens.\
42
+ \ The spherical balloon is 69 m (226 ft) in circumference and carries an enclosed,\
43
+ \ steel gondola. Rising to a height of 150 m (492 ft), it provides a panoramic\
44
+ \ view of the surrounding area for up to 28 passengers."
45
+ - source_sentence: what is iraq's dominant religion
46
  sentences:
47
+ - 'If you are working, consider taking maternity leave as early as you can. This
48
+ makes sense anyway because carrying twins is hard work, and most twins arrive
49
+ earlier than single babies (NCCWCH 2011: 128) . More than half of twins arrive
50
+ early, before 37 weeks (NCCWCH 2011: 120, Tamba 2012) .Talk to your midwife or
51
+ doctor if you are feeling down about your pregnancy (NICE 2011) .f you are working,
52
+ consider taking maternity leave as early as you can. This makes sense anyway because
53
+ carrying twins is hard work, and most twins arrive earlier than single babies
54
+ (NCCWCH 2011: 128) . More than half of twins arrive early, before 37 weeks (NCCWCH
55
+ 2011: 120, Tamba 2012) .'
56
+ - "Introduction. Although Iranâ\x80\x99s state religion is Shiite Islam and the\
57
+ \ majority of its population is ethnically Persian, millions of minorities from\
58
+ \ various ethnic, religious, and linguistic backgrounds also reside in Iran. Among\
59
+ \ these groups are ethnic Kurds, Baluchis, and Azeris.lthough Iranâ\x80\x99s state\
60
+ \ religion is Shiite Islam and the majority of its population is ethnically Persian,\
61
+ \ millions of minorities from various ethnic, religious, and linguistic backgrounds\
62
+ \ also reside in Iran."
63
+ - In today's Republic of Iraq, where Islam is the state religion and claims the
64
+ beliefs of 95 percent of the population, the majority of Iraqis identify with
65
+ Arab culture. The second-largest cultural group is the Kurds, who are in the highlands
66
+ and mountain valleys of the north in a politically autonomous settlement.
67
+ - source_sentence: how many years of education are needed to become a pediatric nurse
68
  sentences:
69
+ - In terms of educational background, pediatric nurse requirements include either
70
+ an Associate's or a Bachelor's degree in Nursing. An Associate's degree (ADN)
71
+ typically takes two years to complete, while a Bachelor's degree (BSN) typically
72
+ takes four years. ADN programs are usually offered by community colleges.
73
+ - "Photo of Oxford Suites Sonoma County - Rohnert Park - Rohnert Park, CA, United\
74
+ \ States Photo of Oxford Suites Sonoma County - Rohnert Park - Rohnert Park, CA,\
75
+ \ United States Living area with king bed by Monique' M. â\x80\x9CAnd there's\
76
+ \ a complimentary reception with 2 drinks, soup and salad bar nightly.â\x80\x9D\
77
+ \ in 2 reviews"
78
+ - 'From there, additional training specific to the care of children is required.
79
+ Pediatric nurses can become certified in the field and may choose to further specialize
80
+ in a particular area. Program Levels: Associate''s degree, bachelor''s degree.'
81
+ - source_sentence: Schliemann recognized five shafts and cleared them like the graves
82
+ mentioned by Pausanias .
83
  sentences:
84
+ - IBM banned the usage of the POWER5+ in its System p5 510Q, 520Q, 550Q and 560Q
85
+ servers.
86
+ - Schliemann cleared five shafts and recognized them as the graves mentioned by
87
+ Pausania .
88
+ - Schliemann recognized five shafts and cleared them like the graves mentioned by
89
+ Pausanias .
90
  pipeline_tag: sentence-similarity
91
  library_name: sentence-transformers
92
  metrics:
 
119
  value: 0.3
120
  name: Cosine Accuracy@1
121
  - type: cosine_accuracy@3
122
+ value: 0.54
123
  name: Cosine Accuracy@3
124
  - type: cosine_accuracy@5
125
+ value: 0.62
126
  name: Cosine Accuracy@5
127
  - type: cosine_accuracy@10
128
+ value: 0.76
129
  name: Cosine Accuracy@10
130
  - type: cosine_precision@1
131
  value: 0.3
132
  name: Cosine Precision@1
133
  - type: cosine_precision@3
134
+ value: 0.18
135
  name: Cosine Precision@3
136
  - type: cosine_precision@5
137
+ value: 0.124
138
  name: Cosine Precision@5
139
  - type: cosine_precision@10
140
+ value: 0.07600000000000001
141
  name: Cosine Precision@10
142
  - type: cosine_recall@1
143
  value: 0.3
144
  name: Cosine Recall@1
145
  - type: cosine_recall@3
146
+ value: 0.54
147
  name: Cosine Recall@3
148
  - type: cosine_recall@5
149
+ value: 0.62
150
  name: Cosine Recall@5
151
  - type: cosine_recall@10
152
+ value: 0.76
153
  name: Cosine Recall@10
154
  - type: cosine_ndcg@10
155
+ value: 0.5241190384704345
156
  name: Cosine Ndcg@10
157
  - type: cosine_mrr@10
158
+ value: 0.4492698412698413
159
  name: Cosine Mrr@10
160
  - type: cosine_map@100
161
+ value: 0.45777964902887497
162
  name: Cosine Map@100
163
  - task:
164
  type: information-retrieval
 
168
  type: NanoNQ
169
  metrics:
170
  - type: cosine_accuracy@1
171
+ value: 0.38
172
  name: Cosine Accuracy@1
173
  - type: cosine_accuracy@3
174
+ value: 0.52
175
  name: Cosine Accuracy@3
176
  - type: cosine_accuracy@5
177
+ value: 0.54
178
  name: Cosine Accuracy@5
179
  - type: cosine_accuracy@10
180
+ value: 0.68
181
  name: Cosine Accuracy@10
182
  - type: cosine_precision@1
183
+ value: 0.38
184
  name: Cosine Precision@1
185
  - type: cosine_precision@3
186
+ value: 0.1733333333333333
187
  name: Cosine Precision@3
188
  - type: cosine_precision@5
189
+ value: 0.11200000000000002
190
  name: Cosine Precision@5
191
  - type: cosine_precision@10
192
+ value: 0.07200000000000001
193
  name: Cosine Precision@10
194
  - type: cosine_recall@1
195
+ value: 0.35
196
  name: Cosine Recall@1
197
  - type: cosine_recall@3
198
+ value: 0.49
199
  name: Cosine Recall@3
200
  - type: cosine_recall@5
201
+ value: 0.52
202
  name: Cosine Recall@5
203
  - type: cosine_recall@10
204
+ value: 0.66
205
  name: Cosine Recall@10
206
  - type: cosine_ndcg@10
207
+ value: 0.5017561161582912
208
  name: Cosine Ndcg@10
209
  - type: cosine_mrr@10
210
+ value: 0.46857142857142864
211
  name: Cosine Mrr@10
212
  - type: cosine_map@100
213
+ value: 0.4585943213547632
214
  name: Cosine Map@100
215
  - task:
216
  type: nano-beir
 
220
  type: NanoBEIR_mean
221
  metrics:
222
  - type: cosine_accuracy@1
223
+ value: 0.33999999999999997
224
  name: Cosine Accuracy@1
225
  - type: cosine_accuracy@3
226
  value: 0.53
227
  name: Cosine Accuracy@3
228
  - type: cosine_accuracy@5
229
+ value: 0.5800000000000001
230
  name: Cosine Accuracy@5
231
  - type: cosine_accuracy@10
232
+ value: 0.72
233
  name: Cosine Accuracy@10
234
  - type: cosine_precision@1
235
+ value: 0.33999999999999997
236
  name: Cosine Precision@1
237
  - type: cosine_precision@3
238
+ value: 0.17666666666666664
239
  name: Cosine Precision@3
240
  - type: cosine_precision@5
241
+ value: 0.11800000000000001
242
  name: Cosine Precision@5
243
  - type: cosine_precision@10
244
+ value: 0.07400000000000001
245
  name: Cosine Precision@10
246
  - type: cosine_recall@1
247
+ value: 0.32499999999999996
248
  name: Cosine Recall@1
249
  - type: cosine_recall@3
250
  value: 0.515
251
  name: Cosine Recall@3
252
  - type: cosine_recall@5
253
+ value: 0.5700000000000001
254
  name: Cosine Recall@5
255
  - type: cosine_recall@10
256
+ value: 0.71
257
  name: Cosine Recall@10
258
  - type: cosine_ndcg@10
259
+ value: 0.5129375773143628
260
  name: Cosine Ndcg@10
261
  - type: cosine_mrr@10
262
+ value: 0.45892063492063495
263
  name: Cosine Mrr@10
264
  - type: cosine_map@100
265
+ value: 0.4581869851918191
266
  name: Cosine Map@100
267
  ---
268
 
 
316
  model = SentenceTransformer("redis/model-b-structured")
317
  # Run inference
318
  sentences = [
319
+ 'Schliemann recognized five shafts and cleared them like the graves mentioned by Pausanias .',
320
+ 'Schliemann recognized five shafts and cleared them like the graves mentioned by Pausanias .',
321
+ 'Schliemann cleared five shafts and recognized them as the graves mentioned by Pausania .',
322
  ]
323
  embeddings = model.encode(sentences)
324
  print(embeddings.shape)
 
327
  # Get the similarity scores for the embeddings
328
  similarities = model.similarity(embeddings, embeddings)
329
  print(similarities)
330
+ # tensor([[1.0000, 1.0000, 0.9779],
331
+ # [1.0000, 1.0000, 0.9779],
332
+ # [0.9779, 0.9779, 1.0000]])
333
  ```
334
 
335
  <!--
 
367
 
368
  | Metric | NanoMSMARCO | NanoNQ |
369
  |:--------------------|:------------|:-----------|
370
+ | cosine_accuracy@1 | 0.3 | 0.38 |
371
+ | cosine_accuracy@3 | 0.54 | 0.52 |
372
+ | cosine_accuracy@5 | 0.62 | 0.54 |
373
+ | cosine_accuracy@10 | 0.76 | 0.68 |
374
+ | cosine_precision@1 | 0.3 | 0.38 |
375
+ | cosine_precision@3 | 0.18 | 0.1733 |
376
+ | cosine_precision@5 | 0.124 | 0.112 |
377
+ | cosine_precision@10 | 0.076 | 0.072 |
378
+ | cosine_recall@1 | 0.3 | 0.35 |
379
+ | cosine_recall@3 | 0.54 | 0.49 |
380
+ | cosine_recall@5 | 0.62 | 0.52 |
381
+ | cosine_recall@10 | 0.76 | 0.66 |
382
+ | **cosine_ndcg@10** | **0.5241** | **0.5018** |
383
+ | cosine_mrr@10 | 0.4493 | 0.4686 |
384
+ | cosine_map@100 | 0.4578 | 0.4586 |
385
 
386
  #### Nano BEIR
387
 
 
399
 
400
  | Metric | Value |
401
  |:--------------------|:-----------|
402
+ | cosine_accuracy@1 | 0.34 |
403
  | cosine_accuracy@3 | 0.53 |
404
+ | cosine_accuracy@5 | 0.58 |
405
+ | cosine_accuracy@10 | 0.72 |
406
+ | cosine_precision@1 | 0.34 |
407
+ | cosine_precision@3 | 0.1767 |
408
+ | cosine_precision@5 | 0.118 |
409
+ | cosine_precision@10 | 0.074 |
410
+ | cosine_recall@1 | 0.325 |
411
  | cosine_recall@3 | 0.515 |
412
+ | cosine_recall@5 | 0.57 |
413
+ | cosine_recall@10 | 0.71 |
414
+ | **cosine_ndcg@10** | **0.5129** |
415
+ | cosine_mrr@10 | 0.4589 |
416
+ | cosine_map@100 | 0.4582 |
417
 
418
  <!--
419
  ## Bias, Risks and Limitations
 
433
 
434
  #### Unnamed Dataset
435
 
436
+ * Size: 111,470 training samples
437
  * Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
438
  * Approximate statistics based on the first 1000 samples:
439
+ | | anchor | positive | negative |
440
+ |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|
441
+ | type | string | string | string |
442
+ | details | <ul><li>min: 4 tokens</li><li>mean: 10.95 tokens</li><li>max: 60 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 67.57 tokens</li><li>max: 128 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 66.64 tokens</li><li>max: 128 tokens</li></ul> |
443
  * Samples:
444
+ | anchor | positive | negative |
445
+ |:----------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
446
+ | <code>how far is sandos caracol eco resort from cancun airport</code> | <code>The Sandos Caracol Eco Resort is 2 miles from the Church of Guadalupe and a 45-minute drive from Cancun Cancún. Airport The Gran Coral Golf Riviera maya is located within the same estate as The. Sandos we speak your! Language Hotel: rooms, 680 Hotel: Chain Sandos & Hotels. resorts</code> | <code>Featuring a spa, 8 restaurants and 2 outdoor pools, Sandos Caracol Eco Resort is set on Playa del Carmen Beach, overlooking Cozumel Island. Its rooms have balconies overlooking the Caribbean Sea. Sandos Caracol Eco Resort is in beautiful gardens and features bright accommodations.</code> |
447
+ | <code>can eggs expire</code> | <code>Here is a link from Georgia Eggs Commission about eggs and expiration dates. The following is from Swedish Medical Center Eggs: If you ve purchased a carton of eggs before the date expires, you should be able to use them safely for three to five weeks after expiration.ere is a link from Georgia Eggs Commission about eggs and expiration dates. The following is from Swedish Medical Center Eggs: If you ve purchased a carton of eggs before the date expires, you should be able to use them safely for three to five weeks after expiration.</code> | <code>The answer to this question may surprise you: while uncooked eggs typically last four to five weeks when properly refrigerated, hard-boiled eggs will only last about a week. This is because egg shells, which are highly porous, are sprayed before sale with a thin coating of mineral oil that seals the egg.</code> |
448
+ | <code>how old are first graders?</code> | <code>First Grade Worksheets Online. 6 and 7 year old kids get their first taste of real schooling in first grade. Help children learn the basics in math, reading, language and science with our printable first grade worksheets. Spelling Worksheets for 1st Grade.</code> | <code>Average BMI percentile-for-age values were 59.5 (28.8) for first-graders, 59.5 (30.5) for third-graders, and 62.4 (31.7) for fifth-graders. The number of participants classified as obese was 144 (25.6% of first-graders, 28.5% of third-graders, and 34.5% of fifth-graders). The percentage of students who reported a reasonable height or weight ranged from 20% (first grade, height) to 92% (fifth grade, weight) (Table). In general, self-report ability was better in older children and when self-reporting weight.</code> |
449
  * Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
450
  ```json
451
  {
 
462
  * Size: 12,386 evaluation samples
463
  * Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
464
  * Approximate statistics based on the first 1000 samples:
465
+ | | anchor | positive | negative |
466
+ |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|
467
+ | type | string | string | string |
468
+ | details | <ul><li>min: 4 tokens</li><li>mean: 11.11 tokens</li><li>max: 66 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 67.99 tokens</li><li>max: 128 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 66.08 tokens</li><li>max: 128 tokens</li></ul> |
469
  * Samples:
470
+ | anchor | positive | negative |
471
+ |:----------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
472
+ | <code>In 1883 , the first schools were built in the vicinity for 400 white and 60 black students .</code> | <code>In 1883 , the first schools were built in the vicinity for 400 white and 60 black students .</code> | <code>In 1883 , the first schools in the area were built for 400 black students and 60 white students .</code> |
473
+ | <code>what is the origin of the name haja</code> | <code>Haja is a Muslim baby Girl name, it is an Arabic originated name. Haja name meaning is In the heart condition through and the lucky number associated with Haja is 5. Find all the relevant details about the Haja Meaning, Origin, Lucky Number and Religion from this page. Average rating of Haja is 1 stars, based on 0 reviews.</code> | <code>Synonomis with the exclamation commonly used in urban circles Holla. Haba is derived from the term, Holla Bitches, which became Haba Litches, which eventually evolved to Habalicious, and finally became just Haba. When seeing a fine female passing by, Russell exclaimed, Haba.</code> |
474
+ | <code>what causes itch rash</code> | <code>A rash is a noticeable change in the texture or color of the skin. The skin may become itchy, bumpy, chapped, scaly, or otherwise irritated. Rashes are caused by a wide range of conditions, including allergies, medication, cosmetics, and various diseases. The rash is often reddish and itchy, with a scaly texture. 2 bug bites: tick bites are of particular concern, as they can transmit disease. 3 psoriasis: a scaly, itchy, red rash that forms along the scalp and joints. 4 dandruff: an itchy, flaky rash on the scalp.</code> | <code>Causes of Similar Symptoms to Behind knee rash. Research the causes of these symptoms that are similar to, or related to, the symptom Behind knee rash: 1 Behind knee itch (14 causes). 2 Knee rash (18 causes).3 Knee pain (122 causes). 4 Knee tingling (6 causes). 5 Knee symptoms (149 causes). 6 Skin itch (1068 causes). 7 Skin rash (461 causes). 8 Insect bite.auses of Similar Symptoms to Behind knee rash. Research the causes of these symptoms that are similar to, or related to, the symptom Behind knee rash: 1 Behind knee itch (14 causes). 2 Knee rash (18 causes).</code> |
475
  * Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
476
  ```json
477
  {
 
487
  - `eval_strategy`: steps
488
  - `per_device_train_batch_size`: 128
489
  - `per_device_eval_batch_size`: 128
490
+ - `learning_rate`: 1e-06
491
+ - `weight_decay`: 0.001
492
  - `max_steps`: 3000
493
  - `warmup_ratio`: 0.1
494
  - `fp16`: True
 
516
  - `gradient_accumulation_steps`: 1
517
  - `eval_accumulation_steps`: None
518
  - `torch_empty_cache_steps`: None
519
+ - `learning_rate`: 1e-06
520
+ - `weight_decay`: 0.001
521
  - `adam_beta1`: 0.9
522
  - `adam_beta2`: 0.999
523
  - `adam_epsilon`: 1e-08
 
630
  ### Training Logs
631
  | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_cosine_ndcg@10 | NanoNQ_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 |
632
  |:------:|:----:|:-------------:|:---------------:|:--------------------------:|:---------------------:|:----------------------------:|
633
+ | 0 | 0 | - | 4.0678 | 0.6259 | 0.6583 | 0.6421 |
634
+ | 0.2874 | 250 | 4.2246 | 3.8520 | 0.6117 | 0.6465 | 0.6291 |
635
+ | 0.5747 | 500 | 3.8138 | 3.1367 | 0.6062 | 0.6457 | 0.6260 |
636
+ | 0.8621 | 750 | 2.9174 | 1.8442 | 0.5837 | 0.5594 | 0.5715 |
637
+ | 1.1494 | 1000 | 1.8256 | 1.2096 | 0.5462 | 0.4989 | 0.5226 |
638
+ | 1.4368 | 1250 | 1.4465 | 1.0779 | 0.5347 | 0.4650 | 0.4998 |
639
+ | 1.7241 | 1500 | 1.3307 | 1.0331 | 0.5358 | 0.4801 | 0.5079 |
640
+ | 2.0115 | 1750 | 1.2785 | 1.0094 | 0.5359 | 0.4848 | 0.5104 |
641
+ | 2.2989 | 2000 | 1.249 | 0.9957 | 0.5282 | 0.4860 | 0.5071 |
642
+ | 2.5862 | 2250 | 1.228 | 0.9865 | 0.5245 | 0.4939 | 0.5092 |
643
+ | 2.8736 | 2500 | 1.2043 | 0.9809 | 0.5235 | 0.5018 | 0.5126 |
644
+ | 3.1609 | 2750 | 1.208 | 0.9771 | 0.5261 | 0.5018 | 0.5139 |
645
+ | 3.4483 | 3000 | 1.2008 | 0.9762 | 0.5241 | 0.5018 | 0.5129 |
646
 
647
 
648
  ### Framework Versions