|
|
--- |
|
|
tags: |
|
|
- sentence-transformers |
|
|
- sentence-similarity |
|
|
- feature-extraction |
|
|
- dense |
|
|
- generated_from_trainer |
|
|
- dataset_size:90000 |
|
|
- loss:MultipleNegativesRankingLoss |
|
|
base_model: thenlper/gte-small |
|
|
widget: |
|
|
- source_sentence: who is the publisher of the norton anthology american literature |
|
|
sentences: |
|
|
- W. W. Norton & Company W. W. Norton & Company is an American publishing company |
|
|
based in New York City. It has been owned wholly by its employees since the early |
|
|
1960s. The company is known for its "Norton Anthologies" (particularly The Norton |
|
|
Anthology of English Literature) and its texts in the Norton Critical Editions |
|
|
series, the latter of which are frequently assigned in university literature courses. |
|
|
- New Orleans La Nouvelle-Orléans (New Orleans) was founded in Spring of 1718 (7 |
|
|
May has become the traditional date to mark the anniversary, but the actual day |
|
|
is unknown[25]) by the French Mississippi Company, under the direction of Jean-Baptiste |
|
|
Le Moyne de Bienville, on land inhabited by the Chitimacha. It was named for Philippe |
|
|
II, Duke of Orléans, who was Regent of the Kingdom of France at the time. His |
|
|
title came from the French city of Orléans. |
|
|
- I Really Like You The music video was directed by Peter Glanz. Jepsen filmed part |
|
|
of the song's music video on 16 February 2015, in front of the Mondrian Hotel |
|
|
in Manhattan alongside Tom Hanks, Justin Bieber and a troupe of dancers. Also |
|
|
making cameo appearances in the video are Rudy Mancuso and Andrew B. Bachelor |
|
|
(A.K.A. King Bach), well-known users of the short-form video sharing application |
|
|
Vine. The video was released on 6 March 2015.[15] CBC Music's Nicolle Weeks described |
|
|
it as "a more affable version" of the music video for The Verve's "Bitter Sweet |
|
|
Symphony" (1997).[16] The music video has been rated as one of 10 Best Music Videos |
|
|
of 2015 (So Far) by the readers of Billboard.[17] |
|
|
- source_sentence: how many members on the house of representatives |
|
|
sentences: |
|
|
- 'United States House of Representatives The composition and powers of the House |
|
|
are established by Article One of the United States Constitution. The House is |
|
|
composed of Representatives who sit in congressional districts that are allocated |
|
|
to each of the 50 states on a basis of population as measured by the U.S. Census, |
|
|
with each district entitled to one representative. Since its inception in 1789, |
|
|
all Representatives have been directly elected. The total number of voting representatives |
|
|
is fixed by law at 435.[1] As of the 2010 Census, the largest delegation is that |
|
|
of California, with fifty-three representatives. Seven states have the smallest |
|
|
delegation possible, a single representative: Alaska, Delaware, Montana, North |
|
|
Dakota, South Dakota, Vermont, and Wyoming.[2]' |
|
|
- Ain't No Mountain High Enough "Ain't No Mountain High Enough" is an R&B/soul song |
|
|
written by Nickolas Ashford & Valerie Simpson in 1966 for the Tamla label, a division |
|
|
of Motown. The composition was first successful as a 1967 hit single recorded |
|
|
by Marvin Gaye and Tammi Terrell, becoming a hit again in 1970 when recorded by |
|
|
former Supremes frontwoman Diana Ross. The song became Ross' first solo number-one |
|
|
hit on the Billboard Hot 100 chart and was nominated for a Grammy Award. |
|
|
- Synthetic element In chemistry, a synthetic element is a chemical element that |
|
|
does not occur naturally on Earth, and can only be created artificially. So far, |
|
|
24 synthetic elements have been created (those with atomic numbers 95–118). |
|
|
All are unstable, decaying with half-lives ranging from 15.6 million years to |
|
|
a few hundred microseconds. |
|
|
- source_sentence: what is the meaning of mbbs and md |
|
|
sentences: |
|
|
- Adductor longus muscle Its main actions is to adduct and laterally rotate the |
|
|
thigh; it can also produce some degree of flexion/anteversion.[1] |
|
|
- Category 6 cable When used for 10/100/1000BASE-T, the maximum allowed length of |
|
|
a Cat 6 cable is up to 100 meters (328 ft). This consists of 90 meters (295 ft) |
|
|
of solid "horizontal" cabling between the patch panel and the wall jack, plus |
|
|
5 meters (16 ft) of stranded patch cable between each jack and the attached device.[7] |
|
|
For 10GBASE-T, an unshielded Cat 6 cable should not exceed 55 meters.[8] |
|
|
- Doctor of Medicine Historically, Australian medical schools have followed the |
|
|
British tradition by conferring the degrees of Bachelor of Medicine and Bachelor |
|
|
of Surgery (MBBS) to its graduates whilst reserving the title of Doctor of Medicine |
|
|
(MD) for their research training degree, analogous to the PhD, or for their honorary |
|
|
doctorates. Although the majority of Australian MBBS degrees have been graduate |
|
|
programs since the 1990s, under the previous Australian Qualifications Framework |
|
|
(AQF) they remained categorized as Level 7 Bachelor's degrees together with other |
|
|
undergraduate programs. |
|
|
- source_sentence: what holds the bone ends of an amphiarthrodial joint together |
|
|
sentences: |
|
|
- Pubic symphysis The pubic symphysis is a nonsynovial amphiarthrodial joint. The |
|
|
name comes from the Greek word "symphysis", meaning "growing together". The width |
|
|
of the pubic symphysis at the front is 3–5 mm greater than its width at the back. |
|
|
This joint is connected by fibrocartilage and may contain a fluid-filled cavity; |
|
|
the center is avascular, possibly due to the nature of the compressive forces |
|
|
passing through this joint, which may lead to harmful vascular disease.[2] The |
|
|
ends of both pubic bones are covered by a thin layer of hyaline cartilage attached |
|
|
to the fibrocartilage. The fibrocartilaginous disk is reinforced by a series of |
|
|
ligaments. These ligaments cling to the fibrocartilaginous disk to the point that |
|
|
fibers intermix with it. |
|
|
- 'John 3:16 In Exodus 4:22, the Israelites as a people are called "my firstborn |
|
|
son" by God using the singular form. In John, the focus shifts to the person of |
|
|
Jesus as representative of that title. The verse is part of the New Testament |
|
|
narrative in the third chapter of John in the discussion at Jerusalem between |
|
|
Jesus and Nicodemus, who is called a "ruler of the Jews". (v.1) After speaking |
|
|
of the necessity of a man being born again before he could "see the kingdom of |
|
|
God", (v.3) Jesus spoke also of "heavenly things" (v.11-13) and of salvation (v.14-17) |
|
|
and the condemnation (v.18,19) of those that do not believe in Jesus. "14 And |
|
|
as Moses lifted up the serpent in the wilderness, even so must the Son of man |
|
|
be lifted up: 15 That whosoever believeth in him should not perish, but have eternal |
|
|
life." (John 3:14-15) Note that verse 15 is nearly identical to the latter part |
|
|
of John 3:16.' |
|
|
- Tony Hadley Anthony Patrick Hadley (born 2 June 1960) is an English singer-songwriter, |
|
|
occasional stage actor and radio presenter. He rose to fame in the 1980s as the |
|
|
lead singer of the new wave band Spandau Ballet before launching a solo career |
|
|
following the group's split in 1990. Hadley is recognisable for his suave image,[1] |
|
|
as well as his powerful blue-eyed soul voice, which has been described by AllMusic |
|
|
as a "dramatic warble".[2] He has also been described as a "top crooner" by the |
|
|
BBC.[3] |
|
|
- source_sentence: what are the 5 liberties of the first amendment |
|
|
sentences: |
|
|
- First Amendment to the United States Constitution The First Amendment (Amendment |
|
|
I) to the United States Constitution prohibits the making of any law respecting |
|
|
an establishment of religion, ensuring that there is no prohibition on the free |
|
|
exercise of religion, abridging the freedom of speech, infringing on the freedom |
|
|
of the press, interfering with the right to peaceably assemble, or prohibiting |
|
|
the petitioning for a governmental redress of grievances. It was adopted on December |
|
|
15, 1791, as one of the ten amendments that constitute the Bill of Rights. |
|
|
- Poor People's Campaign The SCLC announced the campaign on December 4, 1967. King |
|
|
delivered a speech which identified "a kind of social insanity which could lead |
|
|
to national ruin."[23] In January 1968, the SCLC created and distributed an "Economic |
|
|
Fact Sheet" with statistics explaining why the campaign was necessary.[24] King |
|
|
avoided providing specific details about the campaign and attempted to redirect |
|
|
media attention to the values at stake.[25] The Poor People’s Campaign held firm |
|
|
to the movement’s commitment to non-violence. “We are custodians of the philosophy |
|
|
of non-violence,” said King at a press conference. “And it has worked”.[9] King |
|
|
originally wanted the Poor People's Campaign to start in Quitman County, Mississippi |
|
|
because of the intense and visible economic disparity there.[26] |
|
|
- 'Jake and the Never Land Pirates Jake and the Never Land Pirates (also known as |
|
|
Captain Jake and the Never Land Pirates in the fourth season and associated merchandise[1]) |
|
|
is an Annie Award-winning musical and interactive American children''s animated |
|
|
television series shown on Disney Junior. It is based on Disney''s Peter Pan franchise, |
|
|
which in turn is based on the famous book and play by British author J. M. Barrie. |
|
|
It is the first Disney Junior original show following the switch from Playhouse |
|
|
Disney. It stars Sean Ryan Fox from Henry Danger, Megan Richie, Jadon Sand, David |
|
|
Arquette, Corey Burton, Jeff Bennett, Loren Hoskins and Dee Bradley Baker. The |
|
|
title character Jake was previously voiced by Colin Ford, and then later by Cameron |
|
|
Boyce, while Izzy was voiced for the first three seasons by Madison Pettis and |
|
|
Cubby was voiced by Jonathan Morgan Heit. The series is created by Disney veteran |
|
|
Bobs Gannaway, whose works include another Disney Junior series, Mickey Mouse |
|
|
Clubhouse, and films such as Secret of the Wings, The Pirate Fairy and Planes: |
|
|
Fire & Rescue. The last episode aired on November 6, 2016.' |
|
|
pipeline_tag: sentence-similarity |
|
|
library_name: sentence-transformers |
|
|
metrics: |
|
|
- cosine_accuracy@1 |
|
|
- cosine_accuracy@3 |
|
|
- cosine_accuracy@5 |
|
|
- cosine_accuracy@10 |
|
|
- cosine_precision@1 |
|
|
- cosine_precision@3 |
|
|
- cosine_precision@5 |
|
|
- cosine_precision@10 |
|
|
- cosine_recall@1 |
|
|
- cosine_recall@3 |
|
|
- cosine_recall@5 |
|
|
- cosine_recall@10 |
|
|
- cosine_ndcg@10 |
|
|
- cosine_mrr@10 |
|
|
- cosine_map@100 |
|
|
model-index: |
|
|
- name: SentenceTransformer based on thenlper/gte-small |
|
|
results: |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: NanoMSMARCO |
|
|
type: NanoMSMARCO |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.38 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@3 |
|
|
value: 0.58 |
|
|
name: Cosine Accuracy@3 |
|
|
- type: cosine_accuracy@5 |
|
|
value: 0.66 |
|
|
name: Cosine Accuracy@5 |
|
|
- type: cosine_accuracy@10 |
|
|
value: 0.8 |
|
|
name: Cosine Accuracy@10 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.38 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@3 |
|
|
value: 0.19333333333333336 |
|
|
name: Cosine Precision@3 |
|
|
- type: cosine_precision@5 |
|
|
value: 0.132 |
|
|
name: Cosine Precision@5 |
|
|
- type: cosine_precision@10 |
|
|
value: 0.08 |
|
|
name: Cosine Precision@10 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.38 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@3 |
|
|
value: 0.58 |
|
|
name: Cosine Recall@3 |
|
|
- type: cosine_recall@5 |
|
|
value: 0.66 |
|
|
name: Cosine Recall@5 |
|
|
- type: cosine_recall@10 |
|
|
value: 0.8 |
|
|
name: Cosine Recall@10 |
|
|
- type: cosine_ndcg@10 |
|
|
value: 0.5747352409361379 |
|
|
name: Cosine Ndcg@10 |
|
|
- type: cosine_mrr@10 |
|
|
value: 0.5051349206349205 |
|
|
name: Cosine Mrr@10 |
|
|
- type: cosine_map@100 |
|
|
value: 0.5163932476955072 |
|
|
name: Cosine Map@100 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: NanoNQ |
|
|
type: NanoNQ |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.46 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@3 |
|
|
value: 0.64 |
|
|
name: Cosine Accuracy@3 |
|
|
- type: cosine_accuracy@5 |
|
|
value: 0.72 |
|
|
name: Cosine Accuracy@5 |
|
|
- type: cosine_accuracy@10 |
|
|
value: 0.74 |
|
|
name: Cosine Accuracy@10 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.46 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@3 |
|
|
value: 0.21333333333333332 |
|
|
name: Cosine Precision@3 |
|
|
- type: cosine_precision@5 |
|
|
value: 0.15200000000000002 |
|
|
name: Cosine Precision@5 |
|
|
- type: cosine_precision@10 |
|
|
value: 0.08 |
|
|
name: Cosine Precision@10 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.44 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@3 |
|
|
value: 0.6 |
|
|
name: Cosine Recall@3 |
|
|
- type: cosine_recall@5 |
|
|
value: 0.69 |
|
|
name: Cosine Recall@5 |
|
|
- type: cosine_recall@10 |
|
|
value: 0.72 |
|
|
name: Cosine Recall@10 |
|
|
- type: cosine_ndcg@10 |
|
|
value: 0.5958872018988118 |
|
|
name: Cosine Ndcg@10 |
|
|
- type: cosine_mrr@10 |
|
|
value: 0.5641904761904761 |
|
|
name: Cosine Mrr@10 |
|
|
- type: cosine_map@100 |
|
|
value: 0.5591780429569271 |
|
|
name: Cosine Map@100 |
|
|
- task: |
|
|
type: nano-beir |
|
|
name: Nano BEIR |
|
|
dataset: |
|
|
name: NanoBEIR mean |
|
|
type: NanoBEIR_mean |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.42000000000000004 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@3 |
|
|
value: 0.61 |
|
|
name: Cosine Accuracy@3 |
|
|
- type: cosine_accuracy@5 |
|
|
value: 0.69 |
|
|
name: Cosine Accuracy@5 |
|
|
- type: cosine_accuracy@10 |
|
|
value: 0.77 |
|
|
name: Cosine Accuracy@10 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.42000000000000004 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@3 |
|
|
value: 0.20333333333333334 |
|
|
name: Cosine Precision@3 |
|
|
- type: cosine_precision@5 |
|
|
value: 0.14200000000000002 |
|
|
name: Cosine Precision@5 |
|
|
- type: cosine_precision@10 |
|
|
value: 0.08 |
|
|
name: Cosine Precision@10 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.41000000000000003 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@3 |
|
|
value: 0.59 |
|
|
name: Cosine Recall@3 |
|
|
- type: cosine_recall@5 |
|
|
value: 0.675 |
|
|
name: Cosine Recall@5 |
|
|
- type: cosine_recall@10 |
|
|
value: 0.76 |
|
|
name: Cosine Recall@10 |
|
|
- type: cosine_ndcg@10 |
|
|
value: 0.5853112214174749 |
|
|
name: Cosine Ndcg@10 |
|
|
- type: cosine_mrr@10 |
|
|
value: 0.5346626984126983 |
|
|
name: Cosine Mrr@10 |
|
|
- type: cosine_map@100 |
|
|
value: 0.5377856453262171 |
|
|
name: Cosine Map@100 |
|
|
--- |
|
|
|
|
|
# SentenceTransformer based on thenlper/gte-small |
|
|
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [thenlper/gte-small](https://huggingface.co/thenlper/gte-small). It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
- **Model Type:** Sentence Transformer |
|
|
- **Base model:** [thenlper/gte-small](https://huggingface.co/thenlper/gte-small) <!-- at revision 17e1f347d17fe144873b1201da91788898c639cd --> |
|
|
- **Maximum Sequence Length:** 128 tokens |
|
|
- **Output Dimensionality:** 384 dimensions |
|
|
- **Similarity Function:** Cosine Similarity |
|
|
<!-- - **Training Dataset:** Unknown --> |
|
|
<!-- - **Language:** Unknown --> |
|
|
<!-- - **License:** Unknown --> |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
|
|
- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) |
|
|
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) |
|
|
|
|
|
### Full Model Architecture |
|
|
|
|
|
``` |
|
|
SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'BertModel'}) |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
) |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Direct Usage (Sentence Transformers) |
|
|
|
|
|
First install the Sentence Transformers library: |
|
|
|
|
|
```bash |
|
|
pip install -U sentence-transformers |
|
|
``` |
|
|
|
|
|
Then you can load this model and run inference. |
|
|
```python |
|
|
from sentence_transformers import SentenceTransformer |
|
|
|
|
|
# Download from the 🤗 Hub |
|
|
model = SentenceTransformer("redis/model-a-baseline") |
|
|
# Run inference |
|
|
sentences = [ |
|
|
'what are the 5 liberties of the first amendment', |
|
|
'First Amendment to the United States Constitution The First Amendment (Amendment I) to the United States Constitution prohibits the making of any law respecting an establishment of religion, ensuring that there is no prohibition on the free exercise of religion, abridging the freedom of speech, infringing on the freedom of the press, interfering with the right to peaceably assemble, or prohibiting the petitioning for a governmental redress of grievances. It was adopted on December 15, 1791, as one of the ten amendments that constitute the Bill of Rights.', |
|
|
"Jake and the Never Land Pirates Jake and the Never Land Pirates (also known as Captain Jake and the Never Land Pirates in the fourth season and associated merchandise[1]) is an Annie Award-winning musical and interactive American children's animated television series shown on Disney Junior. It is based on Disney's Peter Pan franchise, which in turn is based on the famous book and play by British author J. M. Barrie. It is the first Disney Junior original show following the switch from Playhouse Disney. It stars Sean Ryan Fox from Henry Danger, Megan Richie, Jadon Sand, David Arquette, Corey Burton, Jeff Bennett, Loren Hoskins and Dee Bradley Baker. The title character Jake was previously voiced by Colin Ford, and then later by Cameron Boyce, while Izzy was voiced for the first three seasons by Madison Pettis and Cubby was voiced by Jonathan Morgan Heit. The series is created by Disney veteran Bobs Gannaway, whose works include another Disney Junior series, Mickey Mouse Clubhouse, and films such as Secret of the Wings, The Pirate Fairy and Planes: Fire & Rescue. The last episode aired on November 6, 2016.", |
|
|
] |
|
|
embeddings = model.encode(sentences) |
|
|
print(embeddings.shape) |
|
|
# [3, 384] |
|
|
|
|
|
# Get the similarity scores for the embeddings |
|
|
similarities = model.similarity(embeddings, embeddings) |
|
|
print(similarities) |
|
|
# tensor([[1.0000, 0.7343, 0.0079], |
|
|
# [0.7343, 1.0000, 0.0383], |
|
|
# [0.0079, 0.0383, 1.0000]]) |
|
|
``` |
|
|
|
|
|
<!-- |
|
|
### Direct Usage (Transformers) |
|
|
|
|
|
<details><summary>Click to see the direct usage in Transformers</summary> |
|
|
|
|
|
</details> |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
### Downstream Usage (Sentence Transformers) |
|
|
|
|
|
You can finetune this model on your own dataset. |
|
|
|
|
|
<details><summary>Click to expand</summary> |
|
|
|
|
|
</details> |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
### Out-of-Scope Use |
|
|
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.* |
|
|
--> |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Metrics |
|
|
|
|
|
#### Information Retrieval |
|
|
|
|
|
* Datasets: `NanoMSMARCO` and `NanoNQ` |
|
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
|
|
| Metric | NanoMSMARCO | NanoNQ | |
|
|
|:--------------------|:------------|:-----------| |
|
|
| cosine_accuracy@1 | 0.38 | 0.46 | |
|
|
| cosine_accuracy@3 | 0.58 | 0.64 | |
|
|
| cosine_accuracy@5 | 0.66 | 0.72 | |
|
|
| cosine_accuracy@10 | 0.8 | 0.74 | |
|
|
| cosine_precision@1 | 0.38 | 0.46 | |
|
|
| cosine_precision@3 | 0.1933 | 0.2133 | |
|
|
| cosine_precision@5 | 0.132 | 0.152 | |
|
|
| cosine_precision@10 | 0.08 | 0.08 | |
|
|
| cosine_recall@1 | 0.38 | 0.44 | |
|
|
| cosine_recall@3 | 0.58 | 0.6 | |
|
|
| cosine_recall@5 | 0.66 | 0.69 | |
|
|
| cosine_recall@10 | 0.8 | 0.72 | |
|
|
| **cosine_ndcg@10** | **0.5747** | **0.5959** | |
|
|
| cosine_mrr@10 | 0.5051 | 0.5642 | |
|
|
| cosine_map@100 | 0.5164 | 0.5592 | |
|
|
|
|
|
#### Nano BEIR |
|
|
|
|
|
* Dataset: `NanoBEIR_mean` |
|
|
* Evaluated with [<code>NanoBEIREvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.NanoBEIREvaluator) with these parameters: |
|
|
```json |
|
|
{ |
|
|
"dataset_names": [ |
|
|
"msmarco", |
|
|
"nq" |
|
|
], |
|
|
"dataset_id": "lightonai/NanoBEIR-en" |
|
|
} |
|
|
``` |
|
|
|
|
|
| Metric | Value | |
|
|
|:--------------------|:-----------| |
|
|
| cosine_accuracy@1 | 0.42 | |
|
|
| cosine_accuracy@3 | 0.61 | |
|
|
| cosine_accuracy@5 | 0.69 | |
|
|
| cosine_accuracy@10 | 0.77 | |
|
|
| cosine_precision@1 | 0.42 | |
|
|
| cosine_precision@3 | 0.2033 | |
|
|
| cosine_precision@5 | 0.142 | |
|
|
| cosine_precision@10 | 0.08 | |
|
|
| cosine_recall@1 | 0.41 | |
|
|
| cosine_recall@3 | 0.59 | |
|
|
| cosine_recall@5 | 0.675 | |
|
|
| cosine_recall@10 | 0.76 | |
|
|
| **cosine_ndcg@10** | **0.5853** | |
|
|
| cosine_mrr@10 | 0.5347 | |
|
|
| cosine_map@100 | 0.5378 | |
|
|
|
|
|
<!-- |
|
|
## Bias, Risks and Limitations |
|
|
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
### Recommendations |
|
|
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
|
|
--> |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Dataset |
|
|
|
|
|
#### Unnamed Dataset |
|
|
|
|
|
* Size: 90,000 training samples |
|
|
* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | negative | |
|
|
|:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| |
|
|
| type | string | string | string | |
|
|
| details | <ul><li>min: 9 tokens</li><li>mean: 11.82 tokens</li><li>max: 27 tokens</li></ul> | <ul><li>min: 19 tokens</li><li>mean: 106.2 tokens</li><li>max: 128 tokens</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 104.63 tokens</li><li>max: 128 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | negative | |
|
|
|:----------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
|
| <code>who played in the movie throw momma from the train</code> | <code>Anne Ramsey Angelina (Anne) Ramsey (March 27, 1929[1] – August 11, 1988) was an American stage, television, and film actress. She was best known for portraying Mama Fratelli in The Goonies (1985) and Mrs. Lift, mother of Danny DeVito's protagonist, in Throw Momma from the Train (1987). The latter film saw Ramsey nominated for a Golden Globe Award and the Academy Award for Best Supporting Actress.</code> | <code>Aye Mere Watan Ke Logo "Aye Mere Watan Ke Logo" (Hindi: ऐ मेरे वतन के लोगों; "O' people of my country") is a Hindi patriotic song written by Kavi Pradeep, composed by C. Ramchandra, and performed by Lata Mangeshkar. The song commemorates Indian soldiers who died during the Sino-Indian War in 1962. The song was first performed live by Mangeshkar on 27 January 1963 at the National Stadium in New Delhi in the presence of President Sarvepalli Radhakrishnan and Prime Minister Jawaharlal Nehru, on account of Republic Day (26 January) 1963, which was just two months after the end of the war.</code> | |
|
|
| <code>when was the wall in san diego built</code> | <code>Mexico–United States barrier In September 2017, the U.S. government announced the start of construction of eight prototype barriers made from concrete and other materials.[51][52] On June 3, 2018 the San Diego section of the US border wall construction began. [53]</code> | <code>The Dance of Dragons At the Wall, Jon Snow (Kit Harington) retreats from Hardhome defeated, accompanied by the surviving wildlings, much to the chagrin of some of the Night's Watch. In the North, Stannis Baratheon (Stephen Dillane) reluctantly allows Melisandre (Carice van Houten) to sacrifice his daughter Shireen (Kerry Ingram) after Ramsay Bolton (Iwan Rheon) sabotages his resources, resulting to his army's damaged morale. In Braavos, Arya Stark (Maisie Williams), detours from her mission given by Jaqen H'ghar (Tom Wlaschiha) to reconnoiter Meryn Trant (Ian Beattie) instead. In Dorne, Jaime Lannister (Nikolaj Coster-Waldau) secures Myrcella Baratheon's (Nell Tiger Free) release from Doran Martell's (Alexander Siddig) court against an indignant Ellaria Sand (Indira Varma). In Meereen, the Sons of the Harpy attack the stadium of Daznak's Pit in an attempt to assassinate Daenerys Targaryen (Emilia Clarke), who is rescued by Jorah Mormont (Iain Glen) and her firstborn dragon, Drogon. Lea...</code> | |
|
|
| <code>urology is the study of diseases of the</code> | <code>Urology Urology (from Greek οὖρον ouron "urine" and -λογία -logia "study of"), also known as genitourinary surgery, is the branch of medicine that focuses on surgical and medical diseases of the male and female urinary-tract system and the male reproductive organs. Organs under the domain of urology include the kidneys, adrenal glands, ureters, urinary bladder, urethra, and the male reproductive organs (testes, epididymis, vas deferens, seminal vesicles, prostate, and penis).</code> | <code>The Jackie Gleason Show By far the most memorable and popular of Gleason's characters was blowhard Brooklyn bus driver Ralph Kramden, featured originally in a series of Cavalcade skits known as "The Honeymooners", with Pert Kelton as his wife Alice, and Art Carney as his upstairs neighbor Ed Norton. These were so popular that in 1955 Gleason suspended the variety format and filmed The Honeymooners as a regular half-hour sitcom (television's first spin-off), co-starring Carney, Audrey Meadows (who had replaced the blacklisted Kelton after the earlier move to CBS), and Joyce Randolph. Finishing 19th in the ratings, these 39 episodes were subsequently rerun constantly in syndication, often five nights a week, with the cycle repeating every two months for decades. They are probably the most familiar body of work from 1950s television with the exception of I Love Lucy starring Lucille Ball and Desi Arnaz.</code> | |
|
|
* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: |
|
|
```json |
|
|
{ |
|
|
"scale": 20.0, |
|
|
"similarity_fct": "cos_sim", |
|
|
"gather_across_devices": false |
|
|
} |
|
|
``` |
|
|
|
|
|
### Evaluation Dataset |
|
|
|
|
|
#### Unnamed Dataset |
|
|
|
|
|
* Size: 10,000 evaluation samples |
|
|
* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | negative | |
|
|
|:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| |
|
|
| type | string | string | string | |
|
|
| details | <ul><li>min: 9 tokens</li><li>mean: 11.76 tokens</li><li>max: 24 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 105.95 tokens</li><li>max: 128 tokens</li></ul> | <ul><li>min: 11 tokens</li><li>mean: 105.69 tokens</li><li>max: 128 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | negative | |
|
|
|:-------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
|
| <code>what is the political system that the united states follows in regards to elections</code> | <code>Primary election The United States is one of few countries to select candidates through popular vote in a primary election system; most countries rely on party leaders to vote candidates, as was previously the case in the U.S.[9] In modern politics, primary elections have been described as a significant vehicle for taking decision-making from political insiders to the voters, though this is disputed by select political science research.[10] The selection of candidates for federal, state, and local general elections takes place in primary elections organized by the public administration for the general voting public to participate in for the purpose of nominating the respective parties' official candidates; state voters start the electoral process for governors and legislators through the primary process, as well as for many local officials from city councilors to county commissioners.[11] The candidate who moves from the primary to be successful in the general election takes public off...</code> | <code>Bob Gaudio Robert John "Bob" Gaudio (born November 17, 1942) is an American singer, songwriter, musician, and record producer, and the keyboardist/backing vocalist for The Four Seasons.</code> | |
|
|
| <code>what caused the unusual landscape at the valley of fire</code> | <code>Valley of Fire State Park Complex uplifting and faulting of the region, followed by extensive erosion, have created the present landscape. The rough floor and jagged walls of the park contain brilliant formations of eroded sandstone and sand dunes more than 150 million years old. Other important rock formations include limestones, shales, and conglomerates.[4]</code> | <code>Fundamental Constitutions of Carolina Because the Fundamental Constitutions were drafted during John Locke's service to one of Province of Carolina proprietors, Anthony Ashley Cooper, it is widely alleged that Locke had a major role in the making of the Constitutions. In the view of historian David Armitage[5] and political scientist Vicki Hsueh, the Constitutions were co-authored by Locke and his patron Cooper, known also as 1st Earl of Shaftesbury.[6] However the document was a legal document written for and signed and sealed by the eight Lord proprietors to whom Charles II had granted the colony; Locke was only a paid secretary. He wrote it much as a lawyer writes a will.[4]</code> | |
|
|
| <code>according to the guinness world records what author has the most published works</code> | <code>Guinness World Records The book itself holds a world record, as the best-selling copyrighted book of all time. As of the 2017 edition, it is now in its 62nd year of publication, published in 100 countries and 23 languages. The international franchise has extended beyond print to include television series and museums. The popularity of the franchise has resulted in Guinness World Records becoming the primary international authority on the cataloguing and verification of a huge number of world records; the organisation employs official record adjudicators authorised to verify the authenticity of the setting and breaking of records.[2]</code> | <code>The Big Chill (film) The Big Chill is a 1983 American comedy-drama film directed by Lawrence Kasdan, starring Tom Berenger, Glenn Close, Jeff Goldblum, William Hurt, Kevin Kline, Mary Kay Place, Meg Tilly, and JoBeth Williams. The plot focuses on a group of baby boomers who attended the University of Michigan, reuniting after 15 years when their friend Alex commits suicide. Kevin Costner was cast as Alex, but all scenes showing his face were cut. It was filmed in Beaufort, South Carolina.[2]</code> | |
|
|
* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: |
|
|
```json |
|
|
{ |
|
|
"scale": 20.0, |
|
|
"similarity_fct": "cos_sim", |
|
|
"gather_across_devices": false |
|
|
} |
|
|
``` |
|
|
|
|
|
### Training Hyperparameters |
|
|
#### Non-Default Hyperparameters |
|
|
|
|
|
- `eval_strategy`: steps |
|
|
- `per_device_train_batch_size`: 128 |
|
|
- `per_device_eval_batch_size`: 128 |
|
|
- `learning_rate`: 8e-05 |
|
|
- `weight_decay`: 0.005 |
|
|
- `max_steps`: 1125 |
|
|
- `warmup_ratio`: 0.1 |
|
|
- `fp16`: True |
|
|
- `dataloader_drop_last`: True |
|
|
- `dataloader_num_workers`: 1 |
|
|
- `dataloader_prefetch_factor`: 1 |
|
|
- `load_best_model_at_end`: True |
|
|
- `optim`: adamw_torch |
|
|
- `ddp_find_unused_parameters`: False |
|
|
- `push_to_hub`: True |
|
|
- `hub_model_id`: redis/model-a-baseline |
|
|
- `eval_on_start`: True |
|
|
|
|
|
#### All Hyperparameters |
|
|
<details><summary>Click to expand</summary> |
|
|
|
|
|
- `overwrite_output_dir`: False |
|
|
- `do_predict`: False |
|
|
- `eval_strategy`: steps |
|
|
- `prediction_loss_only`: True |
|
|
- `per_device_train_batch_size`: 128 |
|
|
- `per_device_eval_batch_size`: 128 |
|
|
- `per_gpu_train_batch_size`: None |
|
|
- `per_gpu_eval_batch_size`: None |
|
|
- `gradient_accumulation_steps`: 1 |
|
|
- `eval_accumulation_steps`: None |
|
|
- `torch_empty_cache_steps`: None |
|
|
- `learning_rate`: 8e-05 |
|
|
- `weight_decay`: 0.005 |
|
|
- `adam_beta1`: 0.9 |
|
|
- `adam_beta2`: 0.999 |
|
|
- `adam_epsilon`: 1e-08 |
|
|
- `max_grad_norm`: 1.0 |
|
|
- `num_train_epochs`: 3.0 |
|
|
- `max_steps`: 1125 |
|
|
- `lr_scheduler_type`: linear |
|
|
- `lr_scheduler_kwargs`: {} |
|
|
- `warmup_ratio`: 0.1 |
|
|
- `warmup_steps`: 0 |
|
|
- `log_level`: passive |
|
|
- `log_level_replica`: warning |
|
|
- `log_on_each_node`: True |
|
|
- `logging_nan_inf_filter`: True |
|
|
- `save_safetensors`: True |
|
|
- `save_on_each_node`: False |
|
|
- `save_only_model`: False |
|
|
- `restore_callback_states_from_checkpoint`: False |
|
|
- `no_cuda`: False |
|
|
- `use_cpu`: False |
|
|
- `use_mps_device`: False |
|
|
- `seed`: 42 |
|
|
- `data_seed`: None |
|
|
- `jit_mode_eval`: False |
|
|
- `bf16`: False |
|
|
- `fp16`: True |
|
|
- `fp16_opt_level`: O1 |
|
|
- `half_precision_backend`: auto |
|
|
- `bf16_full_eval`: False |
|
|
- `fp16_full_eval`: False |
|
|
- `tf32`: None |
|
|
- `local_rank`: 0 |
|
|
- `ddp_backend`: None |
|
|
- `tpu_num_cores`: None |
|
|
- `tpu_metrics_debug`: False |
|
|
- `debug`: [] |
|
|
- `dataloader_drop_last`: True |
|
|
- `dataloader_num_workers`: 1 |
|
|
- `dataloader_prefetch_factor`: 1 |
|
|
- `past_index`: -1 |
|
|
- `disable_tqdm`: False |
|
|
- `remove_unused_columns`: True |
|
|
- `label_names`: None |
|
|
- `load_best_model_at_end`: True |
|
|
- `ignore_data_skip`: False |
|
|
- `fsdp`: [] |
|
|
- `fsdp_min_num_params`: 0 |
|
|
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
|
|
- `fsdp_transformer_layer_cls_to_wrap`: None |
|
|
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
|
|
- `parallelism_config`: None |
|
|
- `deepspeed`: None |
|
|
- `label_smoothing_factor`: 0.0 |
|
|
- `optim`: adamw_torch |
|
|
- `optim_args`: None |
|
|
- `adafactor`: False |
|
|
- `group_by_length`: False |
|
|
- `length_column_name`: length |
|
|
- `project`: huggingface |
|
|
- `trackio_space_id`: trackio |
|
|
- `ddp_find_unused_parameters`: False |
|
|
- `ddp_bucket_cap_mb`: None |
|
|
- `ddp_broadcast_buffers`: False |
|
|
- `dataloader_pin_memory`: True |
|
|
- `dataloader_persistent_workers`: False |
|
|
- `skip_memory_metrics`: True |
|
|
- `use_legacy_prediction_loop`: False |
|
|
- `push_to_hub`: True |
|
|
- `resume_from_checkpoint`: None |
|
|
- `hub_model_id`: redis/model-a-baseline |
|
|
- `hub_strategy`: every_save |
|
|
- `hub_private_repo`: None |
|
|
- `hub_always_push`: False |
|
|
- `hub_revision`: None |
|
|
- `gradient_checkpointing`: False |
|
|
- `gradient_checkpointing_kwargs`: None |
|
|
- `include_inputs_for_metrics`: False |
|
|
- `include_for_metrics`: [] |
|
|
- `eval_do_concat_batches`: True |
|
|
- `fp16_backend`: auto |
|
|
- `push_to_hub_model_id`: None |
|
|
- `push_to_hub_organization`: None |
|
|
- `mp_parameters`: |
|
|
- `auto_find_batch_size`: False |
|
|
- `full_determinism`: False |
|
|
- `torchdynamo`: None |
|
|
- `ray_scope`: last |
|
|
- `ddp_timeout`: 1800 |
|
|
- `torch_compile`: False |
|
|
- `torch_compile_backend`: None |
|
|
- `torch_compile_mode`: None |
|
|
- `include_tokens_per_second`: False |
|
|
- `include_num_input_tokens_seen`: no |
|
|
- `neftune_noise_alpha`: None |
|
|
- `optim_target_modules`: None |
|
|
- `batch_eval_metrics`: False |
|
|
- `eval_on_start`: True |
|
|
- `use_liger_kernel`: False |
|
|
- `liger_kernel_config`: None |
|
|
- `eval_use_gather_object`: False |
|
|
- `average_tokens_across_devices`: True |
|
|
- `prompts`: None |
|
|
- `batch_sampler`: batch_sampler |
|
|
- `multi_dataset_batch_sampler`: proportional |
|
|
- `router_mapping`: {} |
|
|
- `learning_rate_mapping`: {} |
|
|
|
|
|
</details> |
|
|
|
|
|
### Training Logs |
|
|
| Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_cosine_ndcg@10 | NanoNQ_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 | |
|
|
|:------:|:----:|:-------------:|:---------------:|:--------------------------:|:---------------------:|:----------------------------:| |
|
|
| 0 | 0 | - | 2.1869 | 0.6259 | 0.6583 | 0.6421 | |
|
|
| 0.3556 | 250 | 0.3907 | 0.0761 | 0.5880 | 0.6146 | 0.6013 | |
|
|
| 0.7112 | 500 | 0.0814 | 0.0680 | 0.5666 | 0.6170 | 0.5918 | |
|
|
| 1.0669 | 750 | 0.0714 | 0.0634 | 0.5580 | 0.5846 | 0.5713 | |
|
|
| 1.4225 | 1000 | 0.0406 | 0.0614 | 0.5747 | 0.5959 | 0.5853 | |
|
|
|
|
|
|
|
|
### Framework Versions |
|
|
- Python: 3.10.18 |
|
|
- Sentence Transformers: 5.2.0 |
|
|
- Transformers: 4.57.3 |
|
|
- PyTorch: 2.9.1+cu128 |
|
|
- Accelerate: 1.12.0 |
|
|
- Datasets: 2.21.0 |
|
|
- Tokenizers: 0.22.1 |
|
|
|
|
|
## Citation |
|
|
|
|
|
### BibTeX |
|
|
|
|
|
#### Sentence Transformers |
|
|
```bibtex |
|
|
@inproceedings{reimers-2019-sentence-bert, |
|
|
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
|
|
author = "Reimers, Nils and Gurevych, Iryna", |
|
|
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
|
|
month = "11", |
|
|
year = "2019", |
|
|
publisher = "Association for Computational Linguistics", |
|
|
url = "https://arxiv.org/abs/1908.10084", |
|
|
} |
|
|
``` |
|
|
|
|
|
#### MultipleNegativesRankingLoss |
|
|
```bibtex |
|
|
@misc{henderson2017efficient, |
|
|
title={Efficient Natural Language Response Suggestion for Smart Reply}, |
|
|
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, |
|
|
year={2017}, |
|
|
eprint={1705.00652}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL} |
|
|
} |
|
|
``` |
|
|
|
|
|
<!-- |
|
|
## Glossary |
|
|
|
|
|
*Clearly define terms in order to be accessible across audiences.* |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
## Model Card Authors |
|
|
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
## Model Card Contact |
|
|
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
|
|
--> |
