🔁 Fine-tuned on custom STEM corpus

README.md

@@ -1,173 +1,552 @@
 ---
-language: en
-license: apache-2.0
-library_name: sentence-transformers
 tags:
 - sentence-transformers
-- feature-extraction
 - sentence-similarity
-- transformers
-datasets:
-- s2orc
-- flax-sentence-embeddings/stackexchange_xml
-- ms_marco
-- gooaq
-- yahoo_answers_topics
-- code_search_net
-- search_qa
-- eli5
-- snli
-- multi_nli
-- wikihow
-- natural_questions
-- trivia_qa
-- embedding-data/sentence-compression
-- embedding-data/flickr30k-captions
-- embedding-data/altlex
-- embedding-data/simple-wiki
-- embedding-data/QQP
-- embedding-data/SPECTER
-- embedding-data/PAQ_pairs
-- embedding-data/WikiAnswers
+- feature-extraction
+- generated_from_trainer
+- dataset_size:5489
+- loss:MultipleNegativesRankingLoss
+base_model: zacbrld/MNLP_M2_document_encoder
+widget:
+- source_sentence: Military activity affects the physical geology. This was first
+    noted through the intensive shelling on the Western Front during World War I,
+    which caused the shattering of the bedrock and changed the rocks' permeability.
+    New minerals, rocks, and land-forms are also a byproduct of nuclear testing.
+  sentences:
+  - 'Silicon can form sigma bonds to other silicon atoms (and disilane is the parent
+    of this class of compounds). However, it is difficult to prepare and isolate SinH2n+2
+    (analogous to the saturated alkane hydrocarbons) with n greater than about 8,
+    as their thermal stability decreases with increases in the number of silicon atoms.  Silanes
+    higher in molecular weight than disilane decompose to polymeric polysilicon hydride
+    and hydrogen.  But with a suitable pair of organic substituents in place of hydrogen
+    on each silicon it is possible to prepare polysilanes (sometimes, erroneously
+    called polysilenes) that are analogues of alkanes. These long chain compounds
+    have surprising electronic properties - high electrical conductivity, for example
+    - arising from sigma delocalization of the electrons in the chain.
+
+    Even silicon–silicon pi bonds are possible. However, these bonds are less stable
+    than the carbon analogues. Disilane and longer silanes are quite reactive compared
+    to alkanes. Disilene and disilynes are quite rare, unlike alkenes and alkynes.
+    Examples of disilynes, long thought to be too unstable to be isolated were reported
+    in 2004.'
+  - 'The increasing sophistication of brain-reading technologies has led many to investigate
+    their potential applications for lie detection. Legally required brain scans arguably
+    violate “the guarantee against self-incrimination” because they differ from acceptable
+    forms of bodily evidence, such as fingerprints or blood samples, in an important
+    way: they are not simply physical, hard evidence, but evidence that is intimately
+    linked to the defendant''s mind. Under US law, brain-scanning technologies might
+    also raise implications for the Fourth Amendment, calling into question whether
+    they constitute an unreasonable search and seizure.'
+  - Military activity affects the physical geology. This was first noted through the
+    intensive shelling on the Western Front during World War I, which caused the shattering
+    of the bedrock and changed the rocks' permeability. New minerals, rocks, and land-forms
+    are also a byproduct of nuclear testing.
+- source_sentence: Right after a bombing in Moscow on September 6, 1999, several anti-nuclear
+    activists were detained under suspicion. Vladimir Slivyak was one of the three
+    arrested under suspicion. He was an activist in the anti-nuclear movement and
+    a Voronezh action camp organizer. After the bombing Slivyak was pushed into a
+    car by several men who claimed to be Moscow police. The police interrogated and
+    threatened Slivyak for around ninety minutes before letting him go. The Moscow
+    police thought environmentalists from the anti-nuclear movement were associated
+    with the bombing since an earlier bombing occurred on August 31 at Manezh Palace
+    in Moscow . After the incident, on August 31, several more bombings occurred which
+    agitated many people, leading to the racially profiled arrest of dark-skinned
+    Muscovites and visitors to the Russian capital.
+  sentences:
+  - The technique works backwards from the target to identify a precursor molecule
+    and an enzyme that converts it into the target, and then a second precursor that
+    can produce the first and so on until a simple, inexpensive molecule becomes the
+    beginning of the series. For each precursor, the enzyme is evolved using induced
+    mutations and natural selection to produce a more productive version. The evolutionary
+    process can be repeated over multiple generations until acceptable productivity
+    is achieved. The process does not require high temperature, high pressure, the
+    use of exotic catalysts or other elements that can increase costs. The enzyme
+    "optimizations" that increase the production of one precursor from another are
+    cumulative in that the same precursor productivity improvements can potentially
+    be leveraged across multiple target molecules.
+  - Right after a bombing in Moscow on September 6, 1999, several anti-nuclear activists
+    were detained under suspicion. Vladimir Slivyak was one of the three arrested
+    under suspicion. He was an activist in the anti-nuclear movement and a Voronezh
+    action camp organizer. After the bombing Slivyak was pushed into a car by several
+    men who claimed to be Moscow police. The police interrogated and threatened Slivyak
+    for around ninety minutes before letting him go. The Moscow police thought environmentalists
+    from the anti-nuclear movement were associated with the bombing since an earlier
+    bombing occurred on August 31 at Manezh Palace in Moscow . After the incident,
+    on August 31, several more bombings occurred which agitated many people, leading
+    to the racially profiled arrest of dark-skinned Muscovites and visitors to the
+    Russian capital.
+  - One of the main sources of information about the Earth's composition comes from
+    understanding the relationship between peridotite and basalt melting. Peridotite
+    makes up most of Earth's mantle. Basalt, which is highly concentrated in the Earth's
+    oceanic crust, is formed when magma reaches the Earth's surface and cools down
+    at a very fast rate. When magma cools, different minerals crystallize at different
+    times depending on the cooling temperature of that respective mineral. This ultimately
+    changes the chemical composition of the melt as different minerals begin to crystallize.
+    Fractional crystallization of elements in basaltic liquids has also been studied
+    to observe the composition of lava in the upper mantle. This concept can be applied
+    by scientists to give insight on the evolution of Earth's mantle and how concentrations
+    of lithophile trace elements have varied over the last 3.5 billion years.
+- source_sentence: 'The group designs numerous structural concepts such as frameworks
+    and floors like Dalle O''Portune and D-Dalle.
+
+    The timber design office of excellence is an entity specializing in the design
+    and optimization of wood construction projects. It stands out for its ability
+    to meet the highest demands in terms of performance, durability and aesthetics,
+    and is thus recognized for its contribution to the realization of ambitious projects
+    in the field of timber construction.'
+  sentences:
+  - 'The group designs numerous structural concepts such as frameworks and floors
+    like Dalle O''Portune and D-Dalle.
+
+    The timber design office of excellence is an entity specializing in the design
+    and optimization of wood construction projects. It stands out for its ability
+    to meet the highest demands in terms of performance, durability and aesthetics,
+    and is thus recognized for its contribution to the realization of ambitious projects
+    in the field of timber construction.'
+  - 'In waterways, the term bridge strike may be used when a water vessel collides
+    with a bridge. This may include a collision to the bridge span or a collision
+    to the bridge support structure such as a pier. Bridge protection systems are
+    used to mitigate the effects of a ship strike.
+
+    In 2014, the United States Coast Guard published statistics that it investigated
+    205 bridge strikes in the eleven years prior to the publication. All of those
+    collisions involved involved a fixed, swing, lift or draw bridge. That number
+    was 1.2% of all vessel collision incidents investigated by the Coast Guard. The
+    primary causal factor was the lack of accurate air draft data, the distance between
+    water surface to the top most part of the vessel.'
+  - 'Post, Stephen Garrard. Encyclopedia of bioethics. Third edition. Macmillan Reference
+    USA, 2003. ISBN 0028657748. ISSN 0950-4125; DOI:10.1108/09504120510573477.  (5-Volume
+    Set; 3062 pages).
+
+    Reich, Warren Thomas Encyclopedia of Bioethics. First edition.  New York: Free
+    Press, 1978.  ISBN 0029261805.  ISBN 978-0029261804.  (4-Volume Set; 1933 pages)
+
+    Reich, Warren Thomas Encyclopedia of Bioethics. Second edition.  New York: Free
+    Press, 1982.  (5-Volume Set; 2950 pages)
+
+    Reich, Warren Thomas Encyclopedia of Bioethics. Third edition.  New York: Simon
+    & Schuster Macmillan, 1995; London: Simon and Schuster and Prentice Hall International,
+    c1995. Rev. ed. (5-Volume Set; 2950 pages; 464 articles) ISBN 0028973550. ISBN
+    978-0028973555.'
+- source_sentence: 'Regression is used to make predictions based on the retrieved
+    data through statistical trends and statistical modeling. Different uses of this
+    technique are used for fetching Photometric redshifts and measurements of physical
+    parameters of stars. The approaches are listed below:
+
+
+    Artificial neural network (ANN)
+
+    Support vector regression (SVR)
+
+    Decision tree
+
+    Random forest
+
+    k-nearest neighbors regression
+
+    Kernel regression
+
+    Principal component regression (PCR)
+
+    Gaussian process
+
+    Least squared regression (LSR)
+
+    Partial least squares regression'
+  sentences:
+  - 'Regression is used to make predictions based on the retrieved data through statistical
+    trends and statistical modeling. Different uses of this technique are used for
+    fetching Photometric redshifts and measurements of physical parameters of stars.
+    The approaches are listed below:
+
+
+    Artificial neural network (ANN)
+
+    Support vector regression (SVR)
+
+    Decision tree
+
+    Random forest
+
+    k-nearest neighbors regression
+
+    Kernel regression
+
+    Principal component regression (PCR)
+
+    Gaussian process
+
+    Least squared regression (LSR)
+
+    Partial least squares regression'
+  - 'Clandestine chemistry is not limited to drugs; it is also associated with explosives,
+    and other illegal chemicals. Of the explosives manufactured illegally, nitroglycerin
+    and acetone peroxide are easiest to produce due to the ease with which the precursors
+    can be acquired.
+
+    Uncle Fester is a writer who commonly writes about different aspects of clandestine
+    chemistry. Secrets of Methamphetamine Manufacture is among his most popular books,
+    and is considered required reading for DEA agents. More of his books deal with
+    other aspects of clandestine chemistry, including explosives, and poisons. Fester
+    is, however, considered by many to be a faulty and unreliable source for information
+    in regard to the clandestine manufacture of chemicals.'
+  - A novel input representation has been developed consisting of a combination of
+    sparse encoding, Blosum encoding, and input derived from hidden Markov models.
+    this method predicts T-cell epitopes for the genome of hepatitis C virus and discuss
+    possible applications of the prediction method to guide the process of rational
+    vaccine design.
+- source_sentence: 'Burray and The Barriers
+
+    Undiscovered Scotland: The Churchill Barriers
+
+    Our Past History: The Churchill Barriers Archived 17 December 2006 at the Wayback
+    Machine
+
+    Okneypics.com: photos of the barrier Archived 15 May 2008 at the Wayback Machine'
+  sentences:
+  - "For a neuron, in the limit of \n  \n    \n      \n        b\n        =\n    \
+    \    0\n      \n    \n    {\\displaystyle b=0}\n  \n, the map becomes 1D, since\
+    \ \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n converges\
+    \ to a constant. If the parameter \n  \n    \n      \n        b\n      \n    \n\
+    \    {\\displaystyle b}\n  \n is scanned in a range, different orbits will be\
+    \ seen, some periodic, others chaotic, that appear between two fixed points, one\
+    \ at \n  \n    \n      \n        x\n        =\n        1\n      \n    \n    {\\\
+    displaystyle x=1}\n  \n ; \n  \n    \n      \n        y\n        =\n        1\n\
+    \      \n    \n    {\\displaystyle y=1}\n  \n and the other close to the value\
+    \ of \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n\
+    \ (which would be the regime excitable).\n\n\n== References =="
+  - 'Cerebellar Purkinje neurons have been proposed to have two distinct bursting
+    modes: dendritically driven, by dendritic Ca2+ spikes, and somatically driven,
+    wherein the persistent Na+ current is the burst initiator and the SK K+ current
+    is the burst terminator. Purkinje neurons may utilise these bursting forms in
+    information coding to the deep cerebellar nuclei.'
+  - 'Burray and The Barriers
+
+    Undiscovered Scotland: The Churchill Barriers
+
+    Our Past History: The Churchill Barriers Archived 17 December 2006 at the Wayback
+    Machine
+
+    Okneypics.com: photos of the barrier Archived 15 May 2008 at the Wayback Machine'
 pipeline_tag: sentence-similarity
+library_name: sentence-transformers
 ---
 
+# SentenceTransformer based on zacbrld/MNLP_M2_document_encoder
+
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [zacbrld/MNLP_M2_document_encoder](https://huggingface.co/zacbrld/MNLP_M2_document_encoder). It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+
+## Model Details
+
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [zacbrld/MNLP_M2_document_encoder](https://huggingface.co/zacbrld/MNLP_M2_document_encoder) <!-- at revision 0256ba97b154a34e25bfdf236061c0fdb0c5d146 -->
+- **Maximum Sequence Length:** 256 tokens
+- **Output Dimensionality:** 384 dimensions
+- **Similarity Function:** Cosine Similarity
+<!-- - **Training Dataset:** Unknown -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
 
-# all-MiniLM-L6-v2
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
 
-## Usage (Sentence-Transformers)
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+### Full Model Architecture
 
 ```
-pip install -U sentence-transformers
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+  (2): Normalize()
+)
 ```
 
-Then you can use the model like this:
-```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
+## Usage
 
-model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
 
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+```bash
+pip install -U sentence-transformers
+```
 
+Then you can load this model and run inference.
 ```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-import torch.nn.functional as F
+from sentence_transformers import SentenceTransformer
 
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+# Download from the 🤗 Hub
+model = SentenceTransformer("zacbrld/MNLP_M2_document_encoder")
+# Run inference
+sentences = [
+    'Burray and The Barriers\nUndiscovered Scotland: The Churchill Barriers\nOur Past History: The Churchill Barriers Archived 17 December 2006 at the Wayback Machine\nOkneypics.com: photos of the barrier Archived 15 May 2008 at the Wayback Machine',
+    'Burray and The Barriers\nUndiscovered Scotland: The Churchill Barriers\nOur Past History: The Churchill Barriers Archived 17 December 2006 at the Wayback Machine\nOkneypics.com: photos of the barrier Archived 15 May 2008 at the Wayback Machine',
+    'Cerebellar Purkinje neurons have been proposed to have two distinct bursting modes: dendritically driven, by dendritic Ca2+ spikes, and somatically driven, wherein the persistent Na+ current is the burst initiator and the SK K+ current is the burst terminator. Purkinje neurons may utilise these bursting forms in information coding to the deep cerebellar nuclei.',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 384]
 
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
 
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### Unnamed Dataset
+
+* Size: 5,489 training samples
+* Columns: <code>sentence_0</code> and <code>sentence_1</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | sentence_0                                                                           | sentence_1                                                                           |
+  |:--------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
+  | type    | string                                                                               | string                                                                               |
+  | details | <ul><li>min: 34 tokens</li><li>mean: 144.23 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 34 tokens</li><li>mean: 144.23 tokens</li><li>max: 256 tokens</li></ul> |
+* Samples:
+  | sentence_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | sentence_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+  |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>In related work, Smoller, Temple, and Vogler propose that this shockwave may have resulted in our part of the universe having a lower density than that surrounding it, causing the accelerated expansion normally attributed to dark energy.  <br>They also propose that this related theory could be tested: a universe with dark energy should give a figure for the cubic correction to redshift versus luminosity C = −0.180 at a = a whereas for Smoller, Temple, and Vogler's alternative C should be positive rather than negative. They give a more precise calculation for their wave model alternative as: the cubic correction to redshift versus luminosity at a = a is C = 0.359.</code>                                                                                                                                                                      | <code>In related work, Smoller, Temple, and Vogler propose that this shockwave may have resulted in our part of the universe having a lower density than that surrounding it, causing the accelerated expansion normally attributed to dark energy.  <br>They also propose that this related theory could be tested: a universe with dark energy should give a figure for the cubic correction to redshift versus luminosity C = −0.180 at a = a whereas for Smoller, Temple, and Vogler's alternative C should be positive rather than negative. They give a more precise calculation for their wave model alternative as: the cubic correction to redshift versus luminosity at a = a is C = 0.359.</code>                                                                                                                                                                      |
+  | <code>Evolution is a central organizing concept in biology. It is the change in heritable characteristics of populations over successive generations. In artificial selection, animals were selectively bred for specific traits.<br> Given that traits are inherited, populations contain a varied mix of traits, and reproduction is able to increase any population, Darwin argued that in the natural world, it was nature that played the role of humans in selecting for specific traits. Darwin inferred that individuals who possessed heritable traits better adapted to their environments are more likely to survive and produce more offspring than other individuals. He further inferred that this would lead to the accumulation of favorable traits over successive generations, thereby increasing the match between the organisms and their environment.</code> | <code>Evolution is a central organizing concept in biology. It is the change in heritable characteristics of populations over successive generations. In artificial selection, animals were selectively bred for specific traits.<br> Given that traits are inherited, populations contain a varied mix of traits, and reproduction is able to increase any population, Darwin argued that in the natural world, it was nature that played the role of humans in selecting for specific traits. Darwin inferred that individuals who possessed heritable traits better adapted to their environments are more likely to survive and produce more offspring than other individuals. He further inferred that this would lead to the accumulation of favorable traits over successive generations, thereby increasing the match between the organisms and their environment.</code> |
+  | <code>The total number of engineers employed in the U.S. in 2015 was roughly 1.6 million. Of these, 278,340 were mechanical engineers (17.28%), the largest discipline by size. In 2012, the median annual income of mechanical engineers in the U.S. workforce was $80,580. The median income was highest when working for the government ($92,030), and lowest in education ($57,090). In 2014, the total number of mechanical engineering jobs was projected to grow 5% over the next decade. As of 2009, the average starting salary was $58,800 with a bachelor's degree.</code>                                                                                                                                                                                                                                                                                             | <code>The total number of engineers employed in the U.S. in 2015 was roughly 1.6 million. Of these, 278,340 were mechanical engineers (17.28%), the largest discipline by size. In 2012, the median annual income of mechanical engineers in the U.S. workforce was $80,580. The median income was highest when working for the government ($92,030), and lowest in education ($57,090). In 2014, the total number of mechanical engineering jobs was projected to grow 5% over the next decade. As of 2009, the average starting salary was $58,800 with a bachelor's degree.</code>                                                                                                                                                                                                                                                                                             |
+* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
+  ```json
+  {
+      "scale": 20.0,
+      "similarity_fct": "cos_sim"
+  }
+  ```
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
+- `num_train_epochs`: 5
+- `multi_dataset_batch_sampler`: round_robin
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: no
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 5e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1
+- `num_train_epochs`: 5
+- `max_steps`: -1
+- `lr_scheduler_type`: linear
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.0
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: False
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `tp_size`: 0
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: batch_sampler
+- `multi_dataset_batch_sampler`: round_robin
+
+</details>
+
+### Training Logs
+| Epoch  | Step | Training Loss |
+|:------:|:----:|:-------------:|
+| 1.4535 | 500  | 0.0002        |
+| 2.9070 | 1000 | 0.0           |
+| 4.3605 | 1500 | 0.0007        |
+
+
+### Framework Versions
+- Python: 3.10.11
+- Sentence Transformers: 3.4.1
+- Transformers: 4.51.3
+- PyTorch: 2.6.0
+- Accelerate: 1.7.0
+- Datasets: 3.6.0
+- Tokenizers: 0.21.1
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
 
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
-model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
+#### MultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
 
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+<!--
+## Glossary
 
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
+*Clearly define terms in order to be accessible across audiences.*
+-->
 
-# Perform pooling
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+<!--
+## Model Card Authors
 
-# Normalize embeddings
-sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
 
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
+<!--
+## Model Card Contact
 
-------
-
-## Background
-
-The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised 
-contrastive learning objective. We used the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model and fine-tuned in on a 
-1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset.
-
-We developed this model during the 
-[Community week using JAX/Flax for NLP & CV](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), 
-organized by Hugging Face. We developed this model as part of the project:
-[Train the Best Sentence Embedding Model Ever with 1B Training Pairs](https://discuss.huggingface.co/t/train-the-best-sentence-embedding-model-ever-with-1b-training-pairs/7354). We benefited from efficient hardware infrastructure to run the project: 7 TPUs v3-8, as well as intervention from Googles Flax, JAX, and Cloud team member about efficient deep learning frameworks.
-
-## Intended uses
-
-Our model is intended to be used as a sentence and short paragraph encoder. Given an input text, it outputs a vector which captures 
-the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
-
-By default, input text longer than 256 word pieces is truncated.
-
-
-## Training procedure
-
-### Pre-training 
-
-We use the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model. Please refer to the model card for more detailed information about the pre-training procedure.
-
-### Fine-tuning 
-
-We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
-We then apply the cross entropy loss by comparing with true pairs.
-
-#### Hyper parameters
-
-We trained our model on a TPU v3-8. We train the model during 100k steps using a batch size of 1024 (128 per TPU core).
-We use a learning rate warm up of 500. The sequence length was limited to 128 tokens. We used the AdamW optimizer with
-a 2e-5 learning rate. The full training script is accessible in this current repository: `train_script.py`.
-
-#### Training data
-
-We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
-We sampled each dataset given a weighted probability which configuration is detailed in the `data_config.json` file.
-
-
-| Dataset                                                  | Paper                                    | Number of training tuples  |
-|--------------------------------------------------------|:----------------------------------------:|:--------------------------:|
-| [Reddit comments (2015-2018)](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit) | [paper](https://arxiv.org/abs/1904.06472) | 726,484,430 |
-| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Abstracts) | [paper](https://aclanthology.org/2020.acl-main.447/) | 116,288,806 |
-| [WikiAnswers](https://github.com/afader/oqa#wikianswers-corpus) Duplicate question pairs | [paper](https://doi.org/10.1145/2623330.2623677) | 77,427,422 |
-| [PAQ](https://github.com/facebookresearch/PAQ) (Question, Answer) pairs | [paper](https://arxiv.org/abs/2102.07033) | 64,371,441 |
-| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Titles) | [paper](https://aclanthology.org/2020.acl-main.447/) | 52,603,982 |
-| [S2ORC](https://github.com/allenai/s2orc) (Title, Abstract) | [paper](https://aclanthology.org/2020.acl-main.447/) | 41,769,185 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Body) pairs  | - | 25,316,456 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title+Body, Answer) pairs  | - | 21,396,559 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Answer) pairs  | - | 21,396,559 |
-| [MS MARCO](https://microsoft.github.io/msmarco/) triplets | [paper](https://doi.org/10.1145/3404835.3462804) | 9,144,553 |
-| [GOOAQ: Open Question Answering with Diverse Answer Types](https://github.com/allenai/gooaq) | [paper](https://arxiv.org/pdf/2104.08727.pdf) | 3,012,496 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 1,198,260 |
-| [Code Search](https://huggingface.co/datasets/code_search_net) | - | 1,151,414 |
-| [COCO](https://cocodataset.org/#home) Image captions | [paper](https://link.springer.com/chapter/10.1007%2F978-3-319-10602-1_48) | 828,395|
-| [SPECTER](https://github.com/allenai/specter) citation triplets | [paper](https://doi.org/10.18653/v1/2020.acl-main.207) | 684,100 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Question, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 681,164 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Question) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 659,896 |
-| [SearchQA](https://huggingface.co/datasets/search_qa) | [paper](https://arxiv.org/abs/1704.05179) | 582,261 |
-| [Eli5](https://huggingface.co/datasets/eli5) | [paper](https://doi.org/10.18653/v1/p19-1346) | 325,475 |
-| [Flickr 30k](https://shannon.cs.illinois.edu/DenotationGraph/) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/229/33) | 317,695 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles) | | 304,525 |
-| AllNLI ([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) | [paper SNLI](https://doi.org/10.18653/v1/d15-1075), [paper MultiNLI](https://doi.org/10.18653/v1/n18-1101) | 277,230 | 
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (bodies) | | 250,519 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles+bodies) | | 250,460 |
-| [Sentence Compression](https://github.com/google-research-datasets/sentence-compression) | [paper](https://www.aclweb.org/anthology/D13-1155/) | 180,000 |
-| [Wikihow](https://github.com/pvl/wikihow_pairs_dataset) | [paper](https://arxiv.org/abs/1810.09305) | 128,542 |
-| [Altlex](https://github.com/chridey/altlex/) | [paper](https://aclanthology.org/P16-1135.pdf) | 112,696 |
-| [Quora Question Triplets](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs) | - | 103,663 |
-| [Simple Wikipedia](https://cs.pomona.edu/~dkauchak/simplification/) | [paper](https://www.aclweb.org/anthology/P11-2117/) | 102,225 |
-| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
-| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
-| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
-| **Total** | | **1,170,060,424** |
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1377e9af0ca0b016a9f2aa584d6fc71ab3ea6804fae21ef9fb1416e2944057ac
+oid sha256:15dd962c5f309aebaa2d1504dfa55c49721a879aa63eea2da2ca58cccaef535e
 size 90864192

tokenizer.json

@@ -2,14 +2,12 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 128,
+    "max_length": 256,
     "strategy": "LongestFirst",
     "stride": 0
   },
   "padding": {
-    "strategy": {
-      "Fixed": 128
-    },
+    "strategy": "BatchLongest",
     "direction": "Right",
     "pad_to_multiple_of": null,
     "pad_id": 0,