Upload folder using huggingface_hub

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 384,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -0,0 +1,129 @@
+---
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
+
+---
+
+# {MODEL_NAME}
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+
+<!--- Describe your model here -->
+
+## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
+
+model = SentenceTransformer('{MODEL_NAME}')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+
+
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
+model = AutoModel.from_pretrained('{MODEL_NAME}')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+
+
+## Evaluation Results
+
+<!--- Describe how your model was evaluated -->
+
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
+
+
+## Training
+The model was trained with the parameters:
+
+**DataLoader**:
+
+`torch.utils.data.dataloader.DataLoader` of length 14004 with parameters:
+```
+{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
+```
+
+**Loss**:
+
+`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters:
+  ```
+  {'scale': 20.0, 'similarity_fct': 'cos_sim'}
+  ```
+
+Parameters of the fit()-Method:
+```
+{
+    "epochs": 10,
+    "evaluation_steps": 3500,
+    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
+    "max_grad_norm": 1,
+    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
+    "optimizer_params": {
+        "lr": 2e-05
+    },
+    "scheduler": "WarmupLinear",
+    "steps_per_epoch": null,
+    "warmup_steps": 1000,
+    "weight_decay": 0.01
+}
+```
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
+
+## Citing & Authors
+
+<!--- Describe where people can find more information -->

added_tokens.json

@@ -0,0 +1,7 @@
+{
+  "[CLS]": 101,
+  "[MASK]": 103,
+  "[PAD]": 0,
+  "[SEP]": 102,
+  "[UNK]": 100
+}

config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "thenlper/gte-small",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.34.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.34.0",
+    "pytorch": "2.0.1+cu117"
+  }
+}

eval/similarity_evaluation_results.csv

@@ -0,0 +1,51 @@
+epoch,steps,cosine_pearson,cosine_spearman,euclidean_pearson,euclidean_spearman,manhattan_pearson,manhattan_spearman,dot_pearson,dot_spearman
+0,3500,0.6730092624893155,0.6668499585808944,0.6971085137790577,0.6957555585872145,0.6964708758844027,0.6949611538604863,0.575295270658501,0.5600086808363787
+0,7000,0.6995284147945056,0.690129117793024,0.7112639045145328,0.7089714994518217,0.7107341583480069,0.7083206837930602,0.6372396594847384,0.621768650283809
+0,10500,0.6904662809504486,0.6795175283070729,0.7045464638903796,0.7010871754366438,0.7044654081800049,0.7007802136217246,0.6196072347291813,0.6021967610494757
+0,14000,0.6774591357523654,0.6701730554950369,0.7010484199523852,0.6978742278594435,0.7003135930280203,0.6968834115073086,0.5932925795437046,0.5772401548651316
+0,-1,0.6778433771673221,0.6707264240910963,0.7012328123732234,0.6979917760299491,0.7005316588285098,0.6970366148024505,0.5943533801665262,0.578436282015549
+1,3500,0.6813314229364082,0.6733664659978372,0.7033587010546669,0.6998302489565303,0.702641199344894,0.6989392617381621,0.600467716797782,0.5831382016289339
+1,7000,0.6975437016388568,0.6860593699995761,0.7080119640091739,0.7041181022107184,0.7071201707984621,0.7032203352762582,0.6402414557210805,0.6224356795269345
+1,10500,0.689774463802734,0.6814888191505427,0.7073881412610419,0.7043045436032216,0.7064761468253781,0.7033286197443842,0.6188964696760643,0.6017746241487872
+1,14000,0.6980207447745232,0.6879275413541941,0.7123978323386528,0.7075029947827628,0.7116478643924118,0.7066588698713117,0.6322173423624808,0.6144531746486682
+1,-1,0.6982072880631237,0.6880697443268611,0.7120782269387158,0.7071862737290933,0.7113393150309902,0.7063212863321947,0.6332867615227291,0.6155048536056114
+2,3500,0.6920793903566108,0.6824844532737326,0.7082650382549175,0.7062236447759548,0.7074618390349171,0.7051200283281289,0.6175831892275353,0.6006102089937847
+2,7000,0.6837806109485104,0.6753417484620858,0.7035354817098134,0.7000663415256339,0.7028149943477154,0.699199394830113,0.6097127014683072,0.592638417777474
+2,10500,0.6916124464950051,0.6820931499594793,0.7087610205764057,0.7047267162694411,0.7080375720360258,0.7040202090838541,0.617243752455075,0.599266323967178
+2,14000,0.6884301651734351,0.6800899201696333,0.7082912093934074,0.7053006798330103,0.7078433696312144,0.7048477402066285,0.6087549159433665,0.5917613249432511
+2,-1,0.6873254597214417,0.6790575778920381,0.7077234262777861,0.7047199406528399,0.707264112551481,0.7042812828039687,0.6072699207089136,0.5901430462897597
+3,3500,0.6817822753194608,0.6745486036123124,0.7010883375622925,0.6974372405494529,0.7008499088854967,0.6971094795189183,0.609388555685221,0.5927591408204803
+3,7000,0.6788122811885764,0.6711735427987083,0.7002547582909425,0.6964091510854682,0.7000236147730783,0.6961031909781199,0.6041713169630925,0.5870405176870384
+3,10500,0.6937340839033828,0.6829066161921535,0.7057770698696868,0.7007794054187914,0.7054261571410938,0.7002595842998083,0.6336027485509074,0.6152097255434368
+3,14000,0.6934060186751118,0.6827827574090468,0.7071975242398304,0.7025528425640724,0.7068703943938127,0.7021965318561851,0.6333678552385257,0.6155843656416959
+3,-1,0.6921608651943452,0.6816531574357049,0.7062827055451419,0.7016981283738991,0.7059559893018446,0.7013300844455428,0.6313915245487013,0.6136104977427939
+4,3500,0.6924992040160505,0.6827676711406654,0.7054178834571802,0.7011270947353269,0.705156168835884,0.7008111683255538,0.6360370380963472,0.618937832129759
+4,7000,0.6844885416809584,0.6758659560491561,0.7004709400817833,0.6957671191457862,0.6999165277483188,0.6950716958094039,0.6221265738454189,0.6050579497316222
+4,10500,0.6812403945466559,0.6724105024634046,0.6997362828626322,0.6952944667306545,0.6993505876473851,0.6947948743542001,0.6141110846912989,0.5962116779265484
+4,14000,0.6805193591785141,0.6720452269108772,0.7009978183847455,0.6965924014270363,0.7006856192052998,0.6961825221401435,0.6077757454563119,0.5897191644798084
+4,-1,0.6805203864398639,0.6719732233509271,0.7009868771441635,0.6965656374074152,0.7006742301382993,0.696171661893959,0.607746187894964,0.589654733614189
+5,3500,0.6861819260699271,0.6782123324734793,0.7047774214326677,0.6999512919404914,0.7047969465637124,0.6998794615441892,0.6187621257096295,0.6020460526350765
+5,7000,0.6933172700161877,0.6828286647951983,0.7076428315172866,0.7034049563716868,0.7076006866472361,0.703268374250089,0.6313605022623708,0.6137764944824465
+5,10500,0.6924437774302714,0.6815407961652115,0.7069234768924141,0.7021670102470472,0.7068856765059967,0.702001675019503,0.6295520477132137,0.6111455360813923
+5,14000,0.6835579525719606,0.6754769949148758,0.7029387668740507,0.698057376425828,0.7029260217872955,0.6978554949313728,0.6161143137880803,0.5985291577742312
+5,-1,0.6839224094007008,0.6758103807935645,0.7032048528247542,0.6983231966601572,0.7032030183712371,0.6981337549097459,0.6165988604647651,0.5990175170835027
+6,3500,0.674191612805674,0.667649372627317,0.6979683855745982,0.6931146997283386,0.6980976036357308,0.6929797499183566,0.6006315882464588,0.5835302207215785
+6,7000,0.6836449171112225,0.6753370347073058,0.7041110784610332,0.6989978572170228,0.7042886515780646,0.6989696628375339,0.6131566170068764,0.5951258656335489
+6,10500,0.6913716315647203,0.6818886451482009,0.70789931815628,0.7026947200042438,0.7079469899959152,0.7025525803518604,0.6277300026378606,0.6102787290788202
+6,14000,0.6856796471044863,0.6772840614126129,0.7042975470400412,0.6997673559920006,0.7044552228205248,0.6997049190187283,0.6171233977955952,0.6002214168488542
+6,-1,0.6856871830445403,0.6772940840288869,0.7043107869939068,0.6997828812567386,0.704467521726275,0.6997261029806117,0.6170691034167138,0.6001475656458563
+7,3500,0.6877878455202227,0.6792284327110132,0.7041732994359371,0.6999706806675877,0.7041203349939745,0.6997205078322757,0.6250117542994582,0.6085324926887467
+7,7000,0.6890670676384872,0.6803879378661597,0.7074715237406517,0.7032170502890037,0.707387521356422,0.7030363844616537,0.6209501014216086,0.6044114240664809
+7,10500,0.6888352958783365,0.6795861899258554,0.7059774637276494,0.7015112851619523,0.7059768198606408,0.7013308723727094,0.6228860910791078,0.6056109562684298
+7,14000,0.6963882588497655,0.6858387945045503,0.7105601184368315,0.7054134260879471,0.710658457325891,0.7053404659206204,0.6341042495295215,0.6162912683369585
+7,-1,0.6963515806079065,0.6857956549600575,0.7105049943746995,0.7053511837926005,0.7106006692882587,0.7052901725060451,0.6341132354602108,0.6163028632868622
+8,3500,0.692311525847834,0.6831067699883744,0.7087884794754102,0.704302888213538,0.7090604200392525,0.7043041425284432,0.6276238228565325,0.6105242630948543
+8,7000,0.6927400334255509,0.6832537986666756,0.7090929710426109,0.7040847012601575,0.7092195986572658,0.7039567726540537,0.6281549161646389,0.6108189730888697
+8,10500,0.6937724045788671,0.6841511735311949,0.7100049250859602,0.7053763953176093,0.7101502887563109,0.7052963055556313,0.6280249660527714,0.6104293898534154
+8,14000,0.6914439026090323,0.682479977494476,0.708337673598075,0.7040431124970559,0.7085180916132201,0.7039781096718114,0.6244446546473045,0.6073872586830461
+8,-1,0.6913090791857512,0.6823883317688529,0.7083041634523823,0.7040302813267318,0.7084899161160984,0.7039653549464728,0.6241322458814149,0.6071117110195822
+9,3500,0.6906126536299643,0.6813455683164258,0.7073673622445957,0.7028839555430166,0.707564124808887,0.7028734082289526,0.6245430165725735,0.607435841425523
+9,7000,0.6895615450315126,0.6805072740544088,0.7073752516031252,0.7030720310356013,0.7075847971567769,0.7030312114291314,0.621105610444555,0.6037490621417091
+9,10500,0.6914819323124138,0.6822682393661237,0.7086670998849309,0.7043429744999562,0.7088495186220839,0.7042825149671318,0.6246065621314404,0.6072620304525425
+9,14000,0.6901903692678109,0.6810670940047023,0.7077128748437392,0.7034413539825357,0.7079028476801082,0.7033820671857173,0.6225515708800942,0.6052581907042425
+9,-1,0.6901903704645067,0.6810671167502476,0.7077128718064,0.7034413926320477,0.7079028453888914,0.7033819823271376,0.6225515702448341,0.6052581710602516

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea88c15218cc1e0e5af23aadf9209bae5064a8f372c19e0b6e34910ff65809df
+size 133506729

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,65 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}