Hugging Face's logo

JohanHeinsen
/
PE_efterlyst_classifier_v2

Text Classification
setfit
Safetensors
sentence-transformers
Danish
bert
generated_from_setfit_trainer
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community
JohanHeinsen commited on
Commit
8b3d685
·
verified ·
1 Parent(s): f615b32

Push model using huggingface_hub.

Browse files
Files changed (13) hide show
  1. 1_Pooling/config.json +10 -0
  2. README.md +339 -0
  3. config.json +24 -0
  4. config_sentence_transformers.json +10 -0
  5. config_setfit.json +4 -0
  6. model.safetensors +3 -0
  7. model_head.pkl +3 -0
  8. modules.json +14 -0
  9. sentence_bert_config.json +4 -0
  10. special_tokens_map.json +37 -0
  11. tokenizer.json +0 -0
  12. tokenizer_config.json +65 -0
  13. vocab.txt +0 -0
1_Pooling/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false,
7
+ "pooling_mode_weightedmean_tokens": false,
8
+ "pooling_mode_lasttoken": false,
9
+ "include_prompt": true
10
+ }
README.md ADDED
@@ -0,0 +1,339 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - setfit
4
+ - sentence-transformers
5
+ - text-classification
6
+ - generated_from_setfit_trainer
7
+ widget:
8
+ - text: Niels Rasmussen, kaldet Niels Slagter, alm. Højde, svær bygget, mørkt Haar,
9
+ iført mørk Stortrøie, Benklæder og Kaskjet, sigtes forTyveri. Anholdes hertil.
10
+ (St. 3, 688.)
11
+ - text: Nicolai Gerhard Ahrenzen (Kbhvn.), 49 Aar.
12
+ - text: Peter Mortensen (Kbhvn.), 31 Aar. Død. P. [2733].
13
+ - text: Claus Tollefsen, 40 Aar.
14
+ - text: 2) Et Fruentimmer, ca. 40 Aar gl., middelaf Højde og Bygning med mørkt Haar,
15
+ iført mørk Kjole graaligt Shavl og over Hovedet et hvidt uldent Tørklæde med en
16
+ rød Kant, og medførte en brun Hankekurv, sigtes for Bedrageri. (St. 4, 177.)
17
+ metrics:
18
+ - accuracy
19
+ - f1
20
+ - precision
21
+ - recall
22
+ pipeline_tag: text-classification
23
+ library_name: setfit
24
+ inference: true
25
+ base_model: JohanHeinsen/Old_News_Segmentation_SBERT_V0.1
26
+ model-index:
27
+ - name: SetFit with JohanHeinsen/Old_News_Segmentation_SBERT_V0.1
28
+ results:
29
+ - task:
30
+ type: text-classification
31
+ name: Text Classification
32
+ dataset:
33
+ name: Unknown
34
+ type: unknown
35
+ split: test
36
+ metrics:
37
+ - type: accuracy
38
+ value: 0.9816666666666667
39
+ name: Accuracy
40
+ - type: f1
41
+ value: 0.9385474860335196
42
+ name: F1
43
+ - type: precision
44
+ value: 0.9230769230769231
45
+ name: Precision
46
+ - type: recall
47
+ value: 0.9545454545454546
48
+ name: Recall
49
+ ---
50
+
51
+ # SetFit with JohanHeinsen/Old_News_Segmentation_SBERT_V0.1
52
+
53
+ This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [JohanHeinsen/Old_News_Segmentation_SBERT_V0.1](https://huggingface.co/JohanHeinsen/Old_News_Segmentation_SBERT_V0.1) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
54
+
55
+ The model has been trained using an efficient few-shot learning technique that involves:
56
+
57
+ 1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
58
+ 2. Training a classification head with features from the fine-tuned Sentence Transformer.
59
+
60
+ ## Model Details
61
+
62
+ ### Model Description
63
+ - **Model Type:** SetFit
64
+ - **Sentence Transformer body:** [JohanHeinsen/Old_News_Segmentation_SBERT_V0.1](https://huggingface.co/JohanHeinsen/Old_News_Segmentation_SBERT_V0.1)
65
+ - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
66
+ - **Maximum Sequence Length:** 512 tokens
67
+ - **Number of Classes:** 2 classes
68
+ <!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
69
+ <!-- - **Language:** Unknown -->
70
+ <!-- - **License:** Unknown -->
71
+
72
+ ### Model Sources
73
+
74
+ - **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
75
+ - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
76
+ - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
77
+
78
+ ### Model Labels
79
+ | Label | Examples |
80
+ |:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
81
+ | 1 | <ul><li>'2) Fattiglem, fhv. Roersbetjent, Jens Hansen, af Fare, har den 15de ds. forladt Fattiggaarden. der og formodes at drive arbejdsløs omkring muligvis er han taget til Kjøbenhavn for at søge Hyre som Sømand. Saafremt han, der er 50 Aar gl., middel af Væxt, og var iført sort rundpullet Hat, blaa Trøje og Benklæder muligvis dog et Par Lærreds ovenpaa – samt Træsko, maatte antræffes, bedes Underretning derom meddelt Byog Herredsfogden i Storehedinge.'</li><li>'1) En Mandsperson, ca. 20 Aar gl., middelstor eller lidt mindre, lyst Polkahaar, ordentlig klædt med mørk Sækfrakke og lyse Buxer, – sigtes for Tyveriet Nr. 3. (St. 2, 291.)'</li><li>'2) Oplysning om, hvor Garversvend Niels Peter Schmidt eller Niels Peter Nielsen Schmidt, født 16de Febr. 1839, maatte opholde sig, bedes meddeelt Muckadell m. fl. Birkers Kontor i Spanget pr. Kværndrup. Paagjældende blev blev den 28de Januar d. A. viseret derfra til Odense, hvorfra han strax igien skal være afgaaet til Fredericia.'</li></ul> |
82
+ | 0 | <ul><li>'2) 2 Høns og en Hane, denne sidste graa med laadne Ben, den ene Høne brunspættet, den anden sort med laadne Ben, ere bortkomne siden den 24. f.M. (St. 7, 448).'</li><li>'Hans Edvard Valdemar Holst (Kbhvn.), 45 Aar. Løsgængeri.'</li><li>'Peter Christian Leyring (Levring), 26 Aar. Betleri.'</li></ul> |
83
+
84
+ ## Evaluation
85
+
86
+ ### Metrics
87
+ | Label | Accuracy | F1 | Precision | Recall |
88
+ |:--------|:---------|:-------|:----------|:-------|
89
+ | **all** | 0.9817 | 0.9385 | 0.9231 | 0.9545 |
90
+
91
+ ## Uses
92
+
93
+ ### Direct Use for Inference
94
+
95
+ First install the SetFit library:
96
+
97
+ ```bash
98
+ pip install setfit
99
+ ```
100
+
101
+ Then you can load this model and run inference.
102
+
103
+ ```python
104
+ from setfit import SetFitModel
105
+
106
+ # Download from the 🤗 Hub
107
+ model = SetFitModel.from_pretrained("setfit_model_id")
108
+ # Run inference
109
+ preds = model("Claus Tollefsen, 40 Aar.")
110
+ ```
111
+
112
+ <!--
113
+ ### Downstream Use
114
+
115
+ *List how someone could finetune this model on their own dataset.*
116
+ -->
117
+
118
+ <!--
119
+ ### Out-of-Scope Use
120
+
121
+ *List how the model may foreseeably be misused and address what users ought not to do with the model.*
122
+ -->
123
+
124
+ <!--
125
+ ## Bias, Risks and Limitations
126
+
127
+ *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
128
+ -->
129
+
130
+ <!--
131
+ ### Recommendations
132
+
133
+ *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
134
+ -->
135
+
136
+ ## Training Details
137
+
138
+ ### Training Set Metrics
139
+ | Training set | Min | Median | Max |
140
+ |:-------------|:----|:--------|:----|
141
+ | Word count | 3 | 20.8907 | 245 |
142
+
143
+ | Label | Training Sample Count |
144
+ |:------|:----------------------|
145
+ | 0 | 1195 |
146
+ | 1 | 205 |
147
+
148
+ ### Training Hyperparameters
149
+ - batch_size: (16, 16)
150
+ - num_epochs: (3, 3)
151
+ - max_steps: -1
152
+ - sampling_strategy: oversampling
153
+ - num_iterations: 12
154
+ - body_learning_rate: (2e-05, 2e-05)
155
+ - head_learning_rate: 2e-05
156
+ - loss: CosineSimilarityLoss
157
+ - distance_metric: cosine_distance
158
+ - margin: 0.25
159
+ - end_to_end: False
160
+ - use_amp: False
161
+ - warmup_proportion: 0.1
162
+ - l2_weight: 0.01
163
+ - seed: 42
164
+ - eval_max_steps: -1
165
+ - load_best_model_at_end: False
166
+
167
+ ### Training Results
168
+ | Epoch | Step | Training Loss | Validation Loss |
169
+ |:------:|:----:|:-------------:|:---------------:|
170
+ | 0.0005 | 1 | 0.1797 | - |
171
+ | 0.0238 | 50 | 0.2091 | - |
172
+ | 0.0476 | 100 | 0.1061 | - |
173
+ | 0.0714 | 150 | 0.0529 | - |
174
+ | 0.0952 | 200 | 0.0491 | - |
175
+ | 0.1190 | 250 | 0.0238 | - |
176
+ | 0.1429 | 300 | 0.0195 | - |
177
+ | 0.1667 | 350 | 0.013 | - |
178
+ | 0.1905 | 400 | 0.0066 | - |
179
+ | 0.2143 | 450 | 0.005 | - |
180
+ | 0.2381 | 500 | 0.0038 | - |
181
+ | 0.2619 | 550 | 0.0038 | - |
182
+ | 0.2857 | 600 | 0.005 | - |
183
+ | 0.3095 | 650 | 0.0062 | - |
184
+ | 0.3333 | 700 | 0.0024 | - |
185
+ | 0.3571 | 750 | 0.0002 | - |
186
+ | 0.3810 | 800 | 0.0003 | - |
187
+ | 0.4048 | 850 | 0.0008 | - |
188
+ | 0.4286 | 900 | 0.0001 | - |
189
+ | 0.4524 | 950 | 0.0006 | - |
190
+ | 0.4762 | 1000 | 0.0022 | - |
191
+ | 0.5 | 1050 | 0.0003 | - |
192
+ | 0.5238 | 1100 | 0.0016 | - |
193
+ | 0.5476 | 1150 | 0.0001 | - |
194
+ | 0.5714 | 1200 | 0.0 | - |
195
+ | 0.5952 | 1250 | 0.0 | - |
196
+ | 0.6190 | 1300 | 0.0 | - |
197
+ | 0.6429 | 1350 | 0.0 | - |
198
+ | 0.6667 | 1400 | 0.0 | - |
199
+ | 0.6905 | 1450 | 0.0 | - |
200
+ | 0.7143 | 1500 | 0.0 | - |
201
+ | 0.7381 | 1550 | 0.0024 | - |
202
+ | 0.7619 | 1600 | 0.0002 | - |
203
+ | 0.7857 | 1650 | 0.0001 | - |
204
+ | 0.8095 | 1700 | 0.0 | - |
205
+ | 0.8333 | 1750 | 0.0 | - |
206
+ | 0.8571 | 1800 | 0.0 | - |
207
+ | 0.8810 | 1850 | 0.0 | - |
208
+ | 0.9048 | 1900 | 0.0 | - |
209
+ | 0.9286 | 1950 | 0.0 | - |
210
+ | 0.9524 | 2000 | 0.0 | - |
211
+ | 0.9762 | 2050 | 0.0 | - |
212
+ | 1.0 | 2100 | 0.0 | - |
213
+ | 1.0238 | 2150 | 0.0 | - |
214
+ | 1.0476 | 2200 | 0.0 | - |
215
+ | 1.0714 | 2250 | 0.0 | - |
216
+ | 1.0952 | 2300 | 0.0 | - |
217
+ | 1.1190 | 2350 | 0.0 | - |
218
+ | 1.1429 | 2400 | 0.0 | - |
219
+ | 1.1667 | 2450 | 0.0 | - |
220
+ | 1.1905 | 2500 | 0.0 | - |
221
+ | 1.2143 | 2550 | 0.0 | - |
222
+ | 1.2381 | 2600 | 0.0 | - |
223
+ | 1.2619 | 2650 | 0.0 | - |
224
+ | 1.2857 | 2700 | 0.0 | - |
225
+ | 1.3095 | 2750 | 0.0 | - |
226
+ | 1.3333 | 2800 | 0.0 | - |
227
+ | 1.3571 | 2850 | 0.0 | - |
228
+ | 1.3810 | 2900 | 0.0 | - |
229
+ | 1.4048 | 2950 | 0.0 | - |
230
+ | 1.4286 | 3000 | 0.0 | - |
231
+ | 1.4524 | 3050 | 0.0 | - |
232
+ | 1.4762 | 3100 | 0.0 | - |
233
+ | 1.5 | 3150 | 0.0 | - |
234
+ | 1.5238 | 3200 | 0.0 | - |
235
+ | 1.5476 | 3250 | 0.0 | - |
236
+ | 1.5714 | 3300 | 0.0 | - |
237
+ | 1.5952 | 3350 | 0.0 | - |
238
+ | 1.6190 | 3400 | 0.0 | - |
239
+ | 1.6429 | 3450 | 0.0 | - |
240
+ | 1.6667 | 3500 | 0.0 | - |
241
+ | 1.6905 | 3550 | 0.0 | - |
242
+ | 1.7143 | 3600 | 0.0 | - |
243
+ | 1.7381 | 3650 | 0.0 | - |
244
+ | 1.7619 | 3700 | 0.0 | - |
245
+ | 1.7857 | 3750 | 0.0 | - |
246
+ | 1.8095 | 3800 | 0.0 | - |
247
+ | 1.8333 | 3850 | 0.0 | - |
248
+ | 1.8571 | 3900 | 0.0 | - |
249
+ | 1.8810 | 3950 | 0.0 | - |
250
+ | 1.9048 | 4000 | 0.0 | - |
251
+ | 1.9286 | 4050 | 0.0 | - |
252
+ | 1.9524 | 4100 | 0.0 | - |
253
+ | 1.9762 | 4150 | 0.0 | - |
254
+ | 2.0 | 4200 | 0.0 | - |
255
+ | 2.0238 | 4250 | 0.0 | - |
256
+ | 2.0476 | 4300 | 0.0 | - |
257
+ | 2.0714 | 4350 | 0.0 | - |
258
+ | 2.0952 | 4400 | 0.0 | - |
259
+ | 2.1190 | 4450 | 0.0 | - |
260
+ | 2.1429 | 4500 | 0.0 | - |
261
+ | 2.1667 | 4550 | 0.0 | - |
262
+ | 2.1905 | 4600 | 0.0 | - |
263
+ | 2.2143 | 4650 | 0.0 | - |
264
+ | 2.2381 | 4700 | 0.0 | - |
265
+ | 2.2619 | 4750 | 0.0 | - |
266
+ | 2.2857 | 4800 | 0.0 | - |
267
+ | 2.3095 | 4850 | 0.0 | - |
268
+ | 2.3333 | 4900 | 0.0 | - |
269
+ | 2.3571 | 4950 | 0.0 | - |
270
+ | 2.3810 | 5000 | 0.0 | - |
271
+ | 2.4048 | 5050 | 0.0 | - |
272
+ | 2.4286 | 5100 | 0.0 | - |
273
+ | 2.4524 | 5150 | 0.0 | - |
274
+ | 2.4762 | 5200 | 0.0 | - |
275
+ | 2.5 | 5250 | 0.0 | - |
276
+ | 2.5238 | 5300 | 0.0 | - |
277
+ | 2.5476 | 5350 | 0.0 | - |
278
+ | 2.5714 | 5400 | 0.0 | - |
279
+ | 2.5952 | 5450 | 0.0 | - |
280
+ | 2.6190 | 5500 | 0.0 | - |
281
+ | 2.6429 | 5550 | 0.0 | - |
282
+ | 2.6667 | 5600 | 0.0 | - |
283
+ | 2.6905 | 5650 | 0.0 | - |
284
+ | 2.7143 | 5700 | 0.0 | - |
285
+ | 2.7381 | 5750 | 0.0 | - |
286
+ | 2.7619 | 5800 | 0.0 | - |
287
+ | 2.7857 | 5850 | 0.0 | - |
288
+ | 2.8095 | 5900 | 0.0 | - |
289
+ | 2.8333 | 5950 | 0.0 | - |
290
+ | 2.8571 | 6000 | 0.0 | - |
291
+ | 2.8810 | 6050 | 0.0 | - |
292
+ | 2.9048 | 6100 | 0.0 | - |
293
+ | 2.9286 | 6150 | 0.0 | - |
294
+ | 2.9524 | 6200 | 0.0 | - |
295
+ | 2.9762 | 6250 | 0.0 | - |
296
+ | 3.0 | 6300 | 0.0 | - |
297
+
298
+ ### Framework Versions
299
+ - Python: 3.11.12
300
+ - SetFit: 1.1.3
301
+ - Sentence Transformers: 4.1.0
302
+ - Transformers: 4.51.3
303
+ - PyTorch: 2.7.0
304
+ - Datasets: 2.19.2
305
+ - Tokenizers: 0.21.1
306
+
307
+ ## Citation
308
+
309
+ ### BibTeX
310
+ ```bibtex
311
+ @article{https://doi.org/10.48550/arxiv.2209.11055,
312
+ doi = {10.48550/ARXIV.2209.11055},
313
+ url = {https://arxiv.org/abs/2209.11055},
314
+ author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
315
+ keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
316
+ title = {Efficient Few-Shot Learning Without Prompts},
317
+ publisher = {arXiv},
318
+ year = {2022},
319
+ copyright = {Creative Commons Attribution 4.0 International}
320
+ }
321
+ ```
322
+
323
+ <!--
324
+ ## Glossary
325
+
326
+ *Clearly define terms in order to be accessible across audiences.*
327
+ -->
328
+
329
+ <!--
330
+ ## Model Card Authors
331
+
332
+ *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
333
+ -->
334
+
335
+ <!--
336
+ ## Model Card Contact
337
+
338
+ *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
339
+ -->
config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BertModel"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "classifier_dropout": null,
7
+ "hidden_act": "gelu",
8
+ "hidden_dropout_prob": 0.1,
9
+ "hidden_size": 768,
10
+ "initializer_range": 0.02,
11
+ "intermediate_size": 3072,
12
+ "layer_norm_eps": 1e-12,
13
+ "max_position_embeddings": 512,
14
+ "model_type": "bert",
15
+ "num_attention_heads": 12,
16
+ "num_hidden_layers": 12,
17
+ "pad_token_id": 0,
18
+ "position_embedding_type": "absolute",
19
+ "torch_dtype": "float32",
20
+ "transformers_version": "4.51.3",
21
+ "type_vocab_size": 2,
22
+ "use_cache": true,
23
+ "vocab_size": 30522
24
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "4.1.0",
4
+ "transformers": "4.51.3",
5
+ "pytorch": "2.7.0"
6
+ },
7
+ "prompts": {},
8
+ "default_prompt_name": null,
9
+ "similarity_fn_name": "cosine"
10
+ }
config_setfit.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "normalize_embeddings": false,
3
+ "labels": null
4
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:473053d9645648c976b810972f8a771079a7ac26030efbeb65150db7e85c52b6
3
+ size 437951328
model_head.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe19928e8d852ef2e16ba57dae9f2d0935a34c2bcd9682326e9801d2891cc01c
3
+ size 7007
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ }
14
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 512,
3
+ "do_lower_case": false
4
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": {
3
+ "content": "[CLS]",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "mask_token": {
10
+ "content": "[MASK]",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "[PAD]",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "sep_token": {
24
+ "content": "[SEP]",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "unk_token": {
31
+ "content": "[UNK]",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ }
37
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "clean_up_tokenization_spaces": true,
45
+ "cls_token": "[CLS]",
46
+ "do_basic_tokenize": true,
47
+ "do_lower_case": true,
48
+ "extra_special_tokens": {},
49
+ "mask_token": "[MASK]",
50
+ "max_length": 512,
51
+ "model_max_length": 512,
52
+ "never_split": null,
53
+ "pad_to_multiple_of": null,
54
+ "pad_token": "[PAD]",
55
+ "pad_token_type_id": 0,
56
+ "padding_side": "right",
57
+ "sep_token": "[SEP]",
58
+ "stride": 0,
59
+ "strip_accents": null,
60
+ "tokenize_chinese_chars": true,
61
+ "tokenizer_class": "BertTokenizer",
62
+ "truncation_side": "right",
63
+ "truncation_strategy": "longest_first",
64
+ "unk_token": "[UNK]"
65
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff