Update README.md

README.md

@@ -1,13 +1,13 @@
 ---
 language:
 - 'no'
+- nn
+- nb
 tags:
 - sentence-transformers
 - sentence-similarity
 - feature-extraction
 - dense
-- generated_from_trainer
-- dataset_size:556367
 - loss:CachedMultipleNegativesRankingLoss
 base_model:
 - ltg/norbert4-base
@@ -49,7 +49,7 @@ library_name: sentence-transformers
 metrics:
 - cosine_accuracy
 model-index:
-- name: SentenceTransformer based on Murhaf/ltg-norbert4-base_ndla
+- name: SentenceTransformer based on NorBERT4-base
   results:
   - task:
       type: triplet
@@ -64,32 +64,15 @@ model-index:
 license: apache-2.0
 ---
 
-# SentenceTransformer based on Murhaf/ltg-norbert4-base_ndla
+# SentenceTransformer based on NorBERT4-base
 
-This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Murhaf/ltg-norbert4-base_ndla](https://huggingface.co/Murhaf/ltg-norbert4-base_ndla) on the [all-nli-norwegian](https://huggingface.co/datasets/Murhaf/all-nli-norwegian) dataset. It maps sentences & paragraphs to a 640-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+NorSBERT4-base is a [Sentence Transformer](https://www.SBERT.net) model finetuned from [ltg/norbert4-base](https://huggingface.co/ltg/norbert4-base). 
+The model maps sentences (and paragraphs) to a 960-dimensional dense vector space and can be used for semantic textual similarity, semantic search, 
+text classification, clustering, among other tasks.
 
-## Model Details
 
-### Model Description
-- **Model Type:** Sentence Transformer
-- **Base model:** [ltg/norbert4-base](https://huggingface.co/ltg/norbert4-base) <!-- at revision 762fb095e1c571e52d8690bf07ec8b65d3551026 -->
-- **Maximum Sequence Length:** 75 tokens
-- **Output Dimensionality:** 640 dimensions
-- **Similarity Function:** Cosine Similarity
-- **Training Dataset:**
-    - [all-nli-norwegian](https://huggingface.co/datasets/Fremtind/all-nli-norwegian)
-- **Language:** no
-<!-- - **License:** Unknown -->
-
-
-### Full Model Architecture
-
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 75, 'do_lower_case': False, 'architecture': 'GptBertModel'})
-  (1): Pooling({'word_embedding_dimension': 640, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-)
-```
+Note: While the fine-tuned sentence-transformer model has a `max_seq_length` of 75 tokens, the base model does not. 
+This means that the sequence length can be increased to 16384 (which is the max length in the base model).
 
 ## Usage
 
@@ -101,136 +84,62 @@ First install the Sentence Transformers library:
 pip install -U sentence-transformers
 ```
 
-Then you can load this model and run inference.
+Then you can load this model and run inference. Note that you should load the model with `trust_remote_code=True` because it needs a custom wrapper (see the [base model](https://huggingface.co/ltg/norbert4-base/blob/main/modeling_gptbert.py) for more details).
 ```python
 from sentence_transformers import SentenceTransformer
 
 # Download from the 🤗 Hub
-model = SentenceTransformer("Murhaf/ltg-norbert4-base_ndla-all-nli")
+model = SentenceTransformer("Fremtind/norsbert4-base", trust_remote_code=True)
 # Run inference
 sentences = [
-    'En mann lager et sandmaleri på gulvet.',
-    'En mann lager kunst.',
-    'En kvinne ødelegger et sandmaleri.',
+    'To personer, en i lyse jeans og en stripete skjorte, spiller biljard.',
+    'Folk spiller biljard',
+    'folk løper',
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
-# [3, 640]
+# [3, 960]
 
 # Get the similarity scores for the embeddings
 similarities = model.similarity(embeddings, embeddings)
 print(similarities)
-# tensor([[1.0000, 0.5608, 0.3858],
-#         [0.5608, 1.0000, 0.2424],
-#         [0.3858, 0.2424, 1.0000]])
+# tensor([[1.0000, 0.7294, 0.1690],
+#         [0.7294, 1.0000, 0.2412],
+#         [0.1690, 0.2412, 1.0000]])
 ```
 
-<!--
-### Direct Usage (Transformers)
-
-<details><summary>Click to see the direct usage in Transformers</summary>
-
-</details>
--->
-
-<!--
-### Downstream Usage (Sentence Transformers)
-
-You can finetune this model on your own dataset.
-
-<details><summary>Click to expand</summary>
-
-</details>
--->
-
-<!--
-### Out-of-Scope Use
-
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
 
 ## Evaluation
 
-### Metrics
+To verify the utility of our models, we evaluated them on a selection of classification and clustering tasks for Norwegian from [MTEBv2](https://embeddings-benchmark.github.io/mteb/). 
 
-#### Triplet
+The heatmap below shows the results of evaluating five sentence-transformers on ten different tasks; 
+three of the sentence-transformer models we have fine-tuned ([Fremtind/norsbert4-large](https://huggingface.co/Fremtind/norsbert4-large), [Fremtind/norsbert4-base](https://huggingface.co/Fremtind/norsbert4-base), [Fremtind/mmBERT-base-norwegian](https://huggingface.co/Fremtind/mmBERT-base-norwegian)) 
+and the other two are relatively popular (and comparable) sentence similarity models ([FFI/SimCSE-NB-BERT-large](https://huggingface.co/FFI/SimCSE-NB-BERT-large) and [NbAiLab/nb-sbert-base](https://huggingface.co/NbAiLab/nb-sbert-base)).
 
-* Dataset: `nob_all_nli_test`
-* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
 
-| Metric              | Value     |
-|:--------------------|:----------|
-| **cosine_accuracy** | **0.947** |
+![newplot](https://cdn-uploads.huggingface.co/production/uploads/6179579cf08e328ce6c12c26/5N4zMC8AeETTRRig_UWsh.png)
 
-<!--
-## Bias, Risks and Limitations
+We ranked the models using **Borda count** (which is used in MTEB), where each model was assigned a number of points based on its relative performance across all evaluated tasks. 
 
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
+| Rank | Model | Borda Points |
+|:----:|:--------------------------------|:-------------:|
+| 1 | **[Fremtind/norsbert4-large](https://huggingface.co/Fremtind/norsbert4-large)** | **44** |
+| 2 | [FFI/SimCSE-NB-BERT-large](https://huggingface.co/FFI/SimCSE-NB-BERT-large) | 40 |
+| 3 | [Fremtind/norsbert4-base](https://huggingface.co/Murhaf/norsbert4-base) | 24 |
+| 4 | [NbAiLab/nb-sbert-base](https://huggingface.co/NbAiLab/nb-sbert-base) | 15 |
+| 5 | [Fremtind/mmBERT-base-norwegian](https://huggingface.co/Fremtind/mmBERT-base-norwegian) | 7 |
+ 
+ 
 
-<!--
-### Recommendations
+## Training Details
 
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
+The model was fine-tuned in two stages.
 
-## Training Details
+In the **first stage**, it was trained in an unsupervised manner following the SimCSE method (Gao et al., 2021). In this setup, the same sentence is encoded twice, and due to dropout (in training mode), the model produces two slightly different embeddings. The training objective is to minimize the distance between these embeddings while maximizing the distance to embeddings of other sentences in the same batch.
+For this stage, we created sentence pairs in three categories from the [NDLA Parallel Paragraphs dataset](https://huggingface.co/datasets/NbAiLab/ndla_parallel_paragraphs): (Bokmål, Bokmål), (Nynorsk, Nynorsk), and (Bokmål, Nynorsk). In the (Bokmål, Bokmål) and (Nynorsk, Nynorsk) pairs, each sentence was paired with itself, leveraging dropout to create embedding variation. In the (Bokmål, Nynorsk) category, cross-lingual sentence pairs were used to align the model’s semantic representations across the two language varieties.
 
-### Training Dataset
-
-#### all-nli-norwegian
-
-* Dataset: [all-nli-norwegian](https://huggingface.co/datasets/Murhaf/all-nli-norwegian) at [98cabde](https://huggingface.co/datasets/Murhaf/all-nli-norwegian/tree/98cabded09bfe5f505757840026ecdf6a357a04c)
-* Size: 556,367 training samples
-* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | anchor                                                                           | positive                                                                          | negative                                                                         |
-  |:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|
-  | type    | string                                                                           | string                                                                            | string                                                                           |
-  | details | <ul><li>min: 6 tokens</li><li>mean: 9.53 tokens</li><li>max: 47 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 12.03 tokens</li><li>max: 40 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 12.7 tokens</li><li>max: 49 tokens</li></ul> |
-* Samples:
-  | anchor                                                         | positive                                        | negative                                                       |
-  |:---------------------------------------------------------------|:------------------------------------------------|:---------------------------------------------------------------|
-  | <code>En person på en hest hopper over et havarert fly.</code> | <code>En person er utendørs, på en hest.</code> | <code>En person er på en diner og bestiller en omelett.</code> |
-  | <code>Barn smiler og vinker til kameraet</code>                | <code>Det er barn til stede</code>              | <code>Barna rynker pannen</code>                               |
-  | <code>En gutt hopper på skateboard midt på en rød bro.</code>  | <code>Gutten gjør et skateboardtriks.</code>    | <code>Gutten skater nedover fortauet.</code>                   |
-* Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
-  ```json
-  {
-      "scale": 20.0,
-      "similarity_fct": "cos_sim",
-      "mini_batch_size": 32,
-      "gather_across_devices": false
-  }
-  ```
-
-### Evaluation Dataset
-
-#### all-nli-norwegian
-
-* Dataset: [all-nli-norwegian](https://huggingface.co/datasets/Murhaf/all-nli-norwegian) at [98cabde](https://huggingface.co/datasets/Murhaf/all-nli-norwegian/tree/98cabded09bfe5f505757840026ecdf6a357a04c)
-* Size: 6,561 evaluation samples
-* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | anchor                                                                            | positive                                                                         | negative                                                                        |
-  |:--------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|
-  | type    | string                                                                            | string                                                                           | string                                                                          |
-  | details | <ul><li>min: 5 tokens</li><li>mean: 17.72 tokens</li><li>max: 74 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 8.98 tokens</li><li>max: 31 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 9.5 tokens</li><li>max: 29 tokens</li></ul> |
-* Samples:
-  | anchor                                                                                                                          | positive                                                  | negative                                                    |
-  |:--------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------|:------------------------------------------------------------|
-  | <code>To kvinner klemmer mens de holder take-away pakker.</code>                                                                | <code>To kvinner holder pakker.</code>                    | <code>Mennene slåss utenfor en deli.</code>                 |
-  | <code>To små barn i blå drakter, en med nummer 9 og en med nummer 2, står på trinn i et bad og vasker hendene i en vask.</code> | <code>To barn i nummererte drakter vasker hendene.</code> | <code>To barn i jakker går til skolen.</code>               |
-  | <code>En mann selger donuts til en kunde under et verdensutstillingsarrangement holdt i byen Angeles</code>                     | <code>En mann selger donuts til en kunde.</code>          | <code>En kvinne drikker kaffen sin på en liten kafé.</code> |
-* Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
-  ```json
-  {
-      "scale": 20.0,
-      "similarity_fct": "cos_sim",
-      "mini_batch_size": 32,
-      "gather_across_devices": false
-  }
-  ```
+In the **second stage**, the model was further fine-tuned on a natural language inference dataset, namely [Fremtind/all-nli-norwegian](https://huggingface.co/datasets/Fremtind/all-nli-norwegian). The dataset is formatted as triplets (anchor, positive, negative), where the _anchor_ is the premise, the _positive_ is an entailment hypothesis, and the _negative_ is a contradiction hypothesis. The objective is to minimize the distance between the anchor and positive while maximizing it between the anchor and negative. This fine-tuning stage follows the 'standard' supervised fine-tuning strategy introduced in Sentence-BERT.
 
 ### Training Hyperparameters
 #### Non-Default Hyperparameters