Update README.md

README.md

@@ -1,99 +1,78 @@
----
-language:
-- 'no'
-tags:
-- sentence-transformers
-- sentence-similarity
-- feature-extraction
-- dense
-- generated_from_trainer
-- dataset_size:556367
-- loss:CachedMultipleNegativesRankingLoss
-base_model:
-- jhu-clsp/mmBERT-base
-widget:
-- source_sentence: Inne i igloen gjør den unge mannen seg klar for sitt overnattingsopphold.
-  sentences:
-  - Folk danser i gaten.
-  - Den unge mannen gjør seg klar for sitt overnattingsopphold.
-  - Den unge mannen gjør seg klar til å dra.
-- source_sentence: >-
-    En kvinne i rullestol snakker med vennen sin mens hun er omgitt av andre
-    mennesker som går i parken.
-  sentences:
-  - Barna blir fotografert.
-  - Kvinnen er utendørs.
-  - Kvinnen spiser en pølse midt i soverommet sitt.
-- source_sentence: En kvinne løper langs en steinete strand.
-  sentences:
-  - En mann og en kvinne ser på frukt og grønnsaker.
-  - En kvinne løper.
-  - En kvinne sitter ved et piknikbord nær den steinete kysten.
-- source_sentence: >-
-    To basketballspillere i svart og hvitt antrekk står på en basketballbane og
-    snakker.
-  sentences:
-  - De to basketballspillerne snakker sammen.
-  - Den unge gutten multitasker.
-  - De to basketballspillerne sitter på benken.
-- source_sentence: En mann lager et sandmaleri på gulvet.
-  sentences:
-  - En mann lager kunst.
-  - På fornøyelsesturen var det to jenter som smilte og lo
-  - En kvinne ødelegger et sandmaleri.
-datasets:
-- Fremtind/all-nli-norwegian
-- NbAiLab/ndla_parallel_paragraphs
-pipeline_tag: sentence-similarity
-library_name: sentence-transformers
-metrics:
-- cosine_accuracy
-model-index:
-- name: SentenceTransformer based on Murhaf/mmBERT-base-ndla
-  results:
-  - task:
-      type: triplet
-      name: Triplet
-    dataset:
-      name: nob all nli test
-      type: nob_all_nli_test
-    metrics:
-    - type: cosine_accuracy
-      value: 0.9509999752044678
-      name: Cosine Accuracy
----
-
-# SentenceTransformer based on Murhaf/mmBERT-base-ndla
-
-This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) on the [all-nli-norwegian](https://huggingface.co/datasets/Fremtind/all-nli-norwegian) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
-
-## Model Details
-
-### Model Description
-- **Model Type:** Sentence Transformer
-- **Base model:** [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) <!-- at revision 8559aa0f57365bb94e7fefea420b6daa118e276a -->
-- **Maximum Sequence Length:** 75 tokens
-- **Output Dimensionality:** 768 dimensions
-- **Similarity Function:** Cosine Similarity
-- **Training Dataset:**
-    - [all-nli-norwegian](https://huggingface.co/datasets/Fremtind/all-nli-norwegian)
-- **Language:** no
-<!-- - **License:** Unknown -->
-
-### Model Sources
-
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
-- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
-
-### Full Model Architecture
-
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 75, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
-  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-)
-```
+---
+language:
+- 'no'
+- nb
+- nn
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- dense
+- loss:CachedMultipleNegativesRankingLoss
+base_model:
+- jhu-clsp/mmBERT-base
+widget:
+- source_sentence: Inne i igloen gjør den unge mannen seg klar for sitt overnattingsopphold.
+  sentences:
+  - Folk danser i gaten.
+  - Den unge mannen gjør seg klar for sitt overnattingsopphold.
+  - Den unge mannen gjør seg klar til å dra.
+- source_sentence: >-
+    En kvinne i rullestol snakker med vennen sin mens hun er omgitt av andre
+    mennesker som går i parken.
+  sentences:
+  - Barna blir fotografert.
+  - Kvinnen er utendørs.
+  - Kvinnen spiser en pølse midt i soverommet sitt.
+- source_sentence: En kvinne løper langs en steinete strand.
+  sentences:
+  - En mann og en kvinne ser på frukt og grønnsaker.
+  - En kvinne løper.
+  - En kvinne sitter ved et piknikbord nær den steinete kysten.
+- source_sentence: >-
+    To basketballspillere i svart og hvitt antrekk står på en basketballbane og
+    snakker.
+  sentences:
+  - De to basketballspillerne snakker sammen.
+  - Den unge gutten multitasker.
+  - De to basketballspillerne sitter på benken.
+- source_sentence: En mann lager et sandmaleri på gulvet.
+  sentences:
+  - En mann lager kunst.
+  - På fornøyelsesturen var det to jenter som smilte og lo
+  - En kvinne ødelegger et sandmaleri.
+datasets:
+- Fremtind/all-nli-norwegian
+- NbAiLab/ndla_parallel_paragraphs
+pipeline_tag: sentence-similarity
+library_name: sentence-transformers
+metrics:
+- cosine_accuracy
+model-index:
+- name: SentenceTransformer based on jhu-clsp/mmBERT-base
+  results:
+  - task:
+      type: triplet
+      name: Triplet
+    dataset:
+      name: nob all nli test
+      type: nob_all_nli_test
+    metrics:
+    - type: cosine_accuracy
+      value: 0.9509999752044678
+      name: Cosine Accuracy
+license: mit
+---
+
+# SentenceTransformer based on jhu-clsp/mmBERT-base
+
+mmBERT-base-norwegian is a [Sentence Transformer](https://www.SBERT.net) model finetuned from the multilingual ModernBERT model [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base). 
+The model maps sentences (and paragraphs) to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, 
+text classification, clustering, among other tasks.
+
+
+Note: While the fine-tuned sentence-transformer model has a `max_seq_length` of 75 tokens, the base model does not. 
+This means that the sequence length can be increased to 8192 (which is the max length in the base model).
 
 ## Usage
 
@@ -110,7 +89,7 @@ Then you can load this model and run inference.
 from sentence_transformers import SentenceTransformer
 
 # Download from the 🤗 Hub
-model = SentenceTransformer("Murhaf/mmBERT-base-ndla-all-nli")
+model = SentenceTransformer("Fremtind/mmBERT-base-norwegian")
 # Run inference
 sentences = [
     'En mann lager et sandmaleri på gulvet.',
@@ -129,123 +108,51 @@ print(similarities)
 #         [0.3718, 0.1509, 1.0000]])
 ```
 
-<!--
-### Direct Usage (Transformers)
-
-<details><summary>Click to see the direct usage in Transformers</summary>
-
-</details>
--->
-
-<!--
-### Downstream Usage (Sentence Transformers)
-
-You can finetune this model on your own dataset.
-
-<details><summary>Click to expand</summary>
-
-</details>
--->
-
-<!--
-### Out-of-Scope Use
-
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
 
 ## Evaluation
 
-### Metrics
+To verify the utility of our models, we evaluated them on a selection of classification and clustering tasks for Norwegian from [MTEBv2](https://embeddings-benchmark.github.io/mteb/). 
 
-#### Triplet
+The heatmap below shows the results of evaluating five sentence-transformers on ten different tasks; 
+three of the sentence-transformer models we have fine-tuned ([Fremtind/norsbert4-large](https://huggingface.co/Fremtind/norsbert4-large), [Fremtind/norsbert4-base](https://huggingface.co/Fremtind/norsbert4-base), [Fremtind/mmBERT-base-norwegian](https://huggingface.co/Fremtind/mmBERT-base-norwegian)) 
+and the other two are relatively popular (and comparable) sentence similarity models ([FFI/SimCSE-NB-BERT-large](https://huggingface.co/FFI/SimCSE-NB-BERT-large) and [NbAiLab/nb-sbert-base](https://huggingface.co/NbAiLab/nb-sbert-base)).
 
-* Dataset: `nob_all_nli_test`
-* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
 
-| Metric              | Value     |
-|:--------------------|:----------|
-| **cosine_accuracy** | **0.951** |
+![newplot](https://cdn-uploads.huggingface.co/production/uploads/6179579cf08e328ce6c12c26/5N4zMC8AeETTRRig_UWsh.png)
 
-<!--
-## Bias, Risks and Limitations
+We ranked the models using **Borda count** (which is used in MTEB), where each model was assigned a number of points based on its relative performance across all evaluated tasks. 
 
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
+| Rank | Model | Borda Points |
+|:----:|:--------------------------------|:-------------:|
+| 1 | **[Fremtind/norsbert4-large](https://huggingface.co/Fremtind/norsbert4-large)** | **44** |
+| 2 | [FFI/SimCSE-NB-BERT-large](https://huggingface.co/FFI/SimCSE-NB-BERT-large) | 40 |
+| 3 | [Fremtind/norsbert4-base](https://huggingface.co/Murhaf/norsbert4-base) | 24 |
+| 4 | [NbAiLab/nb-sbert-base](https://huggingface.co/NbAiLab/nb-sbert-base) | 15 |
+| 5 | Fremtind/mmBERT-base-norwegian | 7 |
+ 
+ 
 
-<!--
-### Recommendations
+## Training Details
 
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
+The model was fine-tuned in two stages.
 
-## Training Details
+In the **first stage**, it was trained in an unsupervised manner following the SimCSE method (Gao et al., 2021). In this setup, the same sentence is encoded twice, and due to dropout (in training mode), the model produces two slightly different embeddings. The training objective is to minimize the distance between these embeddings while maximizing the distance to embeddings of other sentences in the same batch.
+For this stage, we created sentence pairs in three categories from the [NDLA Parallel Paragraphs dataset](https://huggingface.co/datasets/NbAiLab/ndla_parallel_paragraphs): (Bokmål, Bokmål), (Nynorsk, Nynorsk), and (Bokmål, Nynorsk). In the (Bokmål, Bokmål) and (Nynorsk, Nynorsk) pairs, each sentence was paired with itself, leveraging dropout to create embedding variation. In the (Bokmål, Nynorsk) category, cross-lingual sentence pairs were used to align the model’s semantic representations across the two language varieties.
 
-### Training Dataset
-
-#### all-nli-norwegian
-
-* Dataset: [all-nli-norwegian](https://huggingface.co/datasets/Murhaf/all-nli-norwegian) at [98cabde](https://huggingface.co/datasets/Murhaf/all-nli-norwegian/tree/98cabded09bfe5f505757840026ecdf6a357a04c)
-* Size: 556,367 training samples
-* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | anchor                                                                           | positive                                                                          | negative                                                                          |
-  |:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
-  | type    | string                                                                           | string                                                                            | string                                                                            |
-  | details | <ul><li>min: 6 tokens</li><li>mean: 11.7 tokens</li><li>max: 56 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 15.03 tokens</li><li>max: 50 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 15.51 tokens</li><li>max: 55 tokens</li></ul> |
-* Samples:
-  | anchor                                                         | positive                                        | negative                                                       |
-  |:---------------------------------------------------------------|:------------------------------------------------|:---------------------------------------------------------------|
-  | <code>En person på en hest hopper over et havarert fly.</code> | <code>En person er utendørs, på en hest.</code> | <code>En person er på en diner og bestiller en omelett.</code> |
-  | <code>Barn smiler og vinker til kameraet</code>                | <code>Det er barn til stede</code>              | <code>Barna rynker pannen</code>                               |
-  | <code>En gutt hopper på skateboard midt på en rød bro.</code>  | <code>Gutten gjør et skateboardtriks.</code>    | <code>Gutten skater nedover fortauet.</code>                   |
-* Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
-  ```json
-  {
-      "scale": 20.0,
-      "similarity_fct": "cos_sim",
-      "mini_batch_size": 32,
-      "gather_across_devices": false
-  }
-  ```
-
-### Evaluation Dataset
-
-#### all-nli-norwegian
-
-* Dataset: [all-nli-norwegian](https://huggingface.co/datasets/Murhaf/all-nli-norwegian) at [98cabde](https://huggingface.co/datasets/Murhaf/all-nli-norwegian/tree/98cabded09bfe5f505757840026ecdf6a357a04c)
-* Size: 6,561 evaluation samples
-* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | anchor                                                                            | positive                                                                          | negative                                                                          |
-  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
-  | type    | string                                                                            | string                                                                            | string                                                                            |
-  | details | <ul><li>min: 6 tokens</li><li>mean: 21.73 tokens</li><li>max: 75 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 11.14 tokens</li><li>max: 37 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 11.75 tokens</li><li>max: 32 tokens</li></ul> |
-* Samples:
-  | anchor                                                                                                                          | positive                                                  | negative                                                    |
-  |:--------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------|:------------------------------------------------------------|
-  | <code>To kvinner klemmer mens de holder take-away pakker.</code>                                                                | <code>To kvinner holder pakker.</code>                    | <code>Mennene slåss utenfor en deli.</code>                 |
-  | <code>To små barn i blå drakter, en med nummer 9 og en med nummer 2, står på trinn i et bad og vasker hendene i en vask.</code> | <code>To barn i nummererte drakter vasker hendene.</code> | <code>To barn i jakker går til skolen.</code>               |
-  | <code>En mann selger donuts til en kunde under et verdensutstillingsarrangement holdt i byen Angeles</code>                     | <code>En mann selger donuts til en kunde.</code>          | <code>En kvinne drikker kaffen sin på en liten kafé.</code> |
-* Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
-  ```json
-  {
-      "scale": 20.0,
-      "similarity_fct": "cos_sim",
-      "mini_batch_size": 32,
-      "gather_across_devices": false
-  }
-  ```
+In the **second stage**, the model was further fine-tuned on a natural language inference dataset, namely [Fremtind/all-nli-norwegian](https://huggingface.co/datasets/Fremtind/all-nli-norwegian). The dataset is formatted as triplets (anchor, positive, negative), where the _anchor_ is the premise, the _positive_ is an entailment hypothesis, and the _negative_ is a contradiction hypothesis. The objective is to minimize the distance between the anchor and positive while maximizing it between the anchor and negative. This fine-tuning stage follows the 'standard' supervised fine-tuning strategy introduced in Sentence-BERT.
 
 ### Training Hyperparameters
 #### Non-Default Hyperparameters
-
+<details><summary>Click to expand</summary>
+  
 - `eval_strategy`: steps
 - `per_device_train_batch_size`: 512
 - `per_device_eval_batch_size`: 256
 - `num_train_epochs`: 1
 - `warmup_ratio`: 0.1
 - `batch_sampler`: no_duplicates
-
+</details>
+  
 #### All Hyperparameters
 <details><summary>Click to expand</summary>
 
@@ -370,14 +277,10 @@ You can finetune this model on your own dataset.
 
 </details>
 
-### Training Logs
-| Epoch  | Step | Training Loss | Validation Loss | nob_all_nli_test_cosine_accuracy |
-|:------:|:----:|:-------------:|:---------------:|:--------------------------------:|
-| 0.3690 | 100  | 1.7006        | 0.6474          | 0.9420                           |
-| 0.7380 | 200  | 1.1183        | 0.6002          | 0.9510                           |
-
 
 ### Framework Versions
+<details><summary>Click to expand</summary>
+
 - Python: 3.12.11
 - Sentence Transformers: 5.1.1
 - Transformers: 4.56.2
@@ -385,35 +288,8 @@ You can finetune this model on your own dataset.
 - Accelerate: 1.10.1
 - Datasets: 4.1.1
 - Tokenizers: 0.22.1
+</details>
 
-## Citation
-
-### BibTeX
-
-#### Sentence Transformers
-```bibtex
-@inproceedings{reimers-2019-sentence-bert,
-    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
-    author = "Reimers, Nils and Gurevych, Iryna",
-    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
-    month = "11",
-    year = "2019",
-    publisher = "Association for Computational Linguistics",
-    url = "https://arxiv.org/abs/1908.10084",
-}
-```
-
-#### CachedMultipleNegativesRankingLoss
-```bibtex
-@misc{gao2021scaling,
-    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
-    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
-    year={2021},
-    eprint={2101.06983},
-    archivePrefix={arXiv},
-    primaryClass={cs.LG}
-}
-```
 
 <!--
 ## Glossary