Update README.md

README.md

@@ -8,121 +8,144 @@ tags:
 - transformers
 
 ---
+## Model Description:
+[**vietnamese-embedding**](https://huggingface.co/dangvantuan/sentence-camembert-large) is the Embedding Model for Vietnamese language. This model is a specialized sentence-embedding trained specifically for the Vietnamese language, leveraging the robust capabilities of PhoBERT, a pre-trained language model based on the RoBERTa architecture.
+The model utilizes PhoBERT to encode Vietnamese sentences into a 768-dimensional vector space, facilitating a wide range of applications from semantic search to text clustering. The embeddings capture the nuanced meanings of Vietnamese sentences, reflecting both the lexical and contextual layers of the language.
 
-# {Vietnamese Embedding}
-
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
-
-<!--- Describe your model here -->
-
-## Usage (Sentence-Transformers)
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: RobertaModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
+## Training and Fine-tuning process
+The model underwent a rigorous four-stage training and fine-tuning process, each tailored to enhance its ability to generate precise and contextually relevant sentence embeddings for the Vietnamese language. Below is an outline of these stages:
+#### Stage 1: Initial Training
+- Dataset: [ViNLI-SimCSE-supervised](https://huggingface.co/datasets/anti-ai/ViNLI-SimCSE-supervised)
+- Method: Trained using the [SimCSE approach](https://arxiv.org/abs/2104.08821) which employs a supervised contrastive learning framework. The model was optimized using [Triplet Loss](https://www.sbert.net/docs/package_reference/losses.html#tripletloss) to effectively learn from high-quality annotated sentence pairs.
+#### Stage 2: Continued Fine-tuning
+- Dataset: [XNLI-vn ](https://huggingface.co/datasets/xnli/viewer/vi)
+- Method: Continued fine-tuning using Multi-Negative Ranking Loss. This stage focused on improving the model's ability to discern and rank nuanced differences in sentence semantics.
+### Stage 3: Continued Fine-tuning for Semantic Textual Similarity on STS Benchmark
+- Dataset: [STSB-vn](https://huggingface.co/datasets/doanhieung/vi-stsbenchmark)
+- Method: Fine-tuning specifically for the semantic textual similarity benchmark using Siamese BERT-Networks configured with the 'sentence-transformers' library. This stage honed the model's precision in capturing semantic similarity across various types of Vietnamese texts.
+### Stage 4: Advanced Augmentation Fine-tuning
+- Dataset: STSB-vn with generate [silver sample from gold sample](https://www.sbert.net/examples/training/data_augmentation/README.html)
+- Method: Employed an advanced strategy using [Augmented SBERT](https://arxiv.org/abs/2010.08240) with Pair Sampling Strategies, integrating both Cross-Encoder and Bi-Encoder models. This stage further refined the embeddings by enriching the training data dynamically, enhancing the model's robustness and accuracy in understanding and processing complex Vietnamese language constructs.
+
+
+## Usage:
 
 Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
 
 ```
 pip install -U sentence-transformers
+pip install -q pyvi
 ```
 
 Then you can use the model like this:
 
 ```python
 from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
-
-model = SentenceTransformer('{MODEL_NAME}')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
-
-
-
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-
-
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-
-
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
-
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
-model = AutoModel.from_pretrained('{MODEL_NAME}')
-
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
-
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
+from pyvi.ViTokenizer import tokenize
 
-# Perform pooling. In this case, mean pooling.
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+sentences = ["Hà Nội là thủ đô của Việt Nam", "Đà Nẵng là thành phố du lịch"]
+tokenizer_sent = [tokenize(sent) for sent in sentences]
 
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
-
-
-
-## Evaluation Results
-
-<!--- Describe how your model was evaluated -->
-
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
-
-
-## Training
-The model was trained with the parameters:
-
-**DataLoader**:
+model = SentenceTransformer('dangvantuan/vietnamese-embedding')
+embeddings = model.encode(tokenizer_sent)
+print(embeddings)
 
-`torch.utils.data.dataloader.DataLoader` of length 719 with parameters:
-```
-{'batch_size': 8, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
 ```
 
-**Loss**:
-
-`__main__.CosineSimilarityLoss` 
-
-Parameters of the fit()-Method:
-```
-{
-    "epochs": 20,
-    "evaluation_steps": 500,
-    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
-    "max_grad_norm": 1,
-    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
-    "optimizer_params": {
-        "eps": 1e-06,
-        "lr": 5e-06
-    },
-    "scheduler": "WarmupLinear",
-    "steps_per_epoch": null,
-    "warmup_steps": 1438,
-    "weight_decay": 0.01
-}
-```
 
+## Evaluation
+The model can be evaluated as follows on the Vienamese data of stsb.
 
-## Full Model Architecture
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: RobertaModel 
-  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-)
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.readers import InputExample
+from datasets import load_dataset
+from pyvi.ViTokenizer import tokenize
+def convert_dataset(dataset):
+    dataset_samples=[]
+    for df in dataset:
+        score = float(df['score'])/5.0  # Normalize score to range 0 ... 1
+        inp_example = InputExample(texts=[tokenize(df['sentence1']), 
+                                    tokenize(df['sentence2'])], label=score)
+        dataset_samples.append(inp_example)
+    return dataset_samples
+
+# Loading the dataset for evaluation
+vi_sts = load_dataset("doanhieung/vi-stsbenchmark")["train"]
+df_dev = vi_sts.filter(lambda example: example['split'] == 'dev')
+df_test = vi_sts.filter(lambda example: example['split'] == 'test')
+
+# Convert the dataset for evaluation
+
+# For Dev set:
+dev_samples = convert_dataset(df_dev)
+val_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(dev_samples, name='sts-dev')
+val_evaluator(model, output_path="./")
+
+# For Test set:
+test_samples = convert_dataset(df_test)
+test_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(test_samples, name='sts-test')
+test_evaluator(model, output_path="./")
 ```
 
-## Citing & Authors
 
-<!--- Describe where people can find more information -->
+### Test Result:
+The performance is measured using Pearson and Spearman correlation:
+- On dev
+| Model  | Pearson correlation | Spearman correlation  | #params  |
+| ------------- | ------------- | ------------- |------------- |
+| [dangvantuan/vietnamese-embedding](dangvantuan/vietnamese-embedding)| 88.33 |88.2 | 135| 
+| [VoVanPhuc/sup-SimCSE-VietNamese-phobert-base](https://huggingface.co/VoVanPhuc/sup-SimCSE-VietNamese-phobert-base)  | 84.65|84.59 | 135 |
+| [keepitreal/vietnamese-sbert](https://huggingface.co/keepitreal/vietnamese-sbert) | 84.51 | 84.44|135M |
+| [bkai-foundation-models/vietnamese-bi-encoder](https://huggingface.co/bkai-foundation-models/vietnamese-bi-encoder) | 78.05 | 77.94|135 |
+
+### Metric for all dataset of [Semantic Textual Similarity on STS Benchmark](https://huggingface.co/datasets/doanhieung/vi-stsbenchmark)
+
+**Pearson score**
+| Model                                                                                                               | [STSB-vn]   | [STS12-vn ]| [STS13-vn] | [STS14-vn] | [STS15-vn] |    [STS16-vn] | [SICK-fr] | Mean |
+|-----------------------------------------------------------|---------|----------|----------|----------|----------|----------|---------|--------|
+| [dangvantuan/vietnamese-embedding](dangvantuan/vietnamese-embedding)                                                 |84.87	|87.23|	85.39|	82.94|	86.91|	79.39|	82.77|	84.21|
+| [VoVanPhuc/sup-SimCSE-VietNamese-phobert-base](https://huggingface.co/VoVanPhuc/sup-SimCSE-VietNamese-phobert-base)  |81.52|	85.02|	78.22|	75.94|	81.53|	75.39|	77.75|	79.33|
+| [keepitreal/vietnamese-sbert](https://huggingface.co/keepitreal/vietnamese-sbert)                                    |80.54|	78.58|	80.75|	76.98|	82.57|	73.21|	80.16|	78.97|
+| [bkai-foundation-models/vietnamese-bi-encoder](https://huggingface.co/bkai-foundation-models/vietnamese-bi-encoder)  |73.30|	67.84|	71.69|	69.80|	78.40|	74.29|	76.01|	73.04|
+
+
+**Spearman score**
+| Model                                                                                                               | [STSB-vn]   | [STS12-vn ]| [STS13-vn] | [STS14-vn] | [STS15-vn] |    [STS16-vn] | [SICK-fr] | Mean |
+|-----------------------------------------------------------|---------|----------|----------|----------|----------|----------|---------|--------|
+| [dangvantuan/vietnamese-embedding](dangvantuan/vietnamese-embedding)                                                 |84.84|	79.04|	85.30|	81.38|	87.06|	79.95|	79.58|	82.45|
+| [VoVanPhuc/sup-SimCSE-VietNamese-phobert-base](https://huggingface.co/VoVanPhuc/sup-SimCSE-VietNamese-phobert-base)  |81.43|	76.51|	79.19|	74.91|	81.72|	76.57|	76.45|	78.11|
+| [keepitreal/vietnamese-sbert](https://huggingface.co/keepitreal/vietnamese-sbert)                                    |80.16|	69.08|	80.99|	73.67|	82.81|	74.30|	73.40|	76.34|
+| [bkai-foundation-models/vietnamese-bi-encoder](https://huggingface.co/bkai-foundation-models/vietnamese-bi-encoder)  |72.16|	63.86|	71.82|	66.20|	78.62|	74.24|	70.87|	71.11|
+
+## Citation
+
+
+	@article{reimers2019sentence,
+	   title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
+	   author={Nils Reimers, Iryna Gurevych},
+	   journal={https://arxiv.org/abs/1908.10084},
+	   year={2019}
+	}
+
+
+	@article{martin2020camembert,
+	   title={CamemBERT: a Tasty French Language Mode},
+	   author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
+	   journal={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
+	   year={2020}
+	}
+    @article{thakur2020augmented,
+      title={Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks},
+      author={Thakur, Nandan and Reimers, Nils and Daxenberger, Johannes and Gurevych, Iryna},
+      journal={arXiv e-prints},
+      pages={arXiv--2010},
+      year={2020}