Add model card

README.md

@@ -0,0 +1,126 @@
+---
+language:
+  - ar
+  - de
+  - en
+  - es
+  - fr
+  - it
+  - ja
+  - ko
+  - nl
+  - pl
+  - pt
+  - ru
+  - th
+  - tr
+  - zh
+tags:
+  - bert
+  - sentence_embedding
+  - multilingual
+  - google
+  - sentence-similarity
+  - labse
+license: apache-2.0
+datasets:
+  - CommonCrawl
+  - Wikipedia
+---
+
+# LaBSE
+
+## Model description
+
+Smaller Language-agnostic BERT Sentence Encoder (LaBSE) is a BERT-based model distilled from the [original LaBSE model](https://huggingface.co/setu4993/LaBSE) to 15 languages (from the original 109 languages) using the techniques described in the paper ['Load What You Need: Smaller Versions of Multilingual BERT'](https://arxiv.org/abs/2010.05609) by [Ukjae Jeong](https://github.com/jeongukjae/).
+
+- Model: [HuggingFace's model hub](https://huggingface.co/setu4993/smaller-LaBSE).
+- Original model: [TensorFlow Hub](https://tfhub.dev/jeongukjae/smaller_LaBSE_15lang/1).
+- Distillation source: [GitHub](https://github.com/jeongukjae/smaller-labse).
+- Conversion from TensorFlow to PyTorch: [GitHub](https://github.com/setu4993/convert-labse-tf-pt).
+
+## Usage
+
+Using the model:
+
+```python
+import torch
+from transformers import BertModel, BertTokenizerFast
+
+
+tokenizer = BertTokenizerFast.from_pretrained("setu4993/smaller-LaBSE")
+model = BertModel.from_pretrained("setu4993/smaller-LaBSE")
+model = model.eval()
+
+english_sentences = [
+    "dog",
+    "Puppies are nice.",
+    "I enjoy taking long walks along the beach with my dog.",
+]
+english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    english_outputs = model(**english_inputs)
+```
+
+To get the sentence embeddings, use the pooler output:
+
+```python
+english_embeddings = english_outputs.pooler_output
+```
+
+Output for other languages:
+
+```python
+italian_sentences = [
+    "cane",
+    "I cuccioli sono carini.",
+    "Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
+]
+japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
+italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
+japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    italian_outputs = model(**italian_inputs)
+    japanese_outputs = model(**japanese_inputs)
+
+italian_embeddings = italian_outputs.pooler_output
+japanese_embeddings = japanese_outputs.pooler_output
+```
+
+For similarity between sentences, an L2-norm is recommended before calculating the similarity:
+
+```python
+import torch.nn.functional as F
+
+
+def similarity(embeddings_1, embeddings_2):
+    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
+    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
+    return torch.matmul(
+        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
+    )
+
+
+print(similarity(english_embeddings, italian_embeddings))
+print(similarity(english_embeddings, japanese_embeddings))
+print(similarity(italian_embeddings, japanese_embeddings))
+```
+
+## Details
+
+Details about data, training, evaluation and performance metrics are available in the [original paper](https://arxiv.org/abs/2007.01852).
+
+### BibTeX entry and citation info
+
+```bibtex
+@misc{feng2020languageagnostic,
+      title={Language-agnostic BERT Sentence Embedding},
+      author={Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang},
+      year={2020},
+      eprint={2007.01852},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```