Add Model Card

README.md

@@ -0,0 +1,115 @@
+---
+license: apache-2.0
+language: ti
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
+widget:
+- text: "ግራፋይት ኣብ መላእ ዓለም ዳርጋ ብምዕሩይ ዝርጋሐ’ዩ ዝርከብ"
+---
+
+# TiELECTRA BiEncoder Model
+
+This model is a bi-encoder model for the Tigrinya language based on [TiELECTRA-small](https://huggingface.co/fgaim/tielectra-small).
+The model maps sentences & paragraphs to a 256 dimensional dense vector space and can be used for tasks like text embedding, clustering, or semantic search.
+
+This is part of a work that introduces monolingual bi-encoder language models for Tigrinya. For a larger and more powerful model look at [TiRoBERTa-bi-encoder](https://huggingface.co/fgaim/tiroberta-bi-encoder). The models are based on the [sentence-transformers](https://www.sbert.net) architecture and are trained on Tigrinya question-answering and information retrieval datasets. The models are designed to support semantic search tasks, such as information retrieval, text representation, and question answering.
+
+## Using Model with Sentence-Transformers
+
+Using this model becomes easy when you have sentence-transformers installed:
+
+```shell
+pip install -U sentence-transformers
+```
+
+Then use the model as follows:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["ሓደ ሰብኣይ ፈረስ ይጋልብ ኣሎ።", "ሓንቲ ጓል ክራር ትጻወት ኣላ።"]
+
+model = SentenceTransformer('fgaim/tielectra-bi-encoder')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+## Using Model with 🤗 Transformers
+
+Use the transformers library as follows:
+Pass the input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+
+# Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0]  # First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ["ሓደ ሰብኣይ ፈረስ ይጋልብ ኣሎ።", "ሓንቲ ጓል ክራር ትጻወት ኣላ።"]
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained("fgaim/tielectra-bi-encoder")
+model = AutoModel.from_pretrained("fgaim/tielectra-bi-encoder")
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"])
+
+print("Sentence embeddings:", sentence_embeddings)
+```
+
+## Architecture
+
+### Base Model
+
+The model properties:
+
+| Model Size | Layers  | Attn. Heads | Hidden Size  | FFN  | Parameters  | Max. Seq  |
+|------------|----|----|-----|------|------|------|
+| SMALL      | 12 | 4  | 256 | 1024 | 14M  | 512  |
+
+### BiEncoder Model
+
+- Max Seq Length: `512`
+- Word embedding dimension: `256`
+
+```text
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ElectraModel
+
+  (1): Pooling({'word_embedding_dimension': 256, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
+
+## Citation
+
+If you use this model in your product or research, you can cite it as follows:
+
+```bibtex
+@misc{gaim-2024-semantic-search,
+  title     = {{Semantic Search Models for Tigrinya}},
+  author    = {Fitsum Gaim},
+  month     = {January},
+  year      = {2024},
+  publisher = {Hugging Face Hub},
+  doi       = {10.57967/hf/6068},
+  url       = {https://huggingface.co/fgaim/tiroberta-bi-encoder}
+}
+```