Sentence Similarity
sentence-transformers
Safetensors
Transformers
Polish
modernbert
feature-extraction
text-embeddings-inference
Instructions to use OPI-PIB/PolDense-150M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use OPI-PIB/PolDense-150M with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("OPI-PIB/PolDense-150M") sentences = [ "zapytanie: Jak dożyć 100 lat?", "Trzeba zdrowo się odżywiać i uprawiać sport.", "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.", "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Transformers
How to use OPI-PIB/PolDense-150M with Transformers:
# Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("OPI-PIB/PolDense-150M") model = AutoModel.from_pretrained("OPI-PIB/PolDense-150M", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -28,7 +28,7 @@ PolDense models were trained in a three-stage pipeline with two knowledge distil
|
|
| 28 |
|
| 29 |
2. The second stage used a more advanced distillation objective inspired by [Jasper and Stella: distillation of SOTA embedding models](https://arxiv.org/abs/2412.19048). The goal of this stage was to align the student models more closely with the teacher and extract as much retrieval quality as possible from the distillation process. Unlike the first stage, this stage was performed only on Polish texts. The training objective combined three complementary losses: a) a cosine alignment loss; b) a pairwise similarity loss; c) a triplet-style ranking loss. Together, these losses train the student not only to imitate individual teacher vectors, but also to reproduce the teacher's local geometry and ranking behavior. This is particularly important for retrieval, where relative similarity between queries and passages matters more than isolated embedding values. The teacher model in this stage was again [BGE-Multilingual-Gemma2](https://huggingface.co/BAAI/bge-multilingual-gemma2). Models from 17M to 150M parameters were trained on approximately 70 million Polish texts, while the 400M and 1B models were trained on approximately 33 million Polish texts. The corpus included the 20 million Polish texts used in first stage, with additional texts taken from the Polish portion of the [FineTranslations](https://huggingface.co/datasets/HuggingFaceFW/finetranslations) collection. Each model was trained for 5 epochs with a batch size of 128.
|
| 30 |
|
| 31 |
-
3. The final stage fine-tuned the models directly for retrieval using contrastive learning. This stage used
|
| 32 |
|
| 33 |
The figure below presents an evaluation of PolDense models on the [PIRB benchmark](https://huggingface.co/spaces/sdadas/pirb), which consists of 41 Polish retrieval tasks. As shown, each training stage contributed to the model’s final performance:
|
| 34 |
|
|
|
|
| 28 |
|
| 29 |
2. The second stage used a more advanced distillation objective inspired by [Jasper and Stella: distillation of SOTA embedding models](https://arxiv.org/abs/2412.19048). The goal of this stage was to align the student models more closely with the teacher and extract as much retrieval quality as possible from the distillation process. Unlike the first stage, this stage was performed only on Polish texts. The training objective combined three complementary losses: a) a cosine alignment loss; b) a pairwise similarity loss; c) a triplet-style ranking loss. Together, these losses train the student not only to imitate individual teacher vectors, but also to reproduce the teacher's local geometry and ranking behavior. This is particularly important for retrieval, where relative similarity between queries and passages matters more than isolated embedding values. The teacher model in this stage was again [BGE-Multilingual-Gemma2](https://huggingface.co/BAAI/bge-multilingual-gemma2). Models from 17M to 150M parameters were trained on approximately 70 million Polish texts, while the 400M and 1B models were trained on approximately 33 million Polish texts. The corpus included the 20 million Polish texts used in first stage, with additional texts taken from the Polish portion of the [FineTranslations](https://huggingface.co/datasets/HuggingFaceFW/finetranslations) collection. Each model was trained for 5 epochs with a batch size of 128.
|
| 30 |
|
| 31 |
+
3. The final stage fine-tuned the models directly for retrieval using contrastive learning. This stage used 13 retrieval training datasets containing over 4.5 million queries and more than 15 million passages. No ground-truth labels were used during this stage. Instead, the training data was constructed entirely from positive and negative examples selected with the [BGE-Reranker-v2.5-Gemma2-Lightweight](https://huggingface.co/BAAI/bge-reranker-v2.5-gemma2-lightweight) reranker. The models were trained for 10 epochs with a batch size of 1024.
|
| 32 |
|
| 33 |
The figure below presents an evaluation of PolDense models on the [PIRB benchmark](https://huggingface.co/spaces/sdadas/pirb), which consists of 41 Polish retrieval tasks. As shown, each training stage contributed to the model’s final performance:
|
| 34 |
|