sdadas commited on
Commit
3ebd3fc
·
verified ·
1 Parent(s): eaa9d77

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -28,7 +28,7 @@ PolDense models were trained in a three-stage pipeline with two knowledge distil
28
 
29
  2. The second stage used a more advanced distillation objective inspired by [Jasper and Stella: distillation of SOTA embedding models](https://arxiv.org/abs/2412.19048). The goal of this stage was to align the student models more closely with the teacher and extract as much retrieval quality as possible from the distillation process. Unlike the first stage, this stage was performed only on Polish texts. The training objective combined three complementary losses: a) a cosine alignment loss; b) a pairwise similarity loss; c) a triplet-style ranking loss. Together, these losses train the student not only to imitate individual teacher vectors, but also to reproduce the teacher's local geometry and ranking behavior. This is particularly important for retrieval, where relative similarity between queries and passages matters more than isolated embedding values. The teacher model in this stage was again [BGE-Multilingual-Gemma2](https://huggingface.co/BAAI/bge-multilingual-gemma2). Models from 17M to 150M parameters were trained on approximately 70 million Polish texts, while the 400M and 1B models were trained on approximately 33 million Polish texts. The corpus included the 20 million Polish texts used in first stage, with additional texts taken from the Polish portion of the [FineTranslations](https://huggingface.co/datasets/HuggingFaceFW/finetranslations) collection. Each model was trained for 5 epochs with a batch size of 128.
30
 
31
- 3. The final stage fine-tuned the models directly for retrieval using contrastive learning. This stage used 14 retrieval training datasets containing over 4.5 million queries and more than 15 million passages. No ground-truth labels were used during this stage. Instead, the training data was constructed entirely from positive and negative examples selected with the [BGE-Reranker-v2.5-Gemma2-Lightweight](https://huggingface.co/BAAI/bge-reranker-v2.5-gemma2-lightweight) reranker. The models were trained for 10 epochs with a batch size of 1024.
32
 
33
  The figure below presents an evaluation of PolDense models on the [PIRB benchmark](https://huggingface.co/spaces/sdadas/pirb), which consists of 41 Polish retrieval tasks. As shown, each training stage contributed to the model’s final performance:
34
 
 
28
 
29
  2. The second stage used a more advanced distillation objective inspired by [Jasper and Stella: distillation of SOTA embedding models](https://arxiv.org/abs/2412.19048). The goal of this stage was to align the student models more closely with the teacher and extract as much retrieval quality as possible from the distillation process. Unlike the first stage, this stage was performed only on Polish texts. The training objective combined three complementary losses: a) a cosine alignment loss; b) a pairwise similarity loss; c) a triplet-style ranking loss. Together, these losses train the student not only to imitate individual teacher vectors, but also to reproduce the teacher's local geometry and ranking behavior. This is particularly important for retrieval, where relative similarity between queries and passages matters more than isolated embedding values. The teacher model in this stage was again [BGE-Multilingual-Gemma2](https://huggingface.co/BAAI/bge-multilingual-gemma2). Models from 17M to 150M parameters were trained on approximately 70 million Polish texts, while the 400M and 1B models were trained on approximately 33 million Polish texts. The corpus included the 20 million Polish texts used in first stage, with additional texts taken from the Polish portion of the [FineTranslations](https://huggingface.co/datasets/HuggingFaceFW/finetranslations) collection. Each model was trained for 5 epochs with a batch size of 128.
30
 
31
+ 3. The final stage fine-tuned the models directly for retrieval using contrastive learning. This stage used 13 retrieval training datasets containing over 4.5 million queries and more than 15 million passages. No ground-truth labels were used during this stage. Instead, the training data was constructed entirely from positive and negative examples selected with the [BGE-Reranker-v2.5-Gemma2-Lightweight](https://huggingface.co/BAAI/bge-reranker-v2.5-gemma2-lightweight) reranker. The models were trained for 10 epochs with a batch size of 1024.
32
 
33
  The figure below presents an evaluation of PolDense models on the [PIRB benchmark](https://huggingface.co/spaces/sdadas/pirb), which consists of 41 Polish retrieval tasks. As shown, each training stage contributed to the model’s final performance:
34