Update README.md
Browse files
README.md
CHANGED
|
@@ -36,7 +36,7 @@ HTML-LM representations can be reused across a **wide range of downstream applic
|
|
| 36 |
The model was trained from scratch for one pass over the training corpus using a combination of:
|
| 37 |
|
| 38 |
* Masked Language Modeling (MLM)
|
| 39 |
-
* Bag-of-Words prediction from `[CLS]`
|
| 40 |
* Contrastive distillation from [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) and [Seznam SeLLMa 8B](https://blog.seznam.cz/2024/10/diana-hlavacova-sellma-aneb-jak-v-seznamu-krotime-drave-jazykove-modely/)
|
| 41 |
|
| 42 |
## Training data
|
|
|
|
| 36 |
The model was trained from scratch for one pass over the training corpus using a combination of:
|
| 37 |
|
| 38 |
* Masked Language Modeling (MLM)
|
| 39 |
+
* [Bag-of-Words prediction](https://arxiv.org/abs/2401.11248) from `[CLS]`
|
| 40 |
* Contrastive distillation from [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) and [Seznam SeLLMa 8B](https://blog.seznam.cz/2024/10/diana-hlavacova-sellma-aneb-jak-v-seznamu-krotime-drave-jazykove-modely/)
|
| 41 |
|
| 42 |
## Training data
|