|
|
---
|
|
|
library_name: transformers
|
|
|
language:
|
|
|
- ru
|
|
|
- uk
|
|
|
- kk
|
|
|
- be
|
|
|
---
|
|
|
|
|
|
## About model creation
|
|
|
|
|
|
This is a smaller version of the **intfloat/multilingual-e5-base** with only some Russian (Cyrillic in general) and English (fever) tokens (and embeddings) left.
|
|
|
|
|
|
The model created in a similar way as described in this https://medium.com/m/global-identity-2?redirectUrl=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-adapt-a-multilingual-t5-model-for-a-single-language-b9f94f3d9c90 post.
|
|
|
|
|
|
The **CulturaX** dataset was used to search for the required tokens. As a result, out of 250k tokens of the original model, only **69,382** required were left.
|
|
|
|
|
|
## Was the model trained in any way?
|
|
|
|
|
|
No. The tokenizer has been modified, and all changes to token identifiers have been corrected by moving embeddings in the model word_embeddings module to their new places, so **the quality of this model** on Cyrilic (and English) **is exactly the same** as the original one.
|
|
|
|
|
|
## Why do we need this?
|
|
|
|
|
|
This allows you to use significantly less memory during training and also greatly reduces the weight of the model.
|
|
|
|
|
|
## Authors
|
|
|
- Sergei Bratchikov (https://t.me/nlpwanderer) |
