Update README.md
Browse files
README.md
CHANGED
|
@@ -3,9 +3,14 @@ language:
|
|
| 3 |
- ru
|
| 4 |
---
|
| 5 |
|
| 6 |
-
This is a base
|
| 7 |
-
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
The model can be used as-is to produce text embeddings or it can be further fine-tuned for a specific downstream task.
|
| 11 |
|
|
|
|
| 3 |
- ru
|
| 4 |
---
|
| 5 |
|
| 6 |
+
This is a base Longformer model designed for Russian language.
|
| 7 |
+
It was initialized from [blinoff/roberta-base-russian-v0](https://huggingface.co/blinoff/roberta-base-russian-v0) weights and has been modified to support a context length of up to 4096 tokens.
|
| 8 |
+
We fine-tuned it on a dataset of Russian books. For a detailed information check out our post on Habr.
|
| 9 |
+
|
| 10 |
+
Model attributes:
|
| 11 |
+
* 12 attention heads
|
| 12 |
+
* 12 hidden layers
|
| 13 |
+
* 4096 tokens length of context
|
| 14 |
|
| 15 |
The model can be used as-is to produce text embeddings or it can be further fine-tuned for a specific downstream task.
|
| 16 |
|