Update README.md
Browse files
README.md
CHANGED
|
@@ -19,6 +19,8 @@ base_model:
|
|
| 19 |
## Overview
|
| 20 |
This checkpoint continues the pre-training of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on Scandinavian text, extending the model’s knowledge with ~1.2 trillion additional masked-language-model (MLM) tokens drawn from [The Nordic Pile](https://arxiv.org/pdf/2303.17183) and [SWEb](https://arxiv.org/pdf/2410.04456) while preserving the original 8k token context window.
|
| 21 |
|
|
|
|
|
|
|
| 22 |
Our tokenizer is trained from scratch on a subset of 11 985 103 472 tokens.
|
| 23 |
|
| 24 |
The training is done in one stage with 8192 tokens per sample for the whole run.
|
|
|
|
| 19 |
## Overview
|
| 20 |
This checkpoint continues the pre-training of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on Scandinavian text, extending the model’s knowledge with ~1.2 trillion additional masked-language-model (MLM) tokens drawn from [The Nordic Pile](https://arxiv.org/pdf/2303.17183) and [SWEb](https://arxiv.org/pdf/2410.04456) while preserving the original 8k token context window.
|
| 21 |
|
| 22 |
+
This is a **research artefact** and is only intended for **research purposes**.
|
| 23 |
+
|
| 24 |
Our tokenizer is trained from scratch on a subset of 11 985 103 472 tokens.
|
| 25 |
|
| 26 |
The training is done in one stage with 8192 tokens per sample for the whole run.
|