Update README.md

README.md

@@ -3,26 +3,128 @@ license: apache-2.0
 datasets:
 - HPLT/HPLT3.0
 - allenai/olmo-mix-1124
-- allenai/MADLAD-400
 - HuggingFaceFW/finepdfs
+- HuggingFaceTB/finemath
+- LLM360/MegaMath
+- HuggingFaceTB/stack-edu
+- HuggingFaceFW/finepdfs-edu
 language:
-- fi
+- nb
+- nn
+- 'no'
 base_model:
 - allenai/OLMo-2-1124-13B
 library_name: transformers
 tags:
-- finnish
-- suomi
+- norwegian
+- norsk
 - HPLT
 ---
 
+# NorOLMo
+
 ![](https://hplt-project.org/_next/static/media/logo-hplt.8765d2d4.svg)
 
 This is a base (not instruction-tuned) large language model, continually pre-trained on Finnish data starting from the English [OLMo2-13B](https://huggingface.co/allenai/OLMo-2-1124-13B) model.
 
-Our training data mixture included [HPLTv3](https://huggingface.co/datasets/HPLT/HPLT3.0) Finnish, FinePDF Finnish, MADLAD400 Finnish, OLMo-Mix. 
-The model was trained for 16 000 steps on around 150 billion tokens. Intermediate checkpoints are published here as branches.
+The model was trained for 20 000 steps on around 170 billion tokens. Intermediate checkpoints are published here as branches.
+
+## Data Details
+
+### Stage 1 (16 000 steps -- 135B tokens)
+
+Data
+  - [HPLTv3](https://huggingface.co/datasets/HPLT/HPLT3.0) Finnish
+  - FinePDFs Finnish
+  - OLMo-Mix
+
+Data Splits
+| Data                     | Percentage | Unique Tokens | Total Tokens | Number of Documents | Average Document Length |
+| ------------------------ | ---------- | ------------- | ------------ | ------------------- | ----------------------- |
+| HPLT Finnish             | 69.75      | 46.8B         | 93.6B        | 36.5M               | 944                     |
+| FinePDFs Finnish         | 14.45      | 9.7B          | 19.4B        | 1.5M                | 4 895                   |
+| Wiki (OLMo-Mix)          | 0.02       | 0.2B          | 26.8M        | 0.3M                | 690                     |
+| Alg. Stack (OLMo-Mix)    | 0.04       | 0.6B          | 53.7M        | 0.1M                | 4 291                   |
+| Open Web Math (OLMo-Mix) | 0.04       | 0.6B          | 53.7M        | 0.1M                | 4 291                   |
+| ArXiv (OLMo-Mix)         | 0.05       | 1.1B          | 67.1M        | 0.2M                | 5 318                   |
+| PeS2o (OLMo-Mix)         | 0.15       | 2.6B          | 0.2B         | 1.6M                | 1 692                   |
+| DCLM (OLMo-Mix)          | 9.50       | 49.7B         | 12.8B        | 35.1M               | 1 416                   |
+| StarCoder (OLMo-Mix)     | 2.10       | 31.5B         | 8.1B         | 23.6M               | 1 333                   |
+
+> [!NOTE]
+> The number of documents represents the total unique number of documents, not the documents used during training.
+
+> [!NOTE]
+> We only took a portion of OLMo-Mix as our unique data.
+
+### Stage 2 (4 000 steps -- 35B tokens)
+
+Data
+  - [HPLTv3](https://huggingface.co/datasets/HPLT/HPLT3.0) (filtered) Finnish
+  - FinePDFs-Edu Finnish
+  - Stack-Edu
+  - MegaMath Web-Pro
+  - FineMath 4+
+  - InfiWebMath 4+
+
+Data Splits
+| Data                     | Percentage | Unique Tokens | Total Tokens | Number of Documents | Average Document Length |
+| ------------------------ | ---------- | ------------- | ------------ | ------------------- | ----------------------- |
+| HPLT Finnish             | 40.79      | 3.4B          | 13.7B        | 3.1M                | 1 109                   |
+| FinePDFs-Edu Finnish     | 17.84      | 1.5B          | 6.0B         | 0.2M                | 7 081                   |
+| FinePDFs-Edu English     | 15.00      | 7.5B          | 5.0B         | 1.2M                | 6 485                   |
+| Stack-Edu                | 15.00      | 13.2B         | 5.0B         | 15.0M               | 880                     |
+| MegaMath Web-Pro         | 4.76       | 14.0B         | 1.6B         | 15.0M               | 937                     |
+| FineMath 4+              | 3.51       | 10.4B         | 1.2B         | 6.7M                | 1 545                   |
+| InfiWebMath 4+           | 3.09       | 9.1B          | 1.0B         | 6.3M                | 1 447                   |
+
+## Training details
+
+### Stage 1
+
+| Hyperparameter           | Value             |
+| ------------------------ | ----------------- |
+| Embedding train steps    | 1 000             |
+| Warmup steps             | 2 000             |
+| Total train steps        | 16 000            |
+| Learning rate schedule   | Warmup + constant |
+| Learning rate            | 3e-4              |
+| Weight decay             | 1e-1              |
+| Sequence length          | 4 096             |
+| Batch size               | 2 048             |
+| RoPe theta               | 500 000           |
+| Clip grad                | 1.0               |
+| Adam epsilon             | 1e-8              |
+| Adam beta_1              | 0.9               |
+| Adam beta_2              | 0.95              |
+| RMSNorm epsilon          | 1e-6              |
+| Z-loss ratio             | 1e-5              |
+| Diffusion loss ratio     | 2e-2              |
+
+### Stage 2
+
+| Hyperparameter           | Value             |
+| ------------------------ | ----------------- |
+| Decay steps              | 4 000             |
+| Total train steps        | 4 000             |
+| Learning rate schedule   | Linear decay      |
+| Initial learning rate    | 3e-4              |
+| Final learning rate      | 0                 |
+| Weight decay             | 1e-1              |
+| Sequence length          | 16 384            |
+| Batch size               | 512               |
+| RoPe theta               | 2 000 000         |
+| Clip grad                | 1.0               |
+| Adam epsilon             | 1e-8              |
+| Adam beta_1              | 0.9               |
+| Adam beta_2              | 0.95              |
+| RMSNorm epsilon          | 1e-6              |
+| Z-loss ratio             | 1e-5              |
+| Diffusion loss ratio     | 2e-2              |
+
+
+## Acknowledgements
 
 Training was conducted as a part of the [HPLT project](https://hplt-project.org/).
 
-_This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]_
+_This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]_