Update README.md

README.md

@@ -133,25 +133,45 @@ Remarks:
 - The following corpora was excluded from MetaFida: dgt15_sl, classlawiki_sl, tweet_sl, janes_tweet, janes_forum, janes_news
 - Serbian Wikipedia was converted from Cyrillic to Latin
 
+## Training
+
+The model was continually pre-trained on the Booster partition of [Leonardo HPC](https://www.hpc.cineca.it/systems/hardware/leonardo/) using [NVIDIA NeMo 2.0 framework](https://github.com/NVIDIA/NeMo).  The model was trained in BF16-Mixed precision using tensor parallelism across 4 GPUs, sequence parallelism, and activation recomputation. The model was trained across 32 nodes, each containing 4 A100 64GB GPUs. The parallel alignment training took approximately 4 hours and second stage took approximately 40 hours.
+
+The model was trained using a cosine learning rate scheduler with linear warmup and the following hyperparameters.
+
+**Parallel alignment**:
+- warmup steps: 150
+- minimal learning rate: 5e-6
+- maximal learning rate: 2e-5
+- constant steps: 0
+- batch size: 512 (4 million tokens)
+
+**Second stage**:
+- warmup steps: 500
+- minimal learning rate: 5e-6
+- maximal learning rate: 5e-5
+- constant steps: 100
+- batch size: 512 (4 million tokens)
+
 ## Evaluation
 
-The models were evaluated using [Slovene SuperGLUE](https://slobench.cjvt.si/leaderboard/view/3) collection of classification tasks on [SloBench](https://slobench.cjvt.si). Instruct version of the model was also evaluated on translation [from English to Slovene](https://slobench.cjvt.si/leaderboard/view/8) and [from Slovene to English](https://slobench.cjvt.si/leaderboard/view/7) Additionally, we evaluated our models on [Slovenian-LLM-Eval](https://huggingface.co/datasets/cjvt/slovenian-llm-eval).
+The models were evaluated using [Slovene SuperGLUE](https://slobench.cjvt.si/leaderboard/view/3) collection of classification tasks on [SloBench](https://slobench.cjvt.si). Instruct version of the model was also evaluated on translation [from English to Slovene](https://slobench.cjvt.si/leaderboard/view/8) and [from Slovene to English](https://slobench.cjvt.si/leaderboard/view/7). Additionally, we evaluated our models on [Slovenian-LLM-Eval](https://huggingface.co/datasets/cjvt/slovenian-llm-eval). 
 
 Code for evaluation:
 - [SloBench tasks](https://github.com/SloLama/slobench_evaluation)
 - [Slovenian-LLM-Eval](https://github.com/SloLama/slovenian-llm-eval)
 
-## Slovenian-LLM-Eval results
+### Slovenian-LLM-Eval results
 
 Comparison between GaMS models, base Gemma 2 models and SlovenianGPT (open source model for Slovene based on Mistral 7B) is shown in the figure below. All models were evaluated in 0-shot scenario.
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/652d40a78fa1fbb0aae165bb/tDyAjB2dgYXv1dLpFHikd.png)
 
-## Slobench Results
+### Slobench Results
 
 GaMS 2B, 9B and 27B models were evaluated in 3-shot scenario, except for MultiRC and translation tasks, where 0-shot was used. GaMS-9B-Instruct was evaluated in 0-shot scenarion on all tasks. We used guided decoding to ensure the correct format of the responses.
 
-### Slovene SuperGLUE
+#### Slovene SuperGLUE
 
 | Rank | Title                 | Average | BoolQ Accuracy | CB Accuracy | CB F1 Score | CB Average | COPA Accuracy | MultiRC EM | MultiRC F1a Score | MultiRC Average | RTE Accuracy | WSC Accuracy |
 |------|------------------------|---------|---------------|-------------|-------------|------------|--------------|------------|----------------|----------------|-------------|-------------|
@@ -169,7 +189,7 @@ GaMS 2B, 9B and 27B models were evaluated in 3-shot scenario, except for MultiRC
 | 15   | GaMS-1B-Chat          | 0.4570  | 0.8000        | 0.4880      | 0.3023      | 0.3951     | 0.4840       | 0.1081     | 0.2428         | 0.1755         | 0.5172      | 0.3692      |
 
 
-### English to Slovene translation (first 11 models on the benchmark)
+#### English to Slovene translation (first 11 models on the benchmark)
 
 | Rank | Title                           | BERT score | BLEU (avg) | METEOR (avg) | CHRF (avg) | BLEU (corpus) | CHRF (corpus) |
 |------|---------------------------------|------------|------------|--------------|------------|---------------|---------------|
@@ -185,7 +205,7 @@ GaMS 2B, 9B and 27B models were evaluated in 3-shot scenario, except for MultiRC
 | 9    | META LLAMA 3.1 405B             | 0.8705     | 0.2637     | 0.5497       | 0.5930     | 0.3063        | 0.5930        |
 | 11   | RSDO-DS4-NMT 1.2                | 0.8698     | 0.2781     | 0.5602       | 0.5970     | 0.3177        | 0.5970        |
 
-### Slovene to English translation (first 10 models on the benchmark)
+#### Slovene to English translation (first 10 models on the benchmark)
 
 | Rank | Title               | BERT score | BLEU (avg) | METEOR (avg) | CHRF (avg) | BLEU (corpus) | CHRF (corpus) |
 |------|---------------------|------------|------------|--------------|------------|---------------|---------------|