| --- |
| tags: |
| - pytorch |
| - causal-lm |
| metrics: |
| - accuracy |
| language: |
| - sl |
| license: apache-2.0 |
| --- |
| |
| # GPT-sl-base |
|
|
| This model is a Slovene GPT model, based on the [bigscience workshop](https://github.com/bigscience-workshop/Megatron-DeepSpeed) fork of the Megatron. GPT-sl-base was trained on large Slovene corpora: Gigafida, KAS, slWaC, and MaCoCu. |
|
|
| ## Model architecture |
| GPT-sl-base has about 110 million parameters. It consists of 12 transformer layers with a dimension of 768. It has 16 attention heads and can process sequences up to 1024 tokens in length. |
| The tokenizer was trained on a smaller subset of the corpora, and has the vocabulary of 60k tokens. |
|
|
| ## Training |
| The model was trained for about 20 epochs, a total of 390k steps or 102B tokens seen during training. |
|
|
| | Step | Validation Perplexity | |
| |:------:|:---------------------:| |
| | 50000 | 26.801 | |
| | 100000 | 25.574 | |
| | 150000 | 24.773 | |
| | 200000 | 24.099 | |
| | 250000 | 23.336 | |
| | 300000 | 22.607 | |
| | 350000 | 22.329 | |
| | 390000 | 22.293 | |
|
|
