NB-ROBERTA Training Code
This is the current training code for the planned nb-roberta models.
We are currently planning to run the following experiments:
| Name
|
nb-roberta-base-old (C)
|
| Corpus
|
NbAiLab/nb_bert
|
| Pod size
|
v4-64
|
| Batch size
|
62*4*8 = 1984 = 2k
|
| Learning rate
|
3e-4 (RoBERTa article is using 6e-4 and bs=8k)
|
| Number of steps
|
250k
|
| Name
|
nb-roberta-base-ext (B)
|
| Corpus
|
NbAiLab/nbailab_extended
|
| Pod size
|
v4-64
|
| Batch size
|
62*4*8 = 1984 = 2k
|
| Learning rate
|
3e-4 (RoBERTa article is using 6e-4 and bs=8k)
|
| Number of steps
|
250k
|
| Name
|
nb-roberta-large-ext
|
| Corpus
|
NbAiLab/nbailab_extended
|
| Pod size
|
v4-64
|
| Batch size
|
32*4*8 = 2024 = 1k
|
| Learning rate
|
2-e4 (RoBERTa article is using 4e-4 and bs=8k)
|
| Number of steps
|
500k
|
| Name
|
nb-roberta-base-scandi
|
| Corpus
|
NbAiLab/scandinavian
|
| Pod size
|
v4-64
|
| Batch size
|
62*4*8 = 1984 = 2k
|
| Learning rate
|
3e-4 (RoBERTa article is using 6e-4 and bs=8k)
|
| Number of steps
|
250k
|
| Name
|
nb-roberta-large-scandi
|
| Corpus
|
NbAiLab/scandinavian
|
| Pod size
|
v4-64
|
| Batch size
|
32*4*8 = 1024 = 1k
|
| Learning rate
|
2-e4 (RoBERTa article is using 4e-4 and bs=8k)
|
| Number of steps
|
500k
|
Calculations
Some basic that we used when estimating the number of training steps:
- The Scandinavic Corpus is 85GB
- The Scandinavic Corpus contains 13B words
- With a conversion factor of 2.3, this is estimated to around 30B tokens
- 30B tokens / (512 seq length * 3000 batch size) = 20.000 steps
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="NbAiLab/nb-roberta-tpu")