Update README.md

README.md

@@ -1,5 +1,5 @@
-RoBERTurk is pretrained on Oscar Turkish Split (27GB) and a small chunk of C4 Turkish Split (1GB) with sentencepiece BPE tokenizer that is trained on randomly selected 30M sentences from the training data which is composed of 90M sentences. The training data in total contains 5.3B tokens and the vocabulary size is 50K.
-
+RoBERTurk is pretrained on Oscar Turkish Split (27GB) and a small chunk of C4 Turkish Split (1GB) with sentencepiece BPE tokenizer that is trained on randomly selected 30M sentences from the training data, which is composed of 90M sentences. The training data in total contains 5.3B tokens and the vocabulary size is 50K.
+The learning rate is warmed up to the peak value of 1e-5 for the first 10K updates and linearly decayed at $0.01$ rate. The model is pretrained for maximum 600K updates only with sequences of at most T=256 length. 
 
 ## Tokenizer
 
@@ -26,5 +26,6 @@ Additional TODOs are (although some of them can take some time and I may include
 
  -  Using Zemberek as an alternative tokenizer
  -  Adjusting masking algorithm to be able to mask morphologies besides only complete words
- -  Preferably pretraining BPE on the complete training data
+ -  Preferably pretraining BPE on the whole training data
  -  Pretraining with 512 max sequence length + more data
+