Create README.md

README.md

@@ -0,0 +1,31 @@
+---
+license: mit
+datasets:
+- wikipedia
+- bookcorpus
+language:
+- en
+metrics:
+- glue
+library_name: transformers
+---
+
+This is our reproduction using the official HuggingFace `roberta` architecture with a medium size. On the architecture side, RoBERTa is exactly the same as BERT except for its larger vocabulary size.
+
+According to Google's [BERT releases](https://huggingface.co/google/bert_uncased_L-8_H-512_A-8) and [BERT-Medium](https://huggingface.co/google/bert_uncased_L-8_H-512_A-8/blob/main/config.json), a medium sized model should have a config of Layer=8, Hidden=512, #AttnHeads=8, and IntermediateSize=2048. We follow this config to pre-train a RoBERTa-base model for reproduction.
+
+We use the same datasets as BERT (English Wikipedia and Book Corpus) to pre-train for 30k steps with a batch size of 8,192. I also released the reproduction of this dataset [on HuggingFace](https://huggingface.co/datasets/JackBAI/bert_pretrain_datasets).
+
+We utilized DeepSpeed ZeRO-2 for performance optimization.
+
+Other training configuration: 
+
+| Parameter            | Value     |
+|----------------------|-----------|
+| WARMUP_STEPS         | 1800      |
+| LR_DECAY             | linear    |
+| ADAM_EPS             | 1e-6      |
+| ADAM_BETA1           | 0.9       |
+| ADAM_BETA2           | 0.98      |
+| ADAM_WEIGHT_DECAY    | 0.01      |
+| PEAK_LR              | 1e-3      |