Update README.md

README.md

@@ -6,11 +6,37 @@ pipeline_tag: text-generation
 tags:
 - PyTorch
 - Transformers
+- gpt3
+- gpt2
+- Deepspeed
+- Megatron
 thumbnail: "https://github.com/sberbank-ai/mgpt"
 ---
 
 # Multilingual GPT model
 
+We introduce family of autoregressive GPT-like models with 1.3 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus. 
+
+We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism, [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron]() frameworks allows us to effectively parallelize the training and inference steps. Resulting models show performance on par with the recently released [XGLM](https://arxiv.org/pdf/2112.10668.pdf) models at the same time covering more languages and enhance NLP possibilities for low resource languages. 
+
+## Code
+The source code for the mGPT XL model is available on [Github](https://github.com/sberbank-ai/mgpt)
+
+## Paper
+[Arxiv preprint](https://arxiv.org/user)
+
+Cite us:
+```{
+bibtex
+}
+```
+
+## Languages
+
+## Training Data Statistics
+
+
+## Details
 Model was trained with sequence length 1024 using transformers lib by [SberDevices](https://sberdevices.ru/) team on 80B tokens for 3 epochs. After that model was finetuned 1 epoch with sequence length 2048. 
 
 Total training time was around n days on n GPUs for n context and few days on n GPUs for n context.