Create README.md

README.md

@@ -0,0 +1,44 @@
+
+# mGPT
+
+mGPT is pre-trained on the [mC4 dataset](https://huggingface.co/datasets/mc4) using a causal language modeling objective. It was introduced in this [paper](https://arxiv.org/abs/2110.06609) and first released on this page.
+
+## Model description
+
+mGPT is a Transformer-based model which pre-trained on massive multilingual data covering over 101 languages. Similar to GPT-2, It was pre-trained on the raw texts only, with no human labeling. We use the same tokenization and vocabulary as the [mT5 model](https://huggingface.co/google/mt5-base).
+
+## Intended uses
+
+You can use the raw model for text generation or using prompts for adapting it to a downstream task.
+
+## How to use
+
+You can use this model directly with a pipeline for text generation.  Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import MT5Tokenizer, GPT2LMHeadModel, TextGenerationPipeline
+
+tokenizer = MT5Tokenizer.from_pretrained("THUMT/mGPT")
+model = GPT2LMHeadModel.from_pretrained("THUMT/mGPT")
+
+pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
+text = "Replace me by any text you'd like."
+text = pipeline(text, do_sample=True,  max_length=1024)[0]["generated_text"]
+```
+
+## Preprocessing
+
+The texts are tokenized using `sentencepiece` and a vocabulary size of 250,100. The inputs are sequences of 1,024 consecutive tokens. We use `<extra_id_0>` to separate lines in a document.
+
+## BibTeX entry and citation info
+
+```bibtex
+@misc{tan2021msp,
+    title={MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators},
+    author={Zhixing Tan and Xiangwen Zhang and Shuo Wang and Yang Liu},
+    year={2021},
+    eprint={2110.06609},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```