Update README.md

README.md

@@ -1,3 +1,39 @@
 ---
 license: mit
+datasets:
+- p208p2002/wudao
+language:
+- zh
 ---
+# Chinese TinyLlama
+
+A demo project that pretrains a tinyllama on Chinese corpora, with minimal modification to the huggingface transformers code. It serves as a use case to demonstrate how to use the huggingface version [TinyLlama](https://github.com/whyNLP/tinyllama) to pretrain a model on a large corpus.
+
+See the [Github Repo](https://github.com/whyNLP/tinyllama-zh) for more details.
+
+## Usage
+
+```python
+# Load model directly
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("whynlp/tinyllama-zh", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("whynlp/tinyllama-zh")
+```
+
+## Model Details
+
+### Model Description
+
+This model is trained on [WuDaoCorpora Text](https://www.scidb.cn/en/detail?dataSetId=c6a3fe684227415a9db8e21bac4a15ab). The dataset contains about 45B tokens and the model is trained for 2 epochs. The training takes about 6 days on 8 A100 GPUs.
+
+The model uses the `THUDM/chatglm3-6b` tokenizer from huggingface.
+
+- **Model type:** Llama
+- **Language(s) (NLP):** Chinese
+- **License:** MIT
+- **Finetuned from model [optional]:** TinyLlama-2.5T checkpoint
+
+## Uses
+
+The model does not perform very well (The CMMLU result is slightly above 25). For better performance, one may use a better corpus (e.g. [wanjuan](https://opendatalab.org.cn/OpenDataLab/WanJuan1_dot_0)). Again, this project only serves as a demonstration of how to pretrain a TinyLlama on a large corpus.