Update README.md

README.md

@@ -1,3 +1,85 @@
 ---
 license: mit
+language:
+- en
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- llm
+- dataset combination
+- Pretraining
 ---
+
+# SlimPajama-DC
+
+<center><img src="assets/SlimPajama-DC-logo.png" alt="SlimPajama-DC logo" width="200"/></center>
+
+
+**SlimPajama-DC** is a set of 1.3B parameter language models, distinctively trained on the different combinations of 330B subsets of SlimPajama dataset. 
+
+| Details of Dataset Combinations for Different Models       |
+|------------------------------------------------|
+
+<center><img src="assets/data_combination.png" alt="details of dataset combinations" width="800"/></center>
+
+
+Despite being trained on a smaller amount of 330B tokens compared to TinyLlama and Olmo's 3 trillion, SlimPajama-DC surpasses TinyLlama and Olmo in some challenging English tasks. 
+
+| Our tests comprise: (1) AI2 Reasoning Challenge (25-shot); (2) HellaSwag (10-shot); (3) MMLU (5-shot); (4) TruthfulQA (0-shot) |
+|------------------------------------------------|
+<center><img src="assets/res1.png" alt="results" width="880"/></center>
+
+‡ represents the RefinedWeb CC.
+
+| Performance on More Benchmarks             |
+|------------------------------------------------|
+<center><img src="assets/res2.png" alt="results" width="830"/></center>
+
+ARC easy and ARC challenge are evaluated using 25-shot. All other evaluation benchmarks are tested on 0-shot. * represents the results are averaged across multiple sub-items inside each benchmark dataset.
+
+
+# Dataset
+
+Our full dataset is available at [SlimPajama-627B-DC](https://huggingface.co/datasets/MBZUAI-LLM/SlimPajama-627B-DC).
+
+
+# Model Usage
+
+To load a specific checkpoint, use the revision argument as shown below, for example, `SlimPajama-DC-6`. All the revisions can be seen from the branch dropdown in the "Files and versions" tab. If no revision argument is provided, it will load the default checkpoint `SlimPajama-DC-6`.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained(
+    "MBZUAI-LLM/SlimPajama-DC",
+    revision="SlimPajama-DC-6",
+    trust_remote_code=True
+)
+model = AutoModelForCausalLM.from_pretrained(
+    "MBZUAI-LLM/SlimPajama-DC",
+    revision="SlimPajama-DC-6",
+    trust_remote_code=True
+)
+
+prompt = 'int add(int x, int y) {'
+
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+gen_tokens = model.generate(input_ids, do_sample=True, max_length=400)
+
+print("-"*20 + "Output for model"  + 20 * '-')
+print(tokenizer.batch_decode(gen_tokens)[0])
+```
+
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@article{shen2023slimpajama,
+  title={Slimpajama-dc: Understanding data combinations for llm training},
+  author={Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, Eric Xing},
+  journal={arXiv preprint arXiv:2309.10818},
+  year={2023}
+}
+```