Update README.md

README.md

@@ -3,4 +3,56 @@ license: apache-2.0
 datasets:
 - bigcode/the-stack
 - HuggingFaceFW/fineweb
----
+---
+
+
+# Model Details
+
+The TinyCodeLM family of tiny language models (LMs) is a collection of pretrained and instruction tuned generative code models in 150M and 400M sizes. These models are pretrained on a mixture of open-source web text and Python code. The instruction tuned TinyCodeLM models are optimized for Python code synthesis, and are trained on [synthetic edit sequence data generated with the LintSeq algorithm](https://arxiv.org/abs/2410.02749).
+
+Despite being trained on only 72 billion tokens of text, the models outperform many of the available open source Python code synthesis models on HumanEval and MBPP. The TinyCodeLM-LintSeqInstruct models are state-of-the-art on Python synthesis for their size.
+
+**Model Developers** Ulyana Piterbarg, Lerrel Pinto, Rob Fergus (NYU)
+
+**Variations** TinyCodeLM comes in two sizes (150M and 400M parameters) in pretrained and edit sequence instruction tuned variants.
+
+**Input** Text only.
+
+**Output** Models generate text and code. Instruction tuned models generate code via sequences of "diffs".
+
+**Model Architecture** TinyCodeLMs are autoregressive language models with architectures that mimic the two smallest versions of GPT-2 (Radford et al., 2019), while integrating the transformer architecture changes of the OLMo models. 
+
+**Instruction Tuning Data** TinyCodeLMs are instruction tuned on paired instruction and Python edit sequence data. These edit sequences are generated with the LintSeq algorithm over a source dataset of paired instruction and Python programs drawn from the Magicoder and StarCoder2 OSS-Instruct datasets (Wei et al., 2024).
+
+# Benchmarks 
+
+**Pretrained (Temperature 0)**
+
+|**Benchmark**|**TinyCodeLM 150M** |**TinyCodeLM 400M** |
+| :--------------------- | -----------------: | :-----------------: |
+|  HumanEval, pass@1 |    6.1   |      6.7  |
+|  MBPP(+), pass@1 |        5.4   |         6.8  |
+
+
+**Edit Sequence / Instruction Tuned (Temperature-Tuned)**
+
+|**Benchmark** |**TinyCodeLM 150M** |**TinyCodeLM 400M** |
+| :----------- | -----------------: | -----------------: |
+|  HumanEval, pass@1   | 12.8   | 13.4  |
+|  HumanEval, pass@10 |    20.6   |  20.9  |
+|  MBPP(+), pass@1       | 13.6   |   24.4  |
+|  MBPP(+), pass@10     |  24.4   |    29.9  |
+
+
+# Citation
+
+```
+@misc{piterbarg2024training,
+      title={Training Language Models on Synthetic Edit Sequences Improves Code Synthesis}, 
+      author={Ulyana Piterbarg and Lerrel Pinto and Rob Fergus},
+      year={2024},
+      eprint={2410.02749},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```