Update README.md

README.md

@@ -1,8 +1,66 @@
 ---
+language: 
+- en
+datasets:
+- c4
 tags:
-- t5-new-success
+- deep-narrow
+
+license: apache-2.0
 ---
 
-# Test
-Hf T5: -84.66816902160645
-MTF T5: -84.66819763183594
+T5-Efficient-XL is a checkpoint of the [T5 model architecture](https://huggingface.co/docs/transformers/model_doc/t5).
+
+The checkpoint was released with the paper **[Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://arxiv.org/abs/2109.10686)** 
+by *Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler*.
+
+In a nutshell, the paper indicates that a **DeepNarrow** model architecture is favorable for **downstream** performance compared to other model architectures 
+of similar parameter count.
+
+To quote the paper:
+
+> We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased
+> before considering any other forms of uniform scaling across other dimensions. This is largely due to
+> how much depth influences the Pareto-frontier as shown in earlier sections of the paper. Specifically, a
+> tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise,
+> a tall base model might also generally more efficient compared to a large model. We generally find
+> that, regardless of size, even if absolute performance might increase as we continue to stack layers,
+> the relative gain of Pareto-efficiency diminishes as we increase the layers, converging at 32 to 36
+> layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e.,
+> params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params,
+> FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to
+> consider.
+
+To be more precise, *model depth* is defined as the number of transformer blocks that are stacked sequentially.
+A sequence of word embeddings is therefore processed sequentially by each transformer block.
+
+## Details model architecture
+
+The *conventional* T5 architectures are 
+
+| Model | NL | dff | dmodel | dkv | NH | #Params|
+| ----| ---- | ---- | ---- | ---- | ---- | ----|
+| Tiny | 4/4 | 1024 | 256 | 32 | 4 | 16M|
+| Mini | 4/4 | 1536 | 384 | 32 | 8 | 31M|
+| Small | 6/6 | 2048 | 512 | 32 | 8 | 60M|
+| Base | 12/12 | 3072 | 768 | 64 | 12 | 220M|
+| Large | 24/24 | 4096 | 1024 | 64 | 16 | 738M|
+| XL | 24/24 | 16384 | 1024 | 128 | 32 | 3B|
+| XXL | 24/24 | 65536 | 1024 | 128 | 128 | 11B|
+
+## Pre-Training
+
+The checkpoint was pretrained on the [Colossal, Cleaned version of Common Crawl (C4)](https://huggingface.co/datasets/c4) for 524288 steps using 
+the span-based masked language modeling (MLM) objective.
+
+## Downstream Performance
+
+TODO:
+
+
+
+
+
+Pretraining Dataset: [C4](https://huggingface.co/datasets/c4)
+
+