Upload README.md

README.md

@@ -1,5 +1,5 @@
 ---
-language: 
+language:
 - en
 datasets:
 - c4
@@ -13,11 +13,11 @@ license: apache-2.0
 # T5-Efficient-XL (Deep-Narrow version)
 
 T5-Efficient-XL is a variation of [Google's original T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) following the [T5 model architecture](https://huggingface.co/docs/transformers/model_doc/t5).
-It is a *pretrained-only* checkpoint and was released with the 
-paper **[Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://arxiv.org/abs/2109.10686)** 
+It is a *pretrained-only* checkpoint and was released with the
+paper **[Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://arxiv.org/abs/2109.10686)**
 by *Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler*.
 
-In a nutshell, the paper indicates that a **Deep-Narrow** model architecture is favorable for **downstream** performance compared to other model architectures 
+In a nutshell, the paper indicates that a **Deep-Narrow** model architecture is favorable for **downstream** performance compared to other model architectures
 of similar parameter count.
 
 To quote the paper:
@@ -39,11 +39,11 @@ A sequence of word embeddings is therefore processed sequentially by each transf
 
 ## Details model architecture
 
-This model checkpoint - **t5-efficient-xl** - is of model type **XL** with **no** variations.
-It has **2852** million parameters and thus requires **11406 MB** of memory in full precision (*fp32*) 
- or  **5703 MB** of memory in half precision (*fp16* or *bf16*).
+This model checkpoint - **t5-efficient-xl** - is of model type **Xl** with no variations.
+It has **2851.66** million parameters and thus requires *ca.* **11406.62 MB** of memory in full precision (*fp32*)
+ or  **5703.31 MB** of memory in half precision (*fp16* or *bf16*).
 
-The *conventional* T5 architectures are summarized as follows:
+A summary of the *original* T5 model architectures can be seen here:
 
 | Model | nl (el/dl) | ff | dm | kv | nh | #Params|
 | ----| ---- | ---- | ---- | ---- | ---- | ----|
@@ -52,24 +52,24 @@ The *conventional* T5 architectures are summarized as follows:
 | Small | 6/6 | 2048 | 512 | 32 | 8 | 60M|
 | Base | 12/12 | 3072 | 768 | 64 | 12 | 220M|
 | Large | 24/24 | 4096 | 1024 | 64 | 16 | 738M|
-| **XL** | **24/24** | **16384** | **1024** | **128** | **32** | **3B**|
-| XXL | 24/24 | 65536 | 1024 | 128 | 128 | 11B|
+| Xl | 24/24 | 16384 | 1024 | 128 | 32 | 3B|
+| XXl | 24/24 | 65536 | 1024 | 128 | 128 | 11B|
 
 whereas the following abbreviations are used:
 
 | Abbreviation | Definition |
 | ----| ---- |
-| nl | Number of transformer blocks (depth) | 
-| dm | Dimension of embedding vector (output vector of transformers block) | 
-| kv | Dimension of key/value projection matrix | 
-| nh | Number of attention heads | 
+| nl | Number of transformer blocks (depth) |
+| dm | Dimension of embedding vector (output vector of transformers block) |
+| kv | Dimension of key/value projection matrix |
+| nh | Number of attention heads |
 | ff | Dimension of intermediate vector within transformer block (size of feed-forward projection matrix) | 
 | el | Number of transformer blocks in the encoder (encoder depth) | 
 | dl | Number of transformer blocks in the decoder (decoder depth) | 
 | sh | Signifies that attention heads are shared | 
 | skv | Signifies that key-values projection matrices are tied | 
 
-If a model checkpoint has no specific, *el* or *dl* than both the number of encoder- and decoder layers correspond no *nl*.
+If a model checkpoint has no specific, *el* or *dl* than both the number of encoder- and decoder layers correspond to *nl*.
 
 ## Pre-Training