Update README.md

README.md

@@ -14,6 +14,11 @@ This modelcard aims to be a base template for new models. It has been generated
 
 <!-- Provide a longer summary of what this model is. -->
 
+- **Developed by:** xTimeCrystal
+- **Model type:** Softmax self-attention decoder-only transformer
+- **Language(s) (NLP):** English, Chinese, Python
+- **License:** Apache 2.0
+
 This model uses SOTA innovative techniques to train a 200M decoder-only transformer language model using only 10B tokens and on a single RTX 5090 GPU in just 1 day. As shown in the examples below, the model is highly competent in diverse styles and is capable of generating text on various topics, ranging from pure recall to article writing.
 
 The main techniques used were:
@@ -21,18 +26,19 @@ The main techniques used were:
 
 - Aggressive data filtering: by selecting a carefully curated set of educational content and building upon the data efficiency of Muon, we were able to significantly improves capabilities in resource-constrained environments.
 
+- Efficient data bin-packing: as self attention relies heavily on attention sinks, all the sequences started with the start token ('\<s\>') and were truncated at 2048 tokens. However, this led to inefficiencies as over 70% of the processed data was padding. Thus, to alleviate this issue, we used a simple bin packing algorithm that tries to concatenate sequences to all have lengths close to 2048. After this operation, all the sequences had less than 5% padding.
+
 - Float8 pretraining: This model was pretrained using bf16 master weights, fp8 (e4m3) casting with bf16 accumulation, and full bf16 backward. However, quantizing the attention mechanism significantly hurt the loss, so it was kept in bf16. This was found to match the performance of full bf16 training with significantly reduced (\~30% decrease) VRAM usage and much higher (\~20% increase) throughput.
 
 - ReLU^2 activation: This ultra-sparse activation function outperformed SwiGLU ([\[1\]](https://arxiv.org/abs/2109.08668v2), [\[2\]](https://arxiv.org/abs/2402.03804)) while only requiring 2 matmuls, slightly improving throughput.
 
 - Full attention: when pretraining small models, every layer is precious. Thus, we use full attention (no SWA, no GQA) for all layers in the model.
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/66a767dcbe4c3c2683495a8b/L7AuCdoEGrEVprBKIbks2.png)
+- QK Norm without scalars: this enhanced stability as the additional scalars caused loss spikes and massive attention activations.
 
-- **Developed by:** xTimeCrystal
-- **Model type:** Softmax self-attention decoder-only transformer
-- **Language(s) (NLP):** English, Chinese, Python
-- **License:** Apache 2.0
+Overall, these techniques allowed the model to be losslessly trained with a massive batch size of 64 x 2048 tokens and completely spike-free for 110k steps (14B tokens):
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/66a767dcbe4c3c2683495a8b/L7AuCdoEGrEVprBKIbks2.png)
 
 ### Model Sources [optional]
 
@@ -305,20 +311,6 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 [More Information Needed]
 
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
 ## Model Card Contact
 
 [More Information Needed]