Update README.md

README.md

@@ -12,7 +12,7 @@ license: llama2
 Sparse computation is increasingly recognized as an important direction in enhancing the computational efficiency of large language models (LLMs). 
 Previous work has shown that models after relufication are characterised by sparse activation, which naturally introduces a new problem: Which activation function is optimal for sparse LLMs? Although previous works on activation function selection have focused on the performance of LLMs, we argue that the efficiency of sparse computation should also be considered so that the LLMs can proceed with efficient inference while preserving performance.
 
-To answer this question, we pretrain 4 LLMs with different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$^2$ to do more comprehensive experiments.
+To answer this question, we pretrain 4 LLMs with different activation functions, including ReLU, SwiGLU, ReGLU, and Squared ReLU to do more comprehensive experiments.
 
 ### Dataset
 
@@ -22,9 +22,8 @@ We pretrain the model on  100 billion tokens, including:
 * SlimPajama
 
 
-### Training Details
+### Training Hyper-parameters
 
-We jointly optimize the model on the conventional language modeling objective and the knowledge distillation objective. The knowledge distillation objective is to minimize the KL divergence between the teacher model and the student model. The teacher model is the original LLM, and the student model is the ReLU-activated version. Since the size of the fine-tuning data is relatively small, we introduce the knowledge distillation objective to avoid overfitting and enhance the generalization ability of the model, which can be also seen as a technique of label smoothing.
 
 | Parameter             | Value       |
 |-----------------------|-------------|