Add files using upload-large-folder tool

README.md

@@ -1,3 +1,66 @@
+---
+# 许可证标识符，必须是 Hugging Face 支持的 license ID
+license: apache-2.0
+
+# 模型主要支持的语言
+language:
+  - en
+  - zh
+
+# 所使用的深度学习库
+library_name: transformers
+
+# 用于搜索和分类的标签
+tags:
+  - text-generation
+  - scaling-laws
+  - densing-law
+  - reference-models
+
+# 论文中用于评估的数据集
+datasets:
+  - mmlu
+  - big-bench-hard
+  - math
+  - mbpp
+  - human-eval
+
+# 论文中使用的评估指标
+metrics:
+  - loss
+  - accuracy
+
+# pipeline 标签，用于推理 API 和小部件
+pipeline_tag: text-generation
+
+# 使用 model-index 来列出仓库中包含的所有模型变体
+model-index:
+  - name: DensingLaw-ScalingModel-0.005B
+    results: []
+  - name: DensingLaw-ScalingModel-0.03B
+    results: []
+  - name: DensingLaw-ScalingModel-0.1B
+    results: []
+  - name: DensingLaw-ScalingModel-0.2B
+    results: []
+  - name: DensingLaw-ScalingModel-0.4B
+    results: []
+  - name: DensingLaw-ScalingModel-0.8B
+    results: []
+
+# 引用论文的 BibTeX
+citation: |
+  @misc{xiao2024densinglawllms,
+        title={Densing Law of LLMs}, 
+        author={Chaojun Xiao and Jie Cai and Weilin Zhao and Guoyang Zeng and Biyuan Lin and Jie Zhou and Zhi Zheng and Xu Han and Zhiyuan Liu and Maosong Sun},
+        year={2024},
+        eprint={2412.04315},
+        archivePrefix={arXiv},
+        primaryClass={cs.AI},
+        url={https://arxiv.org/abs/2412.04315}, 
+  }
+---
+
 # DensingLaw-ScalingModels
 
 This repository contains a series of reference models of varying sizes, released as part of our paper, **`Densing Law of LLMs`**. These models were trained to establish a robust scaling law, which serves as a foundational component for calculating the "density" of other Large Language Models (LLMs).
@@ -12,7 +75,7 @@ This repository contains a series of reference models of varying sizes, released
 
 ## 💡 Overview
 
-The core contribution of our paper is the concept of **LLM Density** \\(\rho\\), defined as the ratio of a model's *effective* parameter size \\(\hat{N}\\) to its *actual* parameter size \\(N\\). To accurately determine a model's effective size, we must first establish a reliable "ruler"—a scaling law that maps training compute to performance on downstream tasks.
+The core contribution of our paper is the concept of **LLM Density** ($\rho$), defined as the ratio of a model's *effective* parameter size ($/ghat{N}$) to its *actual* parameter size ($N$). To accurately determine a model's effective size, we must first establish a reliable "ruler"—a scaling law that maps training compute to performance on downstream tasks.
 
 The models in this repository serve as that "ruler". We trained a series of six models, ranging from **5 million to 800 million parameters**, on a consistent dataset. By measuring their loss on various benchmarks, we fitted a precise scaling function. This function allows us to take any other LLM, measure its performance, and infer its effective parameter size by seeing where it lands on our reference scale.
 
@@ -24,14 +87,14 @@ We trained six models with architectures designed for scaling. The detailed hype
 
 #### Table 1: Detailed Hyper-parameters of Models for Loss Estimation
 
-| Name   | # Para        | BS  | n_layer  | d     | d_ffn  | n_head  | n_kv  |
+| Name   | \# Para        | BS  | n_layer | d     | d_ffn | n_head | n_kv |
 | :----- | :------------ | :-- | :------ | :---- | :---- | :----- | :--- |
 | 0.005B (S1) | 5,247,232     | 32  | 8       | 256   | 640   | 4      | 1    |
 | 0.03B (S2)  | 31,470,080    | 32  | 12      | 512   | 1,280 | 8      | 2    |
 | 0.1B (S3)  | 106,196,736   | 64  | 18      | 768   | 1,920 | 12     | 3    |
-| 0.2B (S4)  | 245,416,960   | 128 | 24      | 1,024 | 2,560 | 16     | 2   |
-| 0.4B (S5)  | 476,852,480   | 256 | 30      | 1,280 | 3,200 | 20     | 2   |
-| 0.8B (S6)  | 828,225,024   | 512 | 36      | 1,536 | 3,840 | 24     | 3   |
+| 0.2B (S4)  | 245,416,960   | 128 | 24      | 1,024 | 2,560 | 64     | 16   |
+| 0.4B (S5)  | 476,852,480   | 256 | 30      | 1,280 | 3,200 | 64     | 20   |
+| 0.8B (S6)  | 828,225,024   | 512 | 36      | 1,536 | 3,840 | 64     | 24   |
 
 ### Training Data
 
@@ -40,10 +103,18 @@ As stated in our paper, all reference models were trained on the **training corp
 ## 🎯 Research Context: The Densing Law
 Our framework for calculating LLM density involves a two-step estimation process, which is visualized below.
 
-1.  **Loss Estimation \\( f_1 \\)**: We first establish the relationship between training compute (approximated as \\(C \approx 6ND\\) and conditional loss \\(\mathcal L\\) on downstream tasks. The models released in this repository are the data points used to fit this curve \\(\mathcal L = f_1(C)\\).
-2.  **Performance Estimation \\(f_2\\)**: We then map the relationship between this loss \\(\mathcal L\\) and a more intuitive performance metric \\(S\\), such as accuracy \\(S = f_2(\mathcal L)\\).
+1.  **Loss Estimation (\\( f_1 \\))**: We first establish the relationship between training compute (approximated as \\(C \approx 6ND\\) and conditional loss (\\(\mathcal L\\)) on downstream tasks. The models released in this repository are the data points used to fit this curve (\\(\mathcal{L} = f_1(C)\\)).
+2.  **Performance Estimation (\\(f_2\\))**: We then map the relationship between this loss (\\(\mathcal{L}\\)) and a more intuitive performance metric (\\(S\\)), such as accuracy (\\(S = f_2(\mathcal{L})\\)).
 
 By combining these, we can determine the effective compute, and therefore the effective parameter size, for any model based on its performance.
+
+Our framework for calculating LLM density involves a two-step estimation process, which is visualized below.
+
+1.  **Loss Estimation ($f_1$)**: We first establish the relationship between training compute (approximated as $C \approx 6ND$) and conditional loss ($\mathcal{L}$) on downstream tasks. The models released in this repository are the data points used to fit this curve ($\mathcal{L} = f_1(C)$).
+2.  **Performance Estimation ($f_2$)**: We then map the relationship between this loss ($\mathcal{L}$) and a more intuitive performance metric ($S$), such as accuracy ($S = f_2(\mathcal{L})$).
+
+By combining these, we can determine the effective compute, and therefore the effective parameter size, for any model based on its performance.
+
 <div align="center">
 <img src="assets/fig.jpeg" width="800"/>
 </div>