Update README.md

README.md

@@ -11,13 +11,62 @@ pipeline_tag: text-generation
 ## Introduction
 We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasing **a collection of checkpoints during the training process** as following:
 
-| Model | Pre-trained | Mid-trained | Merged (i.e., [WSM](https://arxiv.org/abs/2507.17634)) |
-| --- | --- | --- | --- |
-| Ling-3.0-tiny | [**Ling-3.0-tiny-base-30T**](https://huggingface.co/inclusionAI/Ling-3.0-tiny-base-30T) | [**Ling-3.0-tiny-base-midtrain**](https://huggingface.co/inclusionAI/Ling-3.0-tiny-base-midtrain) | [**Ling-3.0-tiny-base**](https://huggingface.co/inclusionAI/Ling-3.0-tiny-base) |
-| Ling-3.0-flash | [**Ling-3.0-flash-base-30T**](https://huggingface.co/inclusionAI/Ling-3.0-flash-base-30T) | [**Ling-3.0-flash-base-midtrain**](https://huggingface.co/inclusionAI/Ling-3.0-flash-base-midtrain) | [**Ling-3.0-flash-base**](https://huggingface.co/inclusionAI/Ling-3.0-flash-base) |
+<div style="width: 100%; overflow-x: auto;">
+  <table
+    align="center"
+    style="margin: 0 auto; text-align: center; border-collapse: collapse;"
+  >
+    <thead>
+      <tr>
+        <th align="center">Model</th>
+        <th align="center">Pre-trained</th>
+        <th align="center">Mid-trained</th>
+        <th align="center">
+          Merged (i.e., <a href="https://arxiv.org/abs/2507.17634">WSM</a>)
+        </th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td align="center">Ling-3.0-tiny</td>
+        <td align="center">
+          <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny-base-30T">
+            <strong>Ling-3.0-tiny-base-30T</strong>
+          </a>
+        </td>
+        <td align="center">
+          <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny-base-midtrain">
+            <strong>Ling-3.0-tiny-base-midtrain</strong>
+          </a>
+        </td>
+        <td align="center">
+          <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny-base">
+            <strong>Ling-3.0-tiny-base</strong>
+          </a>
+        </td>
+      </tr>
+      <tr>
+        <td align="center">Ling-3.0-flash</td>
+        <td align="center">
+          <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash-base-30T">
+            <strong>Ling-3.0-flash-base-30T</strong>
+          </a>
+        </td>
+        <td align="center">
+          <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash-base-midtrain">
+            <strong>Ling-3.0-flash-base-midtrain</strong>
+          </a>
+        </td>
+        <td align="center">
+          <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash-base">
+            <strong>Ling-3.0-flash-base</strong>
+          </a>
+        </td>
+      </tr>
+    </tbody>
+  </table>
+</div>
 
-
-  
 These checkpoints correspond to different stages of the training process:
 
 + **Pretrained checkpoint** have completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training.
@@ -28,33 +77,368 @@ These checkpoints are released to support continued pretraining, fine-tuning, an
 
 ## Model Overview
 ### Key features
-+ **Highly sparse (1/64) MoE architecture:** 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
-+ **Native hybrid linear attention:** Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
-+ **[Warmup-Stable and Merge](https://arxiv.org/abs/2507.17634):** We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
-+ **Scale Seamlessly:** Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
-
-| Model Type | Base (final checkpoint of pre-training) |
-| --- | --- |
-| Architecture | Hybrid-linear MoE |
-| Parameter Scale | Totoal 124B, Activated 5.1B (Non-emb) |
-| Transformer Layers | 35 KDA + 7 Gated MLA (5:1) |
-| Number of Dense Layers | 2 |
-| Number of Routed Experts | 512 |
-| Number of Shared Experts | 1 |
-| Number of Activated Experts | 8 |
-| Attention Heads | 32 |
-| Hidden Size | 2560 |
-| Expert Intermediate Size | 768 |
-| Dense Intermediate Size | 6144 |
-| Vocabulary Size | 157,184 |
-
-
-<img src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787121194497-5a39e2a5-8f80-4ed8-81df-3304577bf317.png" width="952" title="" crop="0,0,1,1" id="u730ab9e7" class="ne-image">
++ **Highly sparse (1/64) MoE architecture**: 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
++ **Native hybrid linear attention**: Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
++ [**Warmup-Stable and Merge**](https://arxiv.org/abs/2507.17634): We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
++ **Scale Seamlessly**: Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
+
+<div
+  style="
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    gap: 32px;
+    width: 100%;
+    max-width: 1200px;
+    margin: 0 auto;
+    flex-wrap: wrap;
+  "
+>
+  <div style="flex: 1 1 420px; max-width: 560px; overflow-x: auto;">
+    <table style="width: 100%; text-align: center; border-collapse: collapse;">
+      <thead>
+        <tr>
+          <th align="center">Model Type</th>
+          <th align="center">Base (final checkpoint of pre-training)</th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr><td align="center">Architecture</td><td align="center">Hybrid-linear MoE</td></tr>
+        <tr><td align="center">Parameter Scale</td><td align="center">Totoal 124B, Activated 5.1B (Non-emb)</td></tr>
+        <tr><td align="center">Transformer Layers</td><td align="center">35 KDA + 7 Gated MLA (5:1)</td></tr>
+        <tr><td align="center">Number of Dense Layers</td><td align="center">2</td></tr>
+        <tr><td align="center">Number of Routed Experts</td><td align="center">512</td></tr>
+        <tr><td align="center">Number of Shared Experts</td><td align="center">1</td></tr>
+        <tr><td align="center">Number of Activated Experts</td><td align="center">8</td></tr>
+        <tr><td align="center">Attention Heads</td><td align="center">32</td></tr>
+        <tr><td align="center">Hidden Size</td><td align="center">2560</td></tr>
+        <tr><td align="center">Expert Intermediate Size</td><td align="center">768</td></tr>
+        <tr><td align="center">Dense Intermediate Size</td><td align="center">6144</td></tr>
+        <tr><td align="center">Vocabulary Size</td><td align="center">157,184</td></tr>
+      </tbody>
+    </table>
+  </div>
+
+  <div style="flex: 0 1 700px; max-width: 800px;">
+    <img
+      src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787121194497-5a39e2a5-8f80-4ed8-81df-3304577bf317.png"
+      width="520"
+      style="
+        display: block;
+        width: 100%;
+        max-width: 800px;
+        height: auto;
+        margin: 0 auto;
+        border-radius: 10px;
+      "
+      alt=""
+    >
+  </div>
+</div>
+
 
 ## Base Model Evaluation
 To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including mathematics, coding, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., `Ling-3.0-flash-base`, is compared below:
 
-<img src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787147691544-c61cbb0e-568c-4d5e-a2d9-12a709f003d0.png" width="1431.5" title="" crop="0,0,1,1" id="uf61ea7a7" class="ne-image">
+<style>
+  .benchmark-table {
+    border-collapse: collapse;
+    margin: 0 auto;
+  }
+
+  .benchmark-table th,
+  .benchmark-table td {
+    padding: 0.55rem 0.8rem;
+    text-align: center !important;
+    vertical-align: middle;
+  }
+
+  .benchmark-table .ling {
+    background-color: #e7f6ff;
+  }
+
+  .benchmark-table thead .ling {
+    color: #1677ff;
+  }
+
+  .benchmark-table thead tr:first-child .ling {
+    border-radius: 28px 28px 0 0;
+  }
+
+  .benchmark-table tbody tr:last-child .ling {
+    border-radius: 0 0 28px 28px;
+  }
+
+  .benchmark-table .section-start > * {
+    border-top: 2px solid #d0d7de;
+  }
+
+  .benchmark-table ins {
+    text-decoration-thickness: 1px;
+    text-underline-offset: 0.12em;
+  }
+
+  .benchmark-table .table-footnote {
+    padding-top: 0.85rem;
+    text-align: left !important;
+    color: #57606a;
+    font-size: 0.9em;
+  }
+</style>
+
+<div class="benchmark-wrap">
+  <table class="benchmark-table" align="center">
+    <thead>
+      <tr>
+        <th rowspan="2" scope="col" align="center">Domain</th>
+        <th rowspan="2" scope="col" align="center">Benchmark</th>
+        <th rowspan="2" scope="col" align="center">Shot Config</th>
+        <th class="ling ling-top" scope="col" align="center">Ling-3.0-flash-base</th>
+        <th scope="col" align="center">Ling-2.5-flash-base</th>
+        <th scope="col" align="center">NVIDIA-Nemotron-3-Super-base</th>
+        <th scope="col" align="center">Step-3.5-Flash-base</th>
+        <th scope="col" align="center">Hy3-preview-base</th>
+        <th scope="col" align="center">MiMo-V2.5-base</th>
+      </tr>
+      <tr>
+        <th class="ling" align="center">124B A5.1B</th>
+        <th align="center">104B A7.4B</th>
+        <th align="center">120B A12B</th>
+        <th align="center">196B A11B</th>
+        <th align="center">295B A21B</th>
+        <th align="center">310B A15B</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td class="domain" rowspan="6" align="center">Knowledge</td>
+        <td class="benchmark" align="center">CCPM<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>90.00</strong></td>
+        <td align="center">78.38</td>
+        <td align="center">80.77</td>
+        <td align="center">83.90</td>
+        <td align="center"><u>88.64</u></td>
+        <td align="center">88.05</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">ARC-C<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>96.27</strong></td>
+        <td align="center"><u>95.59</u></td>
+        <td align="center"><strong>96.27</strong></td>
+        <td align="center">92.88</td>
+        <td align="center">94.58</td>
+        <td align="center"><u>95.59</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">AGIEval<sub>(Acc)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><u>77.50</u></td>
+        <td align="center">72.39</td>
+        <td align="center">67.08</td>
+        <td align="center">74.27</td>
+        <td align="center"><strong>77.61</strong></td>
+        <td align="center">71.84</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">SimpleQA-Verified<sub>(Acc)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><u>26.90</u></td>
+        <td align="center">15.90</td>
+        <td align="center">21.20</td>
+        <td align="center"><strong>27.80</strong></td>
+        <td align="center">24.20</td>
+        <td align="center">19.10</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MMLU-Pro <sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>67.74</strong></td>
+        <td align="center">61.36</td>
+        <td align="center">63.89</td>
+        <td align="center">63.65</td>
+        <td align="center"><u>66.44</u></td>
+        <td align="center">64.90</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CEval<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><u>89.76</u></td>
+        <td align="center">87.86</td>
+        <td align="center">78.12</td>
+        <td align="center">89.36</td>
+        <td align="center"><strong>90.74</strong></td>
+        <td align="center">88.60</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="7" align="center">Code</td>
+        <td class="benchmark" align="center">HumanEval-Plus <sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>81.10</strong></td>
+        <td align="center"><u>80.49</u></td>
+        <td align="center">78.05</td>
+        <td align="center">78.05</td>
+        <td align="center">79.27</td>
+        <td align="center">76.22</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CruxEval<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">1-shot</td>
+        <td class="ling" align="center"><u>78.50</u></td>
+        <td align="center">76.50</td>
+        <td align="center">72.44</td>
+        <td align="center">71.69</td>
+        <td align="center"><strong>82.94</strong></td>
+        <td align="center">70.06</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MultiPL-E<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">1-shot</td>
+        <td class="ling" align="center"><strong>70.79</strong></td>
+        <td align="center"><u>68.07</u></td>
+        <td align="center">57.28</td>
+        <td align="center">63.21</td>
+        <td align="center">45.77</td>
+        <td align="center">53.57</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">LiveCodeBench<sup>1</sup> <sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">1-shot</td>
+        <td class="ling" align="center"><strong>40.09</strong></td>
+        <td align="center">33.04</td>
+        <td align="center"><u>38.99</u></td>
+        <td align="center">35.68</td>
+        <td align="center">35.90</td>
+        <td align="center">35.50</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">BigCodeBench<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>52.19</strong></td>
+        <td align="center">50.35</td>
+        <td align="center">25.53</td>
+        <td align="center">49.74</td>
+        <td align="center"><u>52.02</u></td>
+        <td align="center">50.18</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">FullStackBench<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><u>51.16</u></td>
+        <td align="center">48.43</td>
+        <td align="center">49.61</td>
+        <td align="center">50.86</td>
+        <td align="center"><strong>54.95</strong></td>
+        <td align="center">50.39</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">LCBench<sup>2</sup> <sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>57.76</strong></td>
+        <td align="center"><u>57.51</u></td>
+        <td align="center">54.22</td>
+        <td align="center">49.67</td>
+        <td align="center">46.94</td>
+        <td align="center">45.12</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="4" align="center">Math</td>
+        <td class="benchmark" align="center">MATH500<sub>(Acc)</sub></td>
+        <td class="shot" align="center">4-shot</td>
+        <td class="ling" align="center"><strong>79.00</strong></td>
+        <td align="center"><u>74.00</u></td>
+        <td align="center">70.80</td>
+        <td align="center">62.80</td>
+        <td align="center">60.60</td>
+        <td align="center">70.00</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">OlympiadBench <sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><u>47.89</u></td>
+        <td align="center">39.31</td>
+        <td align="center"><strong>54.52</strong></td>
+        <td align="center">34.34</td>
+        <td align="center">41.27</td>
+        <td align="center">35.09</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">TheoremQA<sub>(Acc)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><u>61.10</u></td>
+        <td align="center">57.95</td>
+        <td align="center"><strong>72.05</strong></td>
+        <td align="center">56.03</td>
+        <td align="center">60.96</td>
+        <td align="center">51.23</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">OmniMath <sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>47.65</strong></td>
+        <td align="center">36.20</td>
+        <td align="center"><u>38.55</u></td>
+        <td align="center">28.46</td>
+        <td align="center">33.33</td>
+        <td align="center">27.03</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="2" align="center">Reasoning</td>
+        <td class="benchmark" align="center">CommonSenseQA <sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>89.93</strong></td>
+        <td align="center">87.55</td>
+        <td align="center">86.24</td>
+        <td align="center"><u>88.62</u></td>
+        <td align="center">86.98</td>
+        <td align="center">84.60</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">BBH <sub>(EM)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>89.17</strong></td>
+        <td align="center">84.72</td>
+        <td align="center"><u>89.00</u></td>
+        <td align="center">87.19</td>
+        <td align="center">76.00</td>
+        <td align="center">86.22</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="2" align="center">Long-context</td>
+        <td class="benchmark" align="center">LongBench <sub>(Acc)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>52.62</strong></td>
+        <td align="center"><u>42.81</u></td>
+        <td align="center">31.05</td>
+        <td align="center">21.47</td>
+        <td align="center">20.58</td>
+        <td align="center">29.57</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">LEval <sub>(Acc)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling ling-bottom" align="center"><strong>83.24</strong></td>
+        <td align="center"><u>76.89</u></td>
+        <td align="center">58.65</td>
+        <td align="center">60.30</td>
+        <td align="center">66.29</td>
+        <td align="center">73.29</td>
+      </tr>
+    </tbody>
+    <tfoot>
+      <tr>
+        <td class="table-footnote" colspan="7" align="left">
+          <strong>Note:</strong><br>
+          <sup>1</sup> LiveCodeBench (2408-2505)<br>
+          <sup>2</sup> LCBench (2301-2502)
+        </td>
+      </tr>
+    </tfoot>
+  </table>
+</div>
+
+<!-- <img src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787147691544-c61cbb0e-568c-4d5e-a2d9-12a709f003d0.png" width="1431.5" title="" crop="0,0,1,1" id="uf61ea7a7" class="ne-image"> -->
 
 ## **Intended Use**
 Recommended use cases: