Hugging Face's logo

inclusionAI
/
Ling-3.0-tiny-base

Safetensors
bailing_hybrid
custom_code
Model card Files
xet
 Community
yuxintian commited on
Commit
a115ca9
·
verified ·
1 Parent(s): f141aa7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +31 -16
README.md CHANGED
@@ -29,27 +29,42 @@ These checkpoints are released to support continued pretraining, fine-tuning, an
29
  ### Key features
30
  + **Sparse MoE architecture:** 128 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 1.3B parameters per token;
31
  + **Native hybrid linear attention:** Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
32
- + **[**Warmup-Stable and Merge](https://arxiv.org/abs/2507.17634):** We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
33
  + **Scale Seamlessly:** Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
34
 
35
 
36
- | Model Type | Base (final checkpoint of WSM merging) |
37
- |---|---|
38
- | Architecture | Hybrid-linear MoE |
39
- | Parameter Scale | Totoal 7.9B, Activated 1.3B |
40
- | Transformer Layers | 18 KDA + 6 Gated MLA (3:1) |
41
- | Number of Dense Layers | 1 |
42
- | Number of Routed Experts | 128 |
43
- | Number of Shared Experts | 1 |
44
- | Number of Activated Experts | 8 |
45
- | Attention Heads | 16 |
46
- | Hidden Size | 1536 |
47
- | Expert Intermediate Size | 512 |
48
- | Dense Intermediate Size | 4608 |
49
- | Vocabulary Size | 157,184 |
 
 
 
 
 
 
 
 
 
50
 
51
 
52
- <img src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787120625910-bb0c32bd-7e6b-4354-ac65-fa5b747d8bff.png" width="968.5" title="" crop="0,0,1,1" id="ud5147f99" class="ne-image">
 
 
 
 
 
 
53
 
54
  ## Base Model Evaluation
55
  To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including knowledge, coding, mathematics, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., `Ling-3.0-tiny-base`, is compared below:
 
29
  ### Key features
30
  + **Sparse MoE architecture:** 128 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 1.3B parameters per token;
31
  + **Native hybrid linear attention:** Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
32
+ + **[Warmup-Stable and Merge](https://arxiv.org/abs/2507.17634):** We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
33
  + **Scale Seamlessly:** Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
34
 
35
 
36
+ <div style="display: flex; justify-content: center; width: 100%;">
37
+ <table style="text-align: center;">
38
+ <thead>
39
+ <tr>
40
+ <th align="center">Model Type</th>
41
+ <th align="center">Base (final checkpoint of WSM merging)</th>
42
+ </tr>
43
+ </thead>
44
+ <tbody>
45
+ <tr><td align="center">Architecture</td><td align="center">Hybrid-linear MoE</td></tr>
46
+ <tr><td align="center">Parameter Scale</td><td align="center">Totoal 7.9B, Activated 1.3B</td></tr>
47
+ <tr><td align="center">Transformer Layers</td><td align="center">18 KDA + 6 Gated MLA (3:1)</td></tr>
48
+ <tr><td align="center">Number of Dense Layers</td><td align="center">1</td></tr>
49
+ <tr><td align="center">Number of Routed Experts</td><td align="center">128</td></tr>
50
+ <tr><td align="center">Number of Shared Experts</td><td align="center">1</td></tr>
51
+ <tr><td align="center">Number of Activated Experts</td><td align="center">8</td></tr>
52
+ <tr><td align="center">Attention Heads</td><td align="center">16</td></tr>
53
+ <tr><td align="center">Hidden Size</td><td align="center">1536</td></tr>
54
+ <tr><td align="center">Expert Intermediate Size</td><td align="center">512</td></tr>
55
+ <tr><td align="center">Dense Intermediate Size</td><td align="center">4608</td></tr>
56
+ </tbody>
57
+ </table>
58
+ </div>
59
 
60
 
61
+ <p align="center">
62
+ <img
63
+ src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787120625910-bb0c32bd-7e6b-4354-ac65-fa5b747d8bff.png"
64
+ width="969"
65
+ alt=""
66
+ >
67
+ </p>
68
 
69
  ## Base Model Evaluation
70
  To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including knowledge, coding, mathematics, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., `Ling-3.0-tiny-base`, is compared below: