Update README.md
Browse files
README.md
CHANGED
|
@@ -33,7 +33,7 @@ These checkpoints are released to support continued pretraining, fine-tuning, an
|
|
| 33 |
+ **[Warmup-Stable and Merge](https://arxiv.org/abs/2507.17634):** We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
|
| 34 |
+ **Scale Seamlessly:** Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
|
| 35 |
|
| 36 |
-
| Model Type | Base (final checkpoint of
|
| 37 |
| --- | --- |
|
| 38 |
| Architecture | Hybrid-linear MoE |
|
| 39 |
| Parameter Scale | Totoal 124B, Activated 5.1B (Non-emb) |
|
|
|
|
| 33 |
+ **[Warmup-Stable and Merge](https://arxiv.org/abs/2507.17634):** We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
|
| 34 |
+ **Scale Seamlessly:** Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
|
| 35 |
|
| 36 |
+
| Model Type | Base (final checkpoint of WSM merging) |
|
| 37 |
| --- | --- |
|
| 38 |
| Architecture | Hybrid-linear MoE |
|
| 39 |
| Parameter Scale | Totoal 124B, Activated 5.1B (Non-emb) |
|