GDN-2 370M (FineWeb-Edu 5B) — vanilla baseline
| Architecture | GatedDeltaNet-2 (Yang et al. 2025), 370M parameter variant |
| Parameters | 370 M |
| Training data | FineWeb-Edu sample/100BT (5 B-token slice) |
| Tokenizer | TinyLlama v1.1 (vocab = 32 000) |
| Context length | 4 096 (training) |
| Hardware | 8 × NVIDIA H200 141 GB (FSDP) |
| License | Apache-2.0 |
| Trained by | LLM-OS-Models · code at gyunggyung/long-gdn |
1. Purpose
Vanilla GDN-2 370M baseline trained on the same 5B-token slice as
dsc-370m-fineweb-edu-5b-v3, so the DSC vs vanilla comparison is
apples-to-apples (same data, same compute, same hyperparameters).
2. RULER results (vs DSC v3 5B)
DSC v3 vs Vanilla 5B comparison is the 5B verdict that determines whether
DSC is a useful architectural addition to GDN-2. Full results in
docs/DSC_LIVE_STATUS_KO.md. This vanilla checkpoint is the floor that DSC
improvement is measured against.
3. Known limitations
- 5B tokens below Chinchilla optimal (~7.5B for 370M params)
- Long-context (≥16K) retrieval fails completely (vanilla linear attention state collapse) — the gap DSC is designed to close
4. Citation
@misc{gdn2-vanilla-5b-2026,
author = {LLM-OS-Models},
title = {GDN-2 370M FineWeb-Edu 5B vanilla baseline},
year = {2026},
url = {https://huggingface.co/LLM-OS-Models/gdn2-370m-fineweb-edu-5b}
}