GDN-2 370M (FineWeb-Edu 1B) — vanilla baseline
| Architecture | GatedDeltaNet-2 (Yang et al. 2025), 370M parameter variant |
| Parameters | 370 M |
| Training data | FineWeb-Edu sample/100BT (1 B-token slice) |
| Tokenizer | TinyLlama v1.1 (vocab = 32 000) |
| Context length | 4 096 (training) |
| Hardware | 8 × NVIDIA H200 141 GB (FSDP) |
| Loss @ 1B | 3.51 |
| License | Apache-2.0 |
| Trained by | LLM-OS-Models · code at gyunggyung/long-gdn |
1. Purpose
Vanilla GDN-2 370M baseline trained on the same 1B-token slice as
dsc-370m-fineweb-edu-1b-v3, so that DSC vs vanilla comparison is
apples-to-apples (same data, same compute, same hyperparameters).
2. RULER results (vs DSC v3 1B)
DSC v3 wins on 6/8 RULER 4K cells, average +31%. This vanilla checkpoint
is the floor that DSC improvement is measured against. Full table in
docs/DSC_V3_VS_VANILLA_1B_SPEED_LOSS_KO.md.
3. Known limitations
- 1B tokens is far below convergence — research-grade only
- Long-context (≥16K) retrieval fails completely (vanilla linear attention state collapse), which is the gap DSC is designed to close
4. Citation
@misc{gdn2-vanilla-1b-2026,
author = {LLM-OS-Models},
title = {GDN-2 370M FineWeb-Edu 1B vanilla baseline},
year = {2026},
url = {https://huggingface.co/LLM-OS-Models/gdn2-370m-fineweb-edu-1b}
}