DSC v4 370M — FineWeb-Edu 5B pretrain
Continued pretrain of a custom DSC (Distributed Sparse Chunk-routing) variant built on GatedDeltaNet (GDN-2) 370M. This is the v4 architecture trained for 5B tokens on FineWeb-Edu.
Architecture (DSC v4)
- Base: GDN-2 370M (16 layers, d_model=1024)
- chunk_size=256, topk=2 routing
- alpha learnable init=0.5 (final avg=0.2656)
- desc_attn_v (descriptor attention value)
- ent_bias_scale=0 (entropy bias disabled)
Training
- Tokens: 5,000,000,000 (5B)
- Data: FineWeb-Edu sample
- Optimizer: AdamW lr=3e-4, weight_decay=0.01, cosine schedule
- Batch: 2/GPU x 8 = eff_batch 16, seq=4096
- Hardware: 8x H100 80GB (DDP)
- Time: ~9h (5B tokens @ ~155K tok/s)
Loading (custom architecture)
This is NOT a standard HF Transformers checkpoint. Load with the project's DSC code:
The DSC implementation is at: https://github.com/gyunggyung/LLM-OS-Models (under ).
Evaluation
After pretrain (before any fine-tune), RULER NIAH @ 4K:
| task | accuracy_token | note |
|---|---|---|
| niah_single_1 | 0.0175 | sparse needle, hardest for routing |
| niah_multikey_1 | 0.0813 | 1 needle, 1 distractor |
| niah_multikey_2 | 0.0700 | 2 needles, 2 distractors |
| niah_multikey_3 | 0.0799 | 3 needles, 4 distractors |
The model has routing infra but doesn't use it effectively (decorative routing pattern). A2 fine-tune (NIAH memory-only aux loss) recovers single_1 to 0.65+ in 100M tokens. See .
Files
- — PyTorch state_dict (2.0GB, bf16)
- — alpha/router stats snapshot