DSC 370M (FineWeb-Edu 1B) — v4 sanity check
| Status | 🟢 v4 1B sanity check — gate decision input. Tests whether v4's architectural improvements (entropy bias + attention-pool descriptor) preserve v3's already-good 1B loss before escalating to 5B. |
| Architecture | DSC = GDN-2 + Dynamic Sparse Caching + Idea 1 (learnable attention-pool desc_attn_v) + Idea 2 (ent_bias_scale entropy bias on routing logits) |
| Parameters | ~370 M |
| Training data | FineWeb-Edu sample/100BT (1 B-token slice) |
| Tokenizer | TinyLlama v1.1 (vocab = 32 000) |
| Context length | 4 096 (training) |
| Hardware | 8 × NVIDIA H200 141 GB (FSDP) |
| Loss @ 1B | see docs/DSC_LIVE_STATUS_KO.md |
| License | Apache-2.0 |
| Trained by | LLM-OS-Models · code at gyunggyung/long-gdn |
1. v4 gate criteria
The 1B v4 model is evaluated head-to-head against the 1B v3 model on RULER 4K/16K × 4 tasks (8 cells). Gate passes if:
- v4 loss ≤ v3 loss + 0.05
- v4 ≥ v3 - 0.02 in ≥ 5/8 cells
- v4 4K single_1 ≥ 0.30 (collapse floor)
PASS → escalate to v4 5B head-to-head against v3 5B. FAIL → v3 remains the campaign champion, v4 not escalated.
2. What's new in v4
Two architectural improvements on top of v3's leak-fixed foundation:
- Learnable attention-pool descriptor (
desc_attn_v): replaces the fixed [mean/max/softmax-attn] triplet with a learned attention pool so the descriptor can highlight key-relevant features per chunk. - Entropy bias on routing logits (
ent_bias_scale): adds a bias proportional to batch-level routing entropy, letting the router sharpen or smooth its decisions dynamically.
3. Citation
@misc{dsc-v4-1b-2026,
author = {LLM-OS-Models},
title = {DSC 370M FineWeb-Edu 1B v4 sanity check},
year = {2026},
url = {https://huggingface.co/LLM-OS-Models/dsc-370m-fineweb-edu-1b-v4}
}