LLM-OS-Models/dsc-370m-fineweb-edu-5b-v4

DSC 370M (FineWeb-Edu 5B) — v4

Status	🟢 v4 5B — head-to-head contender against v3 5B. Launched only if the v4 1B gate passes. Adds entropy bias + attention-pool descriptor on top of v3's leak-fixed foundation.
Architecture	DSC = GDN-2 + Dynamic Sparse Caching + Idea 1 (attention-pool descriptor) + Idea 2 (entropy bias)
Parameters	~370 M
Training data	FineWeb-Edu sample/100BT (5 B-token slice)
Tokenizer	TinyLlama v1.1 (vocab = 32 000)
Context length	4 096 (training)
Hardware	8 × NVIDIA H200 141 GB (FSDP)
Loss @ 5B	see docs/DSC_LIVE_STATUS_KO.md
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn

---

1. Winner decision (v3 vs v4 at 5B)

v4 must beat v3 in the 5B head-to-head to earn 30B escalation:

• v4 loss ≤ v3 loss + 0.05 (strict regression check)
• v4 ≥ v3 - 0.02 in ≥ 7/12 short-L cells (4K/16K × 6 tasks)
• v4 4K single_1 ≥ 0.30 (collapse floor)
• Long-L bonus (32K/64K/128K): v4 우위 logged as evidence, not required

PASS → winner=v4 → 30B v4 escalation. FAIL → winner=v3 → 30B v3 escalation.

2. What's new in v4

Two architectural improvements on top of v3's leak-fixed foundation:

1. Learnable attention-pool descriptor (desc_attn_v): replaces the fixed [mean/max/softmax-attn] triplet with a learned attention pool so the descriptor can highlight key-relevant features per chunk.
2. Entropy bias on routing logits (ent_bias_scale): adds a bias proportional to batch-level routing entropy, letting the router sharpen or smooth its decisions dynamically.

3. v4 source swap

v4 is applied at training time by copying dsc/v4/dsc.py over off/GatedDeltaNet-2/lit_gpt/dsc.py (with a trap to restore v3 on exit). See pretrain_dsc_370m_5b_v4.sh and pretrain_dsc_370m_30b_v4.sh.

4. Citation

@misc{dsc-v4-5b-2026,
  author = {LLM-OS-Models},
  title  = {DSC 370M FineWeb-Edu 5B v4},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/dsc-370m-fineweb-edu-5b-v4}
}