DSC 370M (FineWeb-Edu 1B) — v3, leak-fixed
| Status | 🟢 v3 — causality leak fixed. v1 had a routing leak; v3 patches seg_query = last-pos-of-prev-chunk + strict triu(diagonal=0) causal mask. CPU causality smoke test verifies future-perturbation invariance on all chunks. |
| Architecture | DSC = GDN-2 + Dynamic Sparse Caching (chunk c=256, top-k=2 routing, multi-res descriptor [mean/max/softmax-attn]) |
| Parameters | ~370 M (GDN-2 370M backbone + small router + combine_alpha) |
| Training data | FineWeb-Edu sample/100BT (1 B-token slice) |
| Tokenizer | TinyLlama v1.1 (vocab = 32 000) |
| Context length | 4 096 (training) |
| Hardware | 8 × NVIDIA H200 141 GB (FSDP) |
| Loss @ 1B | 3.49 (vs vanilla GDN-2 1B = 3.51) |
| License | Apache-2.0 |
| Trained by | LLM-OS-Models · code at gyunggyung/long-gdn |
1. What is DSC?
DSC (Dynamic Sparse Caching) augments GDN-2 with explicit cached state slots
and a learned top-k router that decides which cached slots each token reads
from. Goal: give the recurrence a much larger effective state than the per-head
S_t ∈ R^{d_k × d_v} matrix alone, while keeping training and inference
O(N) in sequence length.
2. What's new in v3
v1 leak mechanism (patched in v3):
seg_querywas computed from chunk-mean of the current chunk, allowing future positions inside chunk_i to influence chunk_i's routing query- Causal mask used
triu(diagonal=1)which still permitted diagonal (j=i) future leakage
v3 fix:
seg_query = torch.zeros_like(normed_chunks[:, :, 0])seg_query[:, 1:] = normed_chunks[:, :-1, -1](use last position of previous chunk as the query for routing decision of current chunk)- Causal mask =
torch.triu(ones(N, N, dtype=bool), diagonal=0)— strict upper triangular including diagonal, so future chunks are fully masked
Verification: dsc/scripts/smoke_test_routing_invariance.py confirms
that perturbing any future chunk's state does NOT change the current chunk's
top-k routing selection or weights (within 1e-6 tolerance).
3. RULER results vs vanilla GDN-2 1B
DSC v3 wins on 6/8 RULER 4K cells, average +31% over vanilla. (Full table
lives in docs/DSC_V3_VS_VANILLA_1B_SPEED_LOSS_KO.md.)
4. Known limitations
- Trained on 1B tokens only — research-grade, not production-ready
- Single seed; multi-seed variance not yet characterized
- Multi-key NIAH still hard for v3 (target of v4 entropy bias + learnable summary, in progress)
5. Citation
@misc{dsc-v3-1b-2026,
author = {LLM-OS-Models},
title = {DSC 370M FineWeb-Edu 1B v3 (leak-fixed)},
year = {2026},
url = {https://huggingface.co/LLM-OS-Models/dsc-370m-fineweb-edu-1b-v3}
}