LLM-OS-Models/dsc-370m-fineweb-edu-5b-v3

DSC 370M (FineWeb-Edu 5B) — v3, leak-fixed

Status	🟢 v3 — causality leak fixed. v1 (commit-prior) had a routing leak; v3 patches seg_query = last-pos-of-prev-chunk + strict triu(diagonal=0) causal mask. CPU causality smoke test verifies future-perturbation invariance on all chunks.
Architecture	DSC = GDN-2 + Dynamic Sparse Caching (chunk c=256, top-k=2 routing, multi-res descriptor [mean/max/softmax-attn])
Parameters	~370 M (GDN-2 370M backbone + small router + combine_alpha)
Training data	FineWeb-Edu sample/100BT (5 B-token slice)
Tokenizer	TinyLlama v1.1 (vocab = 32 000)
Context length	4 096 (training)
Hardware	8 × NVIDIA H200 141 GB (FSDP)
Loss @ 5B	see docs/DSC_LIVE_STATUS_KO.md for final trajectory
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn

---

1. What is DSC?

DSC (Dynamic Sparse Caching) augments GDN-2 with explicit cached state slots and a learned top-k router that decides which cached slots each token reads from. Goal: give the recurrence a much larger effective state than the per-head S_t ∈ R^{d_k × d_v} matrix alone, while keeping training and inference O(N) in sequence length.

2. What's new in v3

v1 leak mechanism (patched):

• seg_query was chunk-mean of current chunk, letting future positions influence chunk_i's routing query
• Causal mask used triu(diagonal=1) which still permitted diagonal (j=i) future leakage

v3 fix:

• seg_query = torch.zeros_like(normed_chunks[:, :, 0])
• seg_query[:, 1:] = normed_chunks[:, :-1, -1] (use last position of previous chunk as the query for the current chunk's routing decision)
• Causal mask = torch.triu(ones(N, N, dtype=bool), diagonal=0) — strict upper triangular including diagonal

Verification: dsc/scripts/smoke_test_routing_invariance.py confirms that perturbing any future chunk's state does NOT change the current chunk's top-k routing selection or weights (within 1e-6 tolerance).

3. RULER results vs vanilla GDN-2 5B

DSC v3 5B vs Vanilla 5B RULER comparison lives in docs/DSC_V3V4_LONG_CONTEXT_HF_PLAN_KO.md. Headline metric: 4K + 16K singlekey + multikey + multiquery + multivalue (8 tasks × 2 lengths = 16 cells). PASS criteria: DSC ≥ Vanilla in ≥12/16 cells.

4. Known limitations

• 5B tokens is well below Chinchilla optimal for 370M params (~7.5B)
• Single seed; multi-seed variance not characterized
• Multi-key NIAH at long-L is the main weakness that v4 (entropy bias + learnable summary) targets

5. Citation

@misc{dsc-v3-5b-2026,
  author = {LLM-OS-Models},
  title  = {DSC 370M FineWeb-Edu 5B v3 (leak-fixed)},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/dsc-370m-fineweb-edu-5b-v3}
}