Add Zenodo DOI to citation (concept 10.5281/zenodo.21146547)

d00ae5c verified 9 days ago

3 kB

license: mit
tags:
  - block-sparse
  - recursive
  - language-model
  - parameter-offloading
  - research
library_name: pytorch

SR-Core — Recursive Block-Sparse Language Models (checkpoints)

Trained checkpoints for the research monograph Recursive Block-Sparse Language Models (SR-Core). Code, the full write-up, and all evaluation data are on GitHub: https://github.com/vijedich/sr-core-recursive-blocklm

SR-Core is a block-sparse recursive LM with a hard working-set guarantee (WS=k): routing is decided once (at recursion step r=1) and reused for all further steps, so exactly k blocks are active per token, independent of bank size or recursion depth. This makes the per-token weight footprint statically bounded — the property the monograph studies for cache-efficient RAM→VRAM parameter streaming.

These are research checkpoints, not a deployment model. At this scale (~19M params, ~6.6M token four-domain corpus) SR-Core does not match dense quality; it trails Dense d24 by ~0.5 nats (seen) / ~0.35 nats (held-out). Its contribution is transfer efficiency, not quality. See the monograph for the full claim boundaries.

Checkpoints

Each .pt is a full training snapshot (model, optimizer, scheduler, RNG state, step, config) — self-describing, loadable via rblm.model_io.load_checkpoint, and resumable for the convergence run (monograph §7.5.6).

File pattern	What it is
`hm_cont_hm_srcore_b64_k8_R6_s{0-3}.pt`	SR-Core b64 k8 R6, 4 seeds — quality evidence
`hm_cont_hm_dense_d24_17k_s0.pt`	Dense d24 @17k — quality ceiling (seen 4.79)
`hm_cont_hm_dense_d24_s0.pt`	Dense d24 @10k — earlier snapshot
`hm_cont_hm_srcore_b64_k16_R4_d256h192_s{0-2}.pt`	Param/compute-matched — the ~0.5-nat gap evidence
`hm_cont_hm_srcore_b64_k8_R6_entmin_r1_lam003_s{0-3}.pt`	Entropy consolidation λ=0.003, 4 seeds
`hm_cont_hm_srcore_b64_k8_R6_entmin_r1_lam005_s{0-3}.pt`	Entropy consolidation λ=0.005, 4 seeds
`hm_cont_hm_srcore_b64_k8_R6_ctrl_17k_s{0,1}.pt`	Entropy-sweep control
`hm_cont_hm_srcore_b64_{k2,k8,k16}_R6_asw_s*.pt`	A-sweep depth-vs-capacity variants

Loading

from rblm import model_io           # from the GitHub repo
model, arch, step = model_io.load_checkpoint("hm_cont_hm_srcore_b64_k8_R6_s0.pt")

Citation

Archived on Zenodo with a citable DOI: 10.5281/zenodo.21146547 (concept DOI, resolves to the latest version; v1.0 is 10.5281/zenodo.21146548).

@misc{jedich2026srcore,
  title  = {Entropy-based Router Consolidation for Cache-Efficient
            Recursive Block-Sparse Language Models},
  author = {Jedich, Viktor},
  year   = {2026},
  note   = {Self-published research monograph},
  doi    = {10.5281/zenodo.21146547},
  url    = {https://github.com/vijedich/sr-core-recursive-blocklm}
}

License: MIT (code and checkpoints). The monograph text is CC BY 4.0.