vijedich/sr-core-recursive-blocklm

SR-Core — Recursive Block-Sparse Language Models (checkpoints)

Trained checkpoints for the research monograph Recursive Block-Sparse Language Models (SR-Core). Code, the full write-up, and all evaluation data are on GitHub: https://github.com/vijedich/sr-core-recursive-blocklm

SR-Core is a block-sparse recursive LM with a hard working-set guarantee (WS=k): routing is decided once (at recursion step r=1) and reused for all further steps, so exactly k blocks are active per token, independent of bank size or recursion depth. This makes the per-token weight footprint statically bounded — the property the monograph studies for cache-efficient RAM→VRAM parameter streaming.

These are research checkpoints, not a deployment model. At this scale (~19M params, ~6.6M token four-domain corpus) SR-Core does not match dense quality; it trails Dense d24 by ~0.5 nats (seen) / ~0.35 nats (held-out). Its contribution is transfer efficiency, not quality. See the monograph for the full claim boundaries.

Checkpoints

Each .pt is a full training snapshot (model, optimizer, scheduler, RNG state, step, config) — self-describing, loadable via rblm.model_io.load_checkpoint, and resumable for the convergence run (monograph §7.5.6).

File pattern	What it is
hm_cont_hm_srcore_b64_k8_R6_s{0-3}.pt	SR-Core b64 k8 R6, 4 seeds — quality evidence
hm_cont_hm_dense_d24_17k_s0.pt	Dense d24 @17k — quality ceiling (seen 4.79)
hm_cont_hm_dense_d24_s0.pt	Dense d24 @10k — earlier snapshot
hm_cont_hm_srcore_b64_k16_R4_d256h192_s{0-2}.pt	Param/compute-matched — the ~0.5-nat gap evidence
hm_cont_hm_srcore_b64_k8_R6_entmin_r1_lam003_s{0-3}.pt	Entropy consolidation λ=0.003, 4 seeds
hm_cont_hm_srcore_b64_k8_R6_entmin_r1_lam005_s{0-3}.pt	Entropy consolidation λ=0.005, 4 seeds
hm_cont_hm_srcore_b64_k8_R6_ctrl_17k_s{0,1}.pt	Entropy-sweep control
hm_cont_hm_srcore_b64_{k2,k8,k16}_R6_asw_s*.pt	A-sweep depth-vs-capacity variants

Loading

from rblm import model_io           # from the GitHub repo
model, arch, step = model_io.load_checkpoint("hm_cont_hm_srcore_b64_k8_R6_s0.pt")

Citation

Archived on Zenodo with a citable DOI: 10.5281/zenodo.21146547 (concept DOI, resolves to the latest version; v1.0 is 10.5281/zenodo.21146548).

@misc{jedich2026srcore,
  title  = {Entropy-based Router Consolidation for Cache-Efficient
            Recursive Block-Sparse Language Models},
  author = {Jedich, Viktor},
  year   = {2026},
  note   = {Self-published research monograph},
  doi    = {10.5281/zenodo.21146547},
  url    = {https://github.com/vijedich/sr-core-recursive-blocklm}
}

License: MIT (code and checkpoints). The monograph text is CC BY 4.0.