| --- |
| license: mit |
| tags: |
| - block-sparse |
| - recursive |
| - language-model |
| - parameter-offloading |
| - research |
| library_name: pytorch |
| --- |
| |
| # SR-Core — Recursive Block-Sparse Language Models (checkpoints) |
|
|
| Trained checkpoints for the research monograph **Recursive Block-Sparse Language Models** |
| (SR-Core). Code, the full write-up, and all evaluation data are on GitHub: |
| **https://github.com/vijedich/sr-core-recursive-blocklm** |
|
|
| SR-Core is a block-sparse recursive LM with a hard working-set guarantee (WS=k): routing is |
| decided once (at recursion step r=1) and reused for all further steps, so exactly *k* blocks |
| are active per token, independent of bank size or recursion depth. This makes the per-token |
| weight footprint statically bounded — the property the monograph studies for cache-efficient |
| RAM→VRAM parameter streaming. |
|
|
| These are **research checkpoints**, not a deployment model. At this scale (~19M params, ~6.6M |
| token four-domain corpus) SR-Core does **not** match dense quality; it trails Dense d24 by |
| ~0.5 nats (seen) / ~0.35 nats (held-out). Its contribution is transfer efficiency, not quality. |
| See the monograph for the full claim boundaries. |
|
|
| ## Checkpoints |
|
|
| Each `.pt` is a full training snapshot (`model`, `optimizer`, `scheduler`, RNG state, |
| `step`, `config`) — self-describing, loadable via `rblm.model_io.load_checkpoint`, and |
| resumable for the convergence run (monograph §7.5.6). |
|
|
| | File pattern | What it is | |
| |---|---| |
| | `hm_cont_hm_srcore_b64_k8_R6_s{0-3}.pt` | SR-Core b64 k8 R6, 4 seeds — quality evidence | |
| | `hm_cont_hm_dense_d24_17k_s0.pt` | Dense d24 @17k — quality ceiling (seen 4.79) | |
| | `hm_cont_hm_dense_d24_s0.pt` | Dense d24 @10k — earlier snapshot | |
| | `hm_cont_hm_srcore_b64_k16_R4_d256h192_s{0-2}.pt` | Param/compute-matched — the ~0.5-nat gap evidence | |
| | `hm_cont_hm_srcore_b64_k8_R6_entmin_r1_lam003_s{0-3}.pt` | Entropy consolidation λ=0.003, 4 seeds | |
| | `hm_cont_hm_srcore_b64_k8_R6_entmin_r1_lam005_s{0-3}.pt` | Entropy consolidation λ=0.005, 4 seeds | |
| | `hm_cont_hm_srcore_b64_k8_R6_ctrl_17k_s{0,1}.pt` | Entropy-sweep control | |
| | `hm_cont_hm_srcore_b64_{k2,k8,k16}_R6_asw_s*.pt` | A-sweep depth-vs-capacity variants | |
|
|
| ## Loading |
|
|
| ```python |
| from rblm import model_io # from the GitHub repo |
| model, arch, step = model_io.load_checkpoint("hm_cont_hm_srcore_b64_k8_R6_s0.pt") |
| ``` |
|
|
| ## Citation |
|
|
| Archived on Zenodo with a citable DOI: **[10.5281/zenodo.21146547](https://doi.org/10.5281/zenodo.21146547)** |
| (concept DOI, resolves to the latest version; v1.0 is `10.5281/zenodo.21146548`). |
|
|
| ```bibtex |
| @misc{jedich2026srcore, |
| title = {Entropy-based Router Consolidation for Cache-Efficient |
| Recursive Block-Sparse Language Models}, |
| author = {Jedich, Viktor}, |
| year = {2026}, |
| note = {Self-published research monograph}, |
| doi = {10.5281/zenodo.21146547}, |
| url = {https://github.com/vijedich/sr-core-recursive-blocklm} |
| } |
| ``` |
|
|
| License: **MIT** (code and checkpoints). The monograph text is CC BY 4.0. |
|
|
