| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - kolmogorov-arnold-networks |
| - kan |
| - small-language-models |
| - interpretability |
| - pruning |
| - babylm |
| - negative-results |
| datasets: |
| - arman-bd/guppylm-60k-generic |
| pipeline_tag: text-generation |
| --- |
| |
| # KAN-LM Study — Model Checkpoints |
|
|
| Trained checkpoints for the paper **"Auditing and Benchmarking KAN Feed-Forward |
| Layers in Small Language Models."** These are the model weights behind every |
| figure and table; the code, experiment scripts, and provenance manifest live in |
| the companion repository. |
|
|
| - **Code + reproduction:** https://github.com/ACS-USP/kan-lm-study |
| - **Paper:** see `paper/` in the code repo |
| - **DOI:** [10.57967/hf/9264](https://doi.org/10.57967/hf/9264) |
|
|
| > These are **not** `transformers`-loadable models. They are `kanprey` checkpoints |
| > (custom KAN/MLP transformer). Load them with the vendored `kan-guppylm` code in |
| > the companion repo (`vendor/kan-guppylm/kanprey`), not `AutoModel`. |
|
|
| ## What's here |
|
|
| `best.pt` for every training run (128 files, ~21.7 GB), organized by regime: |
|
|
| | Regime | Path prefix | Contents | |
| |---|---|---| |
| | GuppyLM screen | `mlp_s*`, `swiglu_s*`, `kan_grid2_s*`, `grkan_corrected_s*`, `basis_confirm/*`, `kat_s*`, `mlpedge_*` | 3-seed architecture screen (d=384, 6 layers, vocab 2,393) | |
| | BabyLM Strict-Small | `babylm/<arch>_s42..s51` | the 61-run matrix (4 critical × 10 seeds + support/low rows), vocab 8,192 | |
| | Grid-size sweep | `gridsweep/*` | KAN grid 2/5/10/20 for the interpretability-vs-capacity sweep | |
| | Wikitext-103 scale | `scale/mlp`, `scale/mlpedge_h8` | GPT-2-small parameter-matched stress test | |
|
|
| - `INVENTORY.tsv` — every `best.pt` with size and path. |
| - `SHA256SUMS` — integrity checksums; verify with `shasum -a 256 -c SHA256SUMS`. |
|
|
| (The 286M ClimbMix GR-KAN stress-test checkpoints are large and tracked separately; |
| see the code repo's `manifest.json`.) |
|
|
| ## Provenance and correctness |
|
|
| - The corrected rational activation uses the **Safe Padé** denominator |
| `Q(x)=1+|b0 x + b1 x^2 + b2 x^3 + b3 x^4|`. **Pre-fix GR-KAN checkpoints are |
| excluded** from all reported evidence and are not in this collection. |
| - Repo commits, the kernel correction, and a figure/table → script → checkpoint map |
| are in `manifest.json` in the code repo. |
|
|
| ## Loading a checkpoint |
|
|
| ```python |
| import torch |
| from kanprey.config import ModelConfig # from vendor/kan-guppylm |
| from kanprey.model import KANpreyLM, MLPTransformer |
| |
| ckpt = torch.load("babylm/grkan_canonical_s42/best.pt", map_location="cpu", weights_only=False) |
| cfg = ckpt["model_cfg"] |
| model = (MLPTransformer if ckpt["model_type"] == "mlp" else KANpreyLM)(cfg) |
| model.load_state_dict(ckpt["model"]); model.eval() |
| ``` |
|
|
| ## Licenses |
|
|
| Weights: MIT. Training data retains its own licenses — GuppyLM (MIT), BabyLM |
| challenge corpus, Wikitext-103 (CC BY-SA 3.0/GFDL), ClimbMix → NVIDIA |
| Nemotron-ClimbMix (CC BY-NC 4.0, research use). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{alves2026kanlm, |
| title = {Auditing and Benchmarking KAN Feed-Forward Layers in Small Language Models}, |
| author = {Alves, Felippe}, |
| year = {2026}, |
| note = {Code: https://github.com/ACS-USP/kan-lm-study} |
| } |
| |
| @misc{acsusp2026kanlmckpts, |
| author = {Agentic Complex Systems - USP}, |
| title = {kan-lm-study-checkpoints}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| doi = {10.57967/hf/9264}, |
| url = {https://huggingface.co/ACS-USP/kan-lm-study-checkpoints} |
| } |
| ``` |
|
|
