| --- |
| license: other |
| license_name: april |
| license_link: LICENSE.md |
| language: |
| - en |
| tags: |
| - language-model |
| - cpu-native |
| - sparse |
| - gated |
| - numpy |
| - no-gpu |
| - acid-research |
| - gsn |
| library_name: custom |
| pipeline_tag: text-generation |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| --- |
| |
| # GSN — Gated Sparse Network (Open Weights) |
|
|
| **Acid Research (ACRS)** |
|
|
| This repository contains trained weights for GSN, a language model architecture designed around one principle: **do not spend compute you do not need.** |
|
|
| Unlike transformers — which run the full model on every input regardless of complexity — GSN gates compute dynamically. Simple inputs take a shallow path. Complex inputs go deeper. The network decides, not the configuration. |
|
|
| For the full architecture, training code, and inference CLI, see [AcidAI/Acid-GSN-Architecture](https://huggingface.co/AcidAI/Acid-GSN-Architecture). |
|
|
| --- |
|
|
| ## Architecture |
|
|
| GSN is built from three core ideas stacked together: |
|
|
| **1. Gated Depth** A lightweight gate network evaluates each input and decides how many layers to activate. Layers that are not needed do not run. Their compute cost is exactly zero. |
|
|
| **2. Sparse Activation** Within each active layer, only the top-k neurons fire. The rest are masked to zero. A 4-layer GSN running at 25% sparsity uses a fraction of the compute a dense model would. |
|
|
| **3. Recurrent Encoder** Input tokens are processed sequentially through a GRU encoder before the gate sees anything. This replaces mean pooling — token order matters, context accumulates, and the gate receives a hidden state that actually encodes sequence structure. |
|
|
| **4. Compute Penalty in Training** The loss function penalizes wasted compute. The model is trained to be cheap, not just accurate. Over time the gate learns: if I can answer correctly with one layer, using two is a mistake. |
|
|
| **No attention. No transformers. O(T) encoding. O(1) per sparse layer.** |
|
|
| --- |
|
|
| ## Files |
|
|
| | File | Description | |
| | --- | --- | |
| | `gsn_fw_weights.npz` | Model weights (~19.7 MB) | |
| | `gsn_fw_opt.npz` | Adam optimizer state — m1/m2 moments and global step for resuming training (~39.4 MB) | |
| | `LICENSE.md` | APRIL license | |
|
|
| --- |
|
|
| ## Usage |
|
|
| ### Requirements |
|
|
| ```bash |
| pip install numpy |
| ``` |
|
|
| No PyTorch. No CUDA. No framework dependency. Pure NumPy. |
|
|
| ### Loading the Weights |
|
|
| ```python |
| import numpy as np |
| |
| # Load weights |
| weights = np.load("gsn_fw_weights.npz") |
| print(list(weights.keys())) # inspect layer names |
| |
| # Load optimizer state (for resuming training) |
| opt_state = np.load("gsn_fw_opt.npz") |
| ``` |
|
|
| For training and inference code, clone the architecture repository: |
|
|
| ```bash |
| # Clone the architecture repo alongside these weights |
| # python infer.py "your prompt here" --max_new 100 |
| ``` |
|
|
| See [AcidAI/Acid-GSN-Architecture](https://huggingface.co/AcidAI/Acid-GSN-Architecture) for the full usage guide. |
|
|
| --- |
|
|
| ## Training Details |
|
|
| **Corpus:** FineWeb-Edu (high-quality educational web text) |
| **Vocabulary:** 1024 BPE tokens |
| **Hardware:** 2 physical CPU cores (Debian Linux) |
| **Training time:** ~25 minutes |
| **Best loss:** ~3.72 |
| **Average gate depth:** ~2 / 4 layers |
|
|
| The gate settled at adaptive depth 2, confirming the compute penalty is functioning as intended. Depth variation is expected to increase on more diverse corpora. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Undertrained checkpoint.** These weights are a research-stage snapshot, not a production model. Coherent generation requires significantly more training on a larger corpus. |
| - **Small vocabulary.** 1024 tokens is minimal. Real deployments should use 8k–32k. |
| - **Gate behavior is corpus-dependent.** A model trained on FineWeb-Edu will gate differently than one trained on code or conversational text. |
| - **Single-sequence inference only.** Batched inference is not yet implemented. |
|
|
| --- |
|
|
| ## License |
|
|
| Licensed under the **Acid Research Protected Interests License (APRIL) v1.0**. |
|
|
| - Free for personal, academic, and non-commercial use. |
| - Derivatives must be open sourced under APRIL. |
| - Commercial use requires written permission from ACRS. |
| - Attribution to Acid Research (ACRS) is required in all derivatives. |
|
|
| See [LICENSE.md](LICENSE.md) for full terms. |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use GSN in research or build on this architecture, please cite: |
|
|
| ``` |
| @misc{gsn2025, |
| title = {GSN: Gated Sparse Network}, |
| author = {Acid Research (ACRS)}, |
| year = {2025}, |
| url = {https://huggingface.co/AcidAI/GSN-Open-Weights} |
| } |
| ``` |
|
|
| --- |
|
|
| ## About Acid Research |
|
|
| Acid Research (ACRS) is an independent AI research organization building CPU-native, economically viable alternatives to transformer-based architectures. |
|
|
| Current architecture portfolio: |
|
|
| - **HAM** — Hebbian Architecture Model |
| - **RSM** — Recurrent State Machine |
| - **RDM** — Recurrent Depth Machine |
| - **IMA** — Intent Machine Architecture |
| - **GSN** — Gated Sparse Network |
|
|
| *"Make it linear, or else your cost ain't going to be."* |
