File size: 4,871 Bytes
b1804fc 51d4d02 b1804fc 51d4d02 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | ---
license: other
license_name: april
license_link: LICENSE.md
language:
- en
tags:
- language-model
- cpu-native
- sparse
- gated
- numpy
- no-gpu
- acid-research
- gsn
library_name: custom
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
---
# GSN — Gated Sparse Network (Open Weights)
**Acid Research (ACRS)**
This repository contains trained weights for GSN, a language model architecture designed around one principle: **do not spend compute you do not need.**
Unlike transformers — which run the full model on every input regardless of complexity — GSN gates compute dynamically. Simple inputs take a shallow path. Complex inputs go deeper. The network decides, not the configuration.
For the full architecture, training code, and inference CLI, see [AcidAI/Acid-GSN-Architecture](https://huggingface.co/AcidAI/Acid-GSN-Architecture).
---
## Architecture
GSN is built from three core ideas stacked together:
**1. Gated Depth** A lightweight gate network evaluates each input and decides how many layers to activate. Layers that are not needed do not run. Their compute cost is exactly zero.
**2. Sparse Activation** Within each active layer, only the top-k neurons fire. The rest are masked to zero. A 4-layer GSN running at 25% sparsity uses a fraction of the compute a dense model would.
**3. Recurrent Encoder** Input tokens are processed sequentially through a GRU encoder before the gate sees anything. This replaces mean pooling — token order matters, context accumulates, and the gate receives a hidden state that actually encodes sequence structure.
**4. Compute Penalty in Training** The loss function penalizes wasted compute. The model is trained to be cheap, not just accurate. Over time the gate learns: if I can answer correctly with one layer, using two is a mistake.
**No attention. No transformers. O(T) encoding. O(1) per sparse layer.**
---
## Files
| File | Description |
| --- | --- |
| `gsn_fw_weights.npz` | Model weights (~19.7 MB) |
| `gsn_fw_opt.npz` | Adam optimizer state — m1/m2 moments and global step for resuming training (~39.4 MB) |
| `LICENSE.md` | APRIL license |
---
## Usage
### Requirements
```bash
pip install numpy
```
No PyTorch. No CUDA. No framework dependency. Pure NumPy.
### Loading the Weights
```python
import numpy as np
# Load weights
weights = np.load("gsn_fw_weights.npz")
print(list(weights.keys())) # inspect layer names
# Load optimizer state (for resuming training)
opt_state = np.load("gsn_fw_opt.npz")
```
For training and inference code, clone the architecture repository:
```bash
# Clone the architecture repo alongside these weights
# python infer.py "your prompt here" --max_new 100
```
See [AcidAI/Acid-GSN-Architecture](https://huggingface.co/AcidAI/Acid-GSN-Architecture) for the full usage guide.
---
## Training Details
**Corpus:** FineWeb-Edu (high-quality educational web text)
**Vocabulary:** 1024 BPE tokens
**Hardware:** 2 physical CPU cores (Debian Linux)
**Training time:** ~25 minutes
**Best loss:** ~3.72
**Average gate depth:** ~2 / 4 layers
The gate settled at adaptive depth 2, confirming the compute penalty is functioning as intended. Depth variation is expected to increase on more diverse corpora.
---
## Limitations
- **Undertrained checkpoint.** These weights are a research-stage snapshot, not a production model. Coherent generation requires significantly more training on a larger corpus.
- **Small vocabulary.** 1024 tokens is minimal. Real deployments should use 8k–32k.
- **Gate behavior is corpus-dependent.** A model trained on FineWeb-Edu will gate differently than one trained on code or conversational text.
- **Single-sequence inference only.** Batched inference is not yet implemented.
---
## License
Licensed under the **Acid Research Protected Interests License (APRIL) v1.0**.
- Free for personal, academic, and non-commercial use.
- Derivatives must be open sourced under APRIL.
- Commercial use requires written permission from ACRS.
- Attribution to Acid Research (ACRS) is required in all derivatives.
See [LICENSE.md](LICENSE.md) for full terms.
---
## Citation
If you use GSN in research or build on this architecture, please cite:
```
@misc{gsn2025,
title = {GSN: Gated Sparse Network},
author = {Acid Research (ACRS)},
year = {2025},
url = {https://huggingface.co/AcidAI/GSN-Open-Weights}
}
```
---
## About Acid Research
Acid Research (ACRS) is an independent AI research organization building CPU-native, economically viable alternatives to transformer-based architectures.
Current architecture portfolio:
- **HAM** — Hebbian Architecture Model
- **RSM** — Recurrent State Machine
- **RDM** — Recurrent Depth Machine
- **IMA** — Intent Machine Architecture
- **GSN** — Gated Sparse Network
*"Make it linear, or else your cost ain't going to be."* |