File size: 3,792 Bytes
85777f5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
---
license: gpl-3.0
tags:
- pytorch
- gpt2
- transformer
- oscillating-activation
- bio-inspired
- language-model
language:
- en
datasets:
- openwebtext
- HuggingFaceTB/smoltalk
pipeline_tag: text-generation
---
# WiggleGPT
A 124M parameter transformer that challenges a 56-year-old assumption in neural network design.
## What Makes It Different?
Since Minsky and Papert's *Perceptrons* (1969), neural networks have relied on **monotonic activation functions** (Sigmoid, ReLU, GELU) — requiring multiple hidden layers to solve non-linearly separable problems like XOR.
WiggleGPT replaces monotonic activations with **learnable oscillating functions**, enabling single neurons to create multiple decision boundaries:
```
f(x) = sin(ωx + φ) · tanh(x) + baseline
```
Where ω (frequency) and φ (phase) are **learnable per-neuron parameters**.
## Results
| Model | Parameters | Val Loss | Notes |
|-------|------------|----------|-------|
| **WiggleGPT** | 124M | **3.1621** | Oscillating activation |
| GPT-2 | 124M | ~3.12 | Standard GELU baseline |
**Within 1.3% of GPT-2 performance** — proving oscillating activations are a functional drop-in replacement at scale.
### The Model Actually Learned to Oscillate
| Parameter | Init | After Training | Change |
|-----------|------|----------------|--------|
| ω mean | 1.0 | 1.096 | +9.6% |
| ω std | 0.1 | **0.602** | **6× increase** |
| ω range | [0.8, 1.2] | [-0.19, 5.17] | Massive expansion |
- **95% of neurons retained active oscillation** (ω > 0.1)
- Some neurons learned frequencies up to ω = 5.17 (five oscillations per unit input)
- Full phase coverage [-π, +π] after training
## Checkpoints
| File | Description |
|------|-------------|
| `ckpt_pretrain.pt` | Base model trained on OpenWebText (~600k iterations) |
| `ckpt_finetune.pt` | Fine-tuned on SmolTalk2 (instruction following) |
## Architecture
| Component | Specification |
|-----------|---------------|
| Parameters | 123,697,920 |
| Layers | 12 |
| Attention Heads | 12 |
| Embedding Dimension | 768 |
| Oscillating Neurons | 36,864 (each with learnable ω, φ, baseline) |
| Normalization | RMSNorm |
| Position Encoding | RoPE (Rotary) |
| Attention | Flash Attention (when available) |
## Usage
See the [GitHub repository](https://github.com/Eden-Eldith/WiggleGPT) for full training, inference, and chat scripts.
```python
# Quick inference example
import torch
from model_bio import GPT, GPTConfig
# Load checkpoint
checkpoint = torch.load('ckpt_pretrain.pt', map_location='cuda')
config = GPTConfig(**checkpoint['config'])
model = GPT(config)
model.load_state_dict(checkpoint['model'])
model.eval()
# Generate text (see sample_bio.py for full implementation)
```
## Training Details
**Pretraining:**
- Dataset: OpenWebText (~9B tokens)
- Iterations: 600,000
- Hardware: RTX 3070 (steps 0–354k) → RTX 5060 Ti 16GB (steps 354k–600k)
- Time: Roughly 20 days total (~15 days on 3070, ~5 days on 5060 Ti)
**Fine-tuning:**
- Dataset: SmolTalk2 (406K examples)
- Oscillation parameters (ω, φ) remained stable — 0.0% of neurons shifted by >0.1
## Citation
```bibtex
@software{wigglegpt2025,
author = {O'Brien, Phillip C.},
title = {WiggleGPT: Revisiting the Monotonicity Assumption in Neural Networks via Oscillating Activation Functions},
year = {2025},
url = {https://github.com/Eden-Eldith/WiggleGPT}
}
```
## Author
**Eden (Phillip C. O'Brien)**
Independent AI Researcher | ORCID: [0009-0007-3961-1182](https://orcid.org/0009-0007-3961-1182)
Built in a garage lab in Gosport, UK. No academic affiliation, no institutional funding — just curiosity and an RTX 3070.
## License
GPL-3.0 — if you build on this, keep it open source.
|