Wiggly!

They do be wigglin'

README.md

@@ -1,3 +1,128 @@
----
-license: gpl-3.0
----
+---
+license: gpl-3.0
+tags:
+  - pytorch
+  - gpt2
+  - transformer
+  - oscillating-activation
+  - bio-inspired
+  - language-model
+language:
+  - en
+datasets:
+  - openwebtext
+  - HuggingFaceTB/smoltalk
+pipeline_tag: text-generation
+---
+
+# WiggleGPT
+
+A 124M parameter transformer that challenges a 56-year-old assumption in neural network design.
+
+![WiggleGPT Architecture](model_architecture.png)
+
+## What Makes It Different?
+
+Since Minsky and Papert's *Perceptrons* (1969), neural networks have relied on **monotonic activation functions** (Sigmoid, ReLU, GELU) — requiring multiple hidden layers to solve non-linearly separable problems like XOR.
+
+WiggleGPT replaces monotonic activations with **learnable oscillating functions**, enabling single neurons to create multiple decision boundaries:
+
+```
+f(x) = sin(ωx + φ) · tanh(x) + baseline
+```
+
+Where ω (frequency) and φ (phase) are **learnable per-neuron parameters**.
+
+## Results
+
+| Model | Parameters | Val Loss | Notes |
+|-------|------------|----------|-------|
+| **WiggleGPT** | 124M | **3.1621** | Oscillating activation |
+| GPT-2 | 124M | ~3.12 | Standard GELU baseline |
+
+**Within 1.3% of GPT-2 performance** — proving oscillating activations are a functional drop-in replacement at scale.
+
+### The Model Actually Learned to Oscillate
+
+| Parameter | Init | After Training | Change |
+|-----------|------|----------------|--------|
+| ω mean | 1.0 | 1.096 | +9.6% |
+| ω std | 0.1 | **0.602** | **6× increase** |
+| ω range | [0.8, 1.2] | [-0.19, 5.17] | Massive expansion |
+
+- **95% of neurons retained active oscillation** (ω > 0.1)
+- Some neurons learned frequencies up to ω = 5.17 (five oscillations per unit input)
+- Full phase coverage [-π, +π] after training
+
+## Checkpoints
+
+| File | Description |
+|------|-------------|
+| `ckpt_pretrain.pt` | Base model trained on OpenWebText (~600k iterations) |
+| `ckpt_finetune.pt` | Fine-tuned on SmolTalk2 (instruction following) |
+
+## Architecture
+
+| Component | Specification |
+|-----------|---------------|
+| Parameters | 123,697,920 |
+| Layers | 12 |
+| Attention Heads | 12 |
+| Embedding Dimension | 768 |
+| Oscillating Neurons | 36,864 (each with learnable ω, φ, baseline) |
+| Normalization | RMSNorm |
+| Position Encoding | RoPE (Rotary) |
+| Attention | Flash Attention (when available) |
+
+## Usage
+
+See the [GitHub repository](https://github.com/Eden-Eldith/WiggleGPT) for full training, inference, and chat scripts.
+
+```python
+# Quick inference example
+import torch
+from model_bio import GPT, GPTConfig
+
+# Load checkpoint
+checkpoint = torch.load('ckpt_pretrain.pt', map_location='cuda')
+config = GPTConfig(**checkpoint['config'])
+model = GPT(config)
+model.load_state_dict(checkpoint['model'])
+model.eval()
+
+# Generate text (see sample_bio.py for full implementation)
+```
+
+## Training Details
+
+**Pretraining:**
+- Dataset: OpenWebText (~9B tokens)
+- Iterations: 600,000
+- Hardware: RTX 3070 (steps 0–354k) → RTX 5060 Ti 16GB (steps 354k–600k)
+- Time: Roughly 20 days total (~15 days on 3070, ~5 days on 5060 Ti)
+
+**Fine-tuning:**
+- Dataset: SmolTalk2 (406K examples)
+- Oscillation parameters (ω, φ) remained stable — 0.0% of neurons shifted by >0.1
+
+## Citation
+
+```bibtex
+@software{wigglegpt2025,
+  author = {O'Brien, Phillip C.},
+  title = {WiggleGPT: Revisiting the Monotonicity Assumption in Neural Networks via Oscillating Activation Functions},
+  year = {2025},
+  url = {https://github.com/Eden-Eldith/WiggleGPT}
+}
+```
+
+## Author
+
+**Eden (Phillip C. O'Brien)**  
+Independent AI Researcher | ORCID: [0009-0007-3961-1182](https://orcid.org/0009-0007-3961-1182)
+
+Built in a garage lab in Gosport, UK. No academic affiliation, no institutional funding — just curiosity and an RTX 3070.
+
+## License
+
+GPL-3.0 — if you build on this, keep it open source.