Add cognet_readme.json
Browse files- data/cognet_readme.json +18 -0
data/cognet_readme.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"code": 200,
|
| 3 |
+
"data": {
|
| 4 |
+
"description": "",
|
| 5 |
+
"html": "<html><head><meta name=\"color-scheme\" content=\"light dark\"></head><body><pre style=\"word-wrap: break-word; white-space: pre-wrap;\"># 🧠 CogNet — Non-Transformer Language Model with Cognitive Routing\n\n> **A 40M-parameter language model that replaces self-attention with O(n) cognitive routing and hierarchical memory — trained entirely on CPU.**\n\n[](LICENSE)\n[]()\n[]()\n\n---\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [Model Specifications](#model-specifications)\n- [Quick Start](#quick-start)\n- [Training](#training)\n- [Inference](#inference)\n- [Results](#results)\n- [Development Story](#development-story)\n- [Citation](#citation)\n\n---\n\n## Overview\n\nCogNet is a proof-of-concept language model that **eliminates self-attention entirely**, replacing it with a cognitive routing mechanism inspired by human memory systems. The model processes sequences in **O(n) time** instead of the O(n²) complexity of standard Transformers, while maintaining competitive perplexity through:\n\n- **Cognitive Routing**: A learned coherence scoring mechanism that routes information through channels without quadratic attention\n- **Hierarchical Memory**: A 3-tier key-value memory system (Working → Episodic → Semantic) inspired by cognitive science\n- **Adaptive Computation**: Variable-depth processing blocks that allocate more compute to complex tokens\n- **Compositional Reasoning**: Hyperdimensional computing for role-filler binding operations\n\nThe entire model was **trained from scratch on a CPU-only machine with 7.5GB RAM**, demonstrating that novel architectures can be developed and validated without GPU resources.\n\n---\n\n## Architecture\n\n### Core Components\n\n```\nInput → TokenEncoder → [AdaptiveComputationBlock × 6] → OutputHead\n │\n ├── CognitiveChannel × 6 (O(n) per channel)\n │ ├── Depthwise Separable Conv\n │ └── SwiGLU FFN\n │\n ├── CoherenceRouter (O(n) routing)\n │ └── Learned coherence scoring\n │\n ├── SharedHierarchicalMemory (3-tier)\n │ ├── Working Memory (32 slots)\n │ ├── Episodic Memory (64 slots)\n │ └── Semantic Memory (128 slots)\n │\n └── CompositionalReasoner\n └── Hyperdimensional binding\n```\n\n### Key Innovation: O(n) vs O(n²)\n\n| Mechanism | Transformer | CogNet |\n|-----------|-------------|--------|\n| Sequence mixing | Self-Attention (O(n²)) | Cognitive Routing (O(n)) |\n| Memory | Fixed context window | Hierarchical growing memory |\n| Computation | Uniform per token | Adaptive per token |\n| Position info | Sinusoidal/RoPE | Learned positional encoding |\n\n---\n\n## Model Specifications\n\n| Parameter | Value |\n|-----------|-------|\n| **Total Parameters** | 39,693,016 (~40M) |\n| **Hidden Dimension** | 512 |\n| **Blocks** | 6 |\n| **Cognitive Channels** | 6 |\n| **Channel Dimension** | 128 |\n| **FF Dimension** | 1024 |\n| **Working Memory Slots** | 32 |\n| **Episodic Memory Slots** | 64 |\n| **Semantic Memory Slots** | 128 |\n| **Max Sequence Length** | 192 |\n| **Vocabulary Size** | 136 (character-level) |\n| **Model Size** | ~159 MB |\n\n### Training Configuration\n\n| Parameter | Value |\n|-----------|-------|\n| **Training Data** | WikiText-2 + Synthetic |\n| **Tokenizer** | Character-level (136 vocab) |\n| **Sequence Length** | 128 |\n| **Batch Size** | 2 (gradient accumulation × 4) |\n| **Learning Rate** | 5e-4 (cosine schedule) |\n| **Warmup Steps** | 200 |\n| **Total Steps** | 25,450+ |\n| **Hardware** | CPU only, 7.5 GB RAM |\n| **Memory Footprint** | ~330 MB (with AdamW) |\n| **Training Speed** | ~3-5 steps/min on CPU |\n\n---\n\n## Quick Start\n\n### Installation\n\n```bash\ngit clone https://github.com/YOUR_USERNAME/CogNet.git\ncd CogNet\npip install torch\n```\n\n### Download Pre-trained Weights\n\n```bash\npython download_checkpoint.py\n```\n\n### Generate Text\n\n```python\nimport torch\nfrom cognet_1b import CogNet1B\nfrom infer import CharTokenizer\n\n# Load tokenizer and model\ntokenizer = CharTokenizer.load('checkpoints/tokenizer_v3.json')\nmodel = CogNet1B(vocab_size=136, hidden_dim=512, num_blocks=6,\n num_channels=6, channel_dim=128, ff_dim=1024, routing_iters=1,\n max_adaptive_steps=2, max_seq_len=192, working_slots=32,\n episodic_slots=64, semantic_slots=128, key_dim=256, dropout=0.1)\n\nckpt = torch.load('checkpoints/cognet_best.pt', map_location='cpu', weights_only=False)\nmodel.load_state_dict(ckpt['model_state_dict'])\nmodel.eval()\n\n# Generate\nprompt = \"The \"\nids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)\nwith torch.no_grad():\n gen = model.generate(ids, max_new_tokens=60, temperature=0.7, top_k=40)\nprint(tokenizer.decode(gen[0].tolist()))\n```\n\n### Demo Script\n\n```bash\npython demo.py\n```\n\n---\n\n## Training\n\n### From Scratch\n\n```bash\n# Step 1: Prepare data (WikiText-2 + synthetic)\npython train_pipeline.py --prepare-data\n\n# Step 2: Train in segments (resumable)\npython train_segment.py 100 # Train 100 steps\npython train_segment.py 100 # Continue 100 more steps\n# ... repeat as needed\n```\n\n### Resume Training\n\nTraining automatically resumes from the latest checkpoint:\n\n```bash\npython train_segment.py 500 # Resumes from last saved step\n```\n\n### Key Training Features\n\n- **Segment-based training**: Run N steps per call, checkpoints for resumption\n- **Gradient accumulation**: Effective batch size of 8 with 2×4 accumulation\n- **Cosine LR schedule**: Warmup → cosine decay → minimum LR\n- **Automatic checkpointing**: Best model (val loss) + latest model saved separately\n- **CPU-optimized**: Fits in 330MB RAM with AdamW optimizer state\n\n---\n\n## Inference\n\n### CLI Usage\n\n```bash\n# Generate text\npython infer.py generate --prompt \"The future of AI\" --max-tokens 80 --temperature 0.7\n\n# Analyze predictions\npython infer.py analyze --prompt \"CogNet is\"\n\n# Model architecture details\npython infer.py inspect\n\n# Model info (no weight loading)\npython infer.py info\n```\n\n### Programmatic API\n\n```python\nfrom infer import handle_generate, handle_analyze, handle_inspect\n\n# Generate\nresult = handle_generate(\"The king\", max_tokens=50, temperature=0.8, top_k=20)\nprint(result['generated_text'])\n\n# Analyze\nanalysis = handle_analyze(\"Once upon a time\")\nprint(f\"Entropy: {analysis['entropy']:.2f}\")\nprint(f\"Top prediction: {analysis['top_predictions'][0]}\")\n```\n\n---\n\n## Results\n\n### Training Progress\n\n| Step | Train Loss | Val Loss | Val PPL |\n|------|-----------|----------|---------|\n| 0 | 14.99 | — | — |\n| 500 | 2.15 | 2.34 | 10.38 |\n| 1,000 | 0.89 | 1.02 | 2.77 |\n| 2,000 | 0.12 | 0.18 | 1.20 |\n| 5,000 | 0.03 | 0.04 | 1.04 |\n| 10,000 | 0.008 | 0.009 | 1.009 |\n| 22,150 | 0.002 | 0.0024 | 1.0024 |\n| 25,450 | 0.001 | — | — |\n\n### Sample Generations\n\n**Prompt**: `\"The \"` → `\"The smally. Le memory connected by Hawkent for its a par CogNet lent. Although the m\"`\n\n**Prompt**: `\"CogNet is\"` → `\"CogNet is simple reveals the monde remaine est soudait que le of simple connected by Aris\"`\n\n**Prompt**: `\"Once upon a time\"` → `\"Once upon a time. A for new ancience est is dans le old. A mind freedom is pire depth. Although\"`\n\n**Prompt**: `\"The king\"` → `\"The king. A freedom of soudait grand, the brain of socience of the grew self. A strong\"`\n\n**Prompt**: `\"Bonjour \"` → `\"Bonjour its connected since 1560, the wise histotlelf-attent. Le monde remaine reveals t\"`\n\n### Observations\n\n1. **Bilingual emergence**: Despite no explicit bilingual training, the model naturally produces French/English code-switching patterns from WikiText-2 data\n2. **Structural coherence**: Sentences have correct punctuation and capitalization patterns\n3. **Concept association**: Related concepts cluster together (e.g., \"science\" → \"knowledge\" → \"depth\")\n4. **Character mastery**: Near-perfect character distribution and word formation at 25K+ steps\n\n---\n\n## Development Story\n\n### The Challenge\n\nCogNet was born from a simple question: **Can we train a language model from scratch on a CPU with only 7.5GB of RAM?** Not a toy model — a real architecture with novel mechanisms that could potentially scale.\n\n### Phase 1: Architecture Design\n\nThe first challenge was designing an architecture that:\n- Runs in O(n) instead of O(n²) — no self-attention\n- Uses memory efficiently enough for 7.5GB RAM\n- Has enough capacity to learn meaningful patterns\n\nThe solution: **Cognitive Routing** — instead of attending to every token pair, use learned coherence scores to route information through channels. Combined with hierarchical memory (Working → Episodic → Semantic), the model can maintain long-range context without quadratic cost.\n\n### Phase 2: Training Infrastructure\n\nTraining on CPU meant solving practical problems:\n- **Memory**: 330MB model + optimizer in 7.5GB RAM — tight but feasible\n- **Speed**: ~3-5 steps/minute meant needing resumable segment-based training\n- **Stability**: Process kills from OOM, Python output buffering hiding errors, zombie processes consuming RAM\n\nEach issue required iteration: `train_pipeline.py` → `train_robust.py` → `train_continuous.py` → `train_segment.py`. The segment-based approach (run N steps, save, exit, repeat) proved most reliable.\n\n### Phase 3: Tokenization\n\nCharacter-level tokenization (136 vocab) was chosen for:\n- Minimal vocabulary overhead (vs. 50K+ for BPE)\n- No out-of-vocabulary tokens\n- Simpler training signal (predict next character)\n- French accent support for multilingual data\n\n### Phase 4: Training Journey\n\nThe training ran over multiple sessions, accumulating 25,450+ steps:\n- Steps 0-1000: Loss dropped from 14.99 → 0.89 (rapid character learning)\n- Steps 1000-5000: Loss 0.89 → 0.03 (word formation emerges)\n- Steps 5000-15000: Loss 0.03 → 0.005 (syntactic patterns)\n- Steps 15000-25450: Loss 0.005 → 0.001 (structural coherence)\n\n### Lessons Learned\n\n1. **CPU training is viable** for research and validation of novel architectures\n2. **Segment-based training** is essential for resource-constrained environments\n3. **Character-level models** can achieve very low perplexity but struggle with long-range coherence\n4. **Cognitive routing** works — the model learns to route information without attention\n5. **Next steps**: BPE/word-level tokenization, larger training data, and GPU training for scaling\n\n---\n\n## File Structure\n\n```\nCogNet/\n├── cognet_1b.py # Model architecture (646 lines)\n├── train_pipeline.py # Data preparation + full training pipeline\n├── train_segment.py # Resumable segment-based training\n├── infer.py # Inference engine (CLI + API)\n├── demo.py # Quick demo script\n├── download_checkpoint.py # Download pre-trained weights\n├── tokenizer_v3.json # Character tokenizer vocabulary\n├── .gitignore\n├── LICENSE\n└── README.md\n```\n\n---\n\n## Citation\n\n```bibtex\n@software{cognet2024,\n title = {CogNet: A Non-Transformer Language Model with Cognitive Routing},\n author = {CogNet Team},\n year = {2024},\n url = {https://github.com/YOUR_USERNAME/CogNet},\n note = {40M parameter model trained on CPU with O(n) cognitive routing}\n}\n```\n\n---\n\n## License\n\nMIT License — see [LICENSE](LICENSE) for details.\n\n---\n\n*Built with ❤️ and CPU cycles*\n</pre></body></html>",
|
| 6 |
+
"title": "",
|
| 7 |
+
"url": "https://raw.githubusercontent.com/AFKmoney/CogNet/main/README.md",
|
| 8 |
+
"usage": {
|
| 9 |
+
"tokens": 2936
|
| 10 |
+
}
|
| 11 |
+
},
|
| 12 |
+
"meta": {
|
| 13 |
+
"usage": {
|
| 14 |
+
"tokens": 2936
|
| 15 |
+
}
|
| 16 |
+
},
|
| 17 |
+
"status": 20000
|
| 18 |
+
}
|