Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
license: mit
|
| 4 |
+
tags:
|
| 5 |
+
- symbiogenesis
|
| 6 |
+
- multi-organelle
|
| 7 |
+
- monarch-mixer
|
| 8 |
+
- philosophy
|
| 9 |
+
- pytorch
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# SymbioGPT-10M
|
| 13 |
+
|
| 14 |
+
Multi-organelle GPT language model (11.6M params) trained on classical philosophy texts.
|
| 15 |
+
|
| 16 |
+
## Architecture
|
| 17 |
+
|
| 18 |
+
**SymbioGPT** extends the [SymbioSLM](https://huggingface.co/LisaMegaWatts/SymbioSLM) architecture by adding CausalSelfAttention as a 4th organelle, all fused via a learned per-channel OrganelleGate with learnable temperature.
|
| 19 |
+
|
| 20 |
+
| Organelle | Function | Complexity |
|
| 21 |
+
|-----------|----------|------------|
|
| 22 |
+
| CausalDepthwiseConv1d | Local n-gram detection | O(n) |
|
| 23 |
+
| MonarchMatrix | Sub-quadratic global mixing via butterfly matrices | O(n√n) |
|
| 24 |
+
| LongConv | Dense causal convolution with exponential decay | O(n) |
|
| 25 |
+
| CausalSelfAttention | Multi-head attention with RoPE | O(n²) |
|
| 26 |
+
|
| 27 |
+
Plus: RMSNorm, SwiGLU FFN, SkipGate residuals, weight-tied output projection.
|
| 28 |
+
|
| 29 |
+
## Model Details
|
| 30 |
+
|
| 31 |
+
| Parameter | Value |
|
| 32 |
+
|-----------|-------|
|
| 33 |
+
| d_model | 320 |
|
| 34 |
+
| n_layers | 8 |
|
| 35 |
+
| n_heads | 5 |
|
| 36 |
+
| head_dim | 64 |
|
| 37 |
+
| context_length | 256 |
|
| 38 |
+
| vocab_size | 2000 (BPE) |
|
| 39 |
+
| n_monarch_heads | 1 |
|
| 40 |
+
| Total params | 11.6M |
|
| 41 |
+
|
| 42 |
+
## Files
|
| 43 |
+
|
| 44 |
+
- `symbio_best.pt` — Best checkpoint (PyTorch state_dict, torch.compile format)
|
| 45 |
+
- `symbio_final.pt` — Final checkpoint
|
| 46 |
+
- `vocab.json` — BPE vocabulary (2000 tokens)
|
| 47 |
+
- `merges.txt` — BPE merge rules
|
| 48 |
+
|
| 49 |
+
## Usage
|
| 50 |
+
|
| 51 |
+
Try it live: [SymbioGPT-10M Space](https://huggingface.co/spaces/LisaMegaWatts/SymbioGPT-10M-space)
|
| 52 |
+
|
| 53 |
+
## Links
|
| 54 |
+
|
| 55 |
+
- **Source**: [DavinciDreams/symbiogenesis](https://github.com/DavinciDreams/symbiogenesis)
|
| 56 |
+
- **Training data**: Classical philosophy corpus (Aristotle, Plato, Seneca, Marcus Aurelius, etc.)
|