---
language: en
license: mit
tags:
  - symbiogenesis
  - multi-organelle
  - monarch-mixer
  - philosophy
  - pytorch
---

# SymbioGPT-10M

Multi-organelle GPT language model (11.6M params) trained on classical philosophy texts.

## Architecture

**SymbioGPT** extends the [SymbioSLM](https://huggingface.co/LisaMegaWatts/SymbioSLM) architecture by adding CausalSelfAttention as a 4th organelle, all fused via a learned per-channel OrganelleGate with learnable temperature.

| Organelle | Function | Complexity |
|-----------|----------|------------|
| CausalDepthwiseConv1d | Local n-gram detection | O(n) |
| MonarchMatrix | Sub-quadratic global mixing via butterfly matrices | O(n√n) |
| LongConv | Dense causal convolution with exponential decay | O(n) |
| CausalSelfAttention | Multi-head attention with RoPE | O(n²) |

Plus: RMSNorm, SwiGLU FFN, SkipGate residuals, weight-tied output projection.

## Model Details

| Parameter | Value |
|-----------|-------|
| d_model | 320 |
| n_layers | 8 |
| n_heads | 5 |
| head_dim | 64 |
| context_length | 256 |
| vocab_size | 2000 (BPE) |
| n_monarch_heads | 1 |
| Total params | 11.6M |

## Files

- `symbio_best.pt` — Best checkpoint (PyTorch state_dict, torch.compile format)
- `symbio_final.pt` — Final checkpoint
- `vocab.json` — BPE vocabulary (2000 tokens)
- `merges.txt` — BPE merge rules

## Usage

Try it live: [SymbioGPT-10M Space](https://huggingface.co/spaces/LisaMegaWatts/SymbioGPT-10M-space)

## Links

- **Source**: [DavinciDreams/symbiogenesis](https://github.com/DavinciDreams/symbiogenesis)
- **Training data**: Classical philosophy corpus (Aristotle, Plato, Seneca, Marcus Aurelius, etc.)