README.md

---
license: mit
language:
  - en
tags:
  - neural-cellular-automata
  - nca
  - 3d-brain
  - language-generation
  - bio-inspired
  - alternative-architecture
  - pytorch
  - research
library_name: pytorch
pipeline_tag: text-generation
model-index:
  - name: nca3d-brain-v5
    results:
      - task:
          type: text-generation
          name: Next Word Prediction
        dataset:
          type: custom
          name: Multi-dataset (TinyStories, WikiText, BookCorpus, IMDB, Q&A)
        metrics:
          - type: accuracy
            value: 10.7
            name: Eval Word Accuracy (30K vocab)
          - type: accuracy
            value: 16.9
            name: Train Word Accuracy (30K vocab)
---

# NCA 3D Brain — Neural Cellular Automata for Language

**A radically different neural architecture: a 3D grid of "mini-neurons" that learns language through local wave propagation, inspired by protein folding and biological neural communication.**

<p align="center">
  <img src="images/grid_depth_slices.png" alt="Wave propagation through the 3D brain" width="800"/>
  <br>
  <em>Information propagating as waves through the 3D grid — from input (z=0) to output (z=15)</em>
</p>

## What is this?

A cube of 4,096 "mini-neurons" (16×16×16 grid) where each cell only communicates with its immediate neighbors. Information travels as waves through the cube. No attention mechanism (unlike Transformers), no sequential layers — it's a mathematical organism where language understanding **emerges** from local signal propagation.

**35.4M parameters. Generates coherent English phrases of 6+ words.**

This is **not** a Transformer. This is not an RNN. This is a **3D cellular automaton** that learned to speak.

## Key Results

| Metric | Value |
|--------|-------|
| Eval word accuracy | **10.7%** (over 30K vocabulary) |
| Train word accuracy | 16.9% |
| Train/eval gap | **1.6x** (best generalization of all versions) |
| Longest coherent output | "she started to play together again" (6 words) |
| Parameters | 35.4M |
| Model size | 68 MB |
| Grid | 16×16×16 = 4,096 cells |
| Cell dimension | 256 |
| Training | 16 epochs, ~10.7h on NVIDIA B200 |

### Grammar emergence

The model learned grammatical categories without explicit supervision:

```
"the" + _        → nouns/adjectives (correct category)
"the big" + _    → nouns ("dog", "house", "girl")
"the dog" + _    → verbs ("was", "ran", "had")
"she wanted" + _ → "to" (infinitive structure)
```

### Best generations

```
"she started to play together again"
"the little girl wanted to play with her parents"
"he said that he was very happy"
"in the morning she went to the garden"
```

## Architecture

```
INPUT (face z=0)          THINKING (interior)         OUTPUT (face z=15)
┌─────────────┐          ┌─────────────┐             ┌─────────────┐
│ Tokens are  │   →→→    │ 3D waves    │    →→→      │ Prediction  │
│ injected    │  waves   │ propagate   │   waves     │ is read     │
│ into skin   │          │ N steps     │             │ from pole   │
└─────────────┘          └─────────────┘             └─────────────┘
```

### How it works

1. **Injection**: Input tokens are embedded and injected into the z=0 face of the cube
2. **Propagation**: For N steps, each cell updates based on its 26 neighbors via Conv3d
3. **Reading**: The opposite face (z=15) is average-pooled and projected to vocabulary logits

### Key innovations

- **Dilated convolutions** with cycle [1, 2, 4, 8] — in 4 steps each cell "sees" the entire grid
- **Synaptic fatigue**: 2 dedicated channels that inhibit over-firing cells, preventing repetition
- **Dual chemical pathways**: Standard Conv3d + depthwise-separable Conv3d (diversity in transition rules)
- **Gated residual updates**: Each cell decides how much to change per step via sigmoid gate

### Emergent phenomena

<p align="center">
  <img src="images/brain_map_3d_diff.png" alt="3D brain activation map" width="500"/>
  <br>
  <em>3D brain map showing where "good" sentences activate more (red) vs "bad" sentences (blue)</em>
</p>

- **Functional hemispheres**: x=12 region produces better language than x=6
- **3 thinking phases**: chaos (steps 1-5) → eureka (6-7) → decision (8-15)
- **Grammar in center, semantics in periphery** of the grid
- **Semantic clustering**: animals, family, nature, objects form distinct spatial clusters
- **Emotion highway**: emotional content activates a specific depth layer (z=12)

<p align="center">
  <img src="images/channel_map_grammar_vs_semantic.png" alt="Grammar vs Semantic channels" width="700"/>
  <br>
  <em>Channel specialization: grammar channels (red) vs semantic channels (blue) — the model spontaneously separated syntax from meaning</em>
</p>

## Quick Start

```python
import torch
import torch.nn.functional as F
import json
from model import NCA3D_Fatigue

# Load dictionary
word2num = {k: int(v) for k, v in json.load(open("word_dictionary_30k.json")).items()}
num2word = {v: k for k, v in word2num.items()}

# Load model
model = NCA3D_Fatigue()
model.load_state_dict(torch.load("model_phase4c_v5_fatigue_best.pt", map_location="cpu"))
model.eval()

# Predict next word
context = ["the", "little", "girl"]
ids = [word2num[w] for w in context]
with torch.no_grad():
    logits = model(torch.tensor([ids]), n_steps=15)
    pred_id = logits.argmax(-1).item()
    print(f"'{' '.join(context)}' → '{num2word.get(pred_id, '?')}'")

# Generate a sequence
from inference import generate
print(generate(model, word2num, num2word, ["she", "wanted", "to"], max_words=8))
```

## Model Architecture Details

```
Component                    Shape                          Params
────────────────────────────────────────────────────────────────────
word_embed                   Embedding(30006, 384)          11.5M
embed_proj                   Linear(384, 256)               98K
pos_embed                    Embedding(52, 256)             13K
init_state                   (1, 256, 16, 16, 16)           1.05M
trans1.conv1 (dilated)       Conv3d(256→512, k=3³)          3.5M
trans1.conv2 (dilated)       Conv3d(512→256, k=3³)          3.5M
trans2.dw_conv (dilated)     Conv3d(256→256, k=3³, groups)  6.9K
trans2.pw_conv               Conv3d(256→256, k=1)           65K
gate_conv                    Conv3d(256→256, k=1)           65K
norm (GroupNorm)              32 groups, 256 ch              512
out_proj                     256→512→30006                  15.6M
────────────────────────────────────────────────────────────────────
TOTAL                                                       ~35.4M
```

## Training Details

- **Base model**: Continued from v4 (dilated Conv3d + 30K vocab)
- **Datasets**: WikiText-103, TinyStories, BookCorpus, IMDB, ROCStories, CNN/DailyMail, TriviaQA, Natural Questions, ELI5 (10 datasets total)
- **Schedule**: 3 phases — 8ep×750K (aggressive) + 5ep×500K (consolidation) + 3ep×350K (refinement)
- **Learning rates**: 5e-4→3e-4 | 2e-4→1e-4 | 8e-5→3e-5
- **Hardware**: NVIDIA B200, 178GB VRAM peak
- **Training time**: ~10.7 hours
- **Loss**: Cross-entropy on 30K word vocabulary, multi-step loss (steps 7-16)

## Project History

This model is the result of an extensive research journey:

| Phase | What | Result |
|-------|------|--------|
| 1 | Arithmetic (8³ grid, 499K params) | 98.4% on unseen data |
| 2A | 15 semantic relations | 98.2% test, 87.5% generalization |
| 2B | 100 semantic relations | 73.4% test (85.5% without "similar") |
| 2B-v3 | 184 relations (grammar + semantics) | 93.5% overall |
| 3B | Q&A from relations | 85% direct, 75% novel |
| 3C | Transitive reasoning | 52.5% holdout, 83.3% novel chains |
| 4 | Language as arithmetic | 50.2% char accuracy, grammar emerges |
| 4B | Multi-step loss | 55.4% char accuracy |
| 4C-v1→v4 | Word embeddings, dilated conv, 30K vocab | Incremental improvements |
| **4C-v5** | **Synaptic fatigue + intensive training** | **10.7% eval, 6+ word coherence** |

**113 documented discoveries** across all phases.

## Why this matters

Transformers dominate NLP, but they have fundamental constraints:
- O(n²) attention complexity
- Fixed depth (always N layers regardless of problem difficulty)
- No spatial locality between neurons
- Billions of parameters required

NCA 3D Brain shows that **local communication + iterative propagation** can produce language-like behavior with:
- O(n) complexity (each cell only sees neighbors)
- Variable thinking depth (more steps = more reasoning)
- Spatial structure with emergent functional zones
- Orders of magnitude fewer parameters

This is early-stage research. The model doesn't compete with Transformers on quality — but it demonstrates that a fundamentally different computational paradigm can learn language structure.

## Limitations

- Accuracy is low compared to any Transformer (10.7% next-word prediction on 30K vocab)
- Autoregressive generation accumulates errors — quality degrades after 6-8 words
- Embeddings are partially disorganized (Zipf's law — rare words get few updates)
- No extrapolation to longer contexts than trained on
- CPU inference only (no optimized CUDA kernels)

## Citation

```bibtex
@misc{quintela2026nca3d,
  title={NCA 3D Brain: Neural Cellular Automata for Language Processing},
  author={Cristian Quintela},
  year={2026},
  url={https://huggingface.co/killking69/nca3d-brain-v5}
}
```

## Author

**Cristian Quintela**

## License

MIT