SymbolicLight-PoC

SymbolicLight: A natively-trained Neuro-Symbolic Spiking Language Architecture. Replaces dense attention with SparseTCAM and LIF neurons for 87-91% activation sparsity. (168M PoC Snapshot)

📢 Release Notice: Inference-Only Snapshot

This repository contains the inference-only snapshot of the SymbolicLight-PoC architecture (168M parameters). We provide the full model definition (model.py), autoregressive generation scripts (generate.py), and pre-trained weights (best.pt) to allow the community to verify the 90% activation sparsity, the logic of SparseTCAM routing, and the effectiveness of the Bayesian Decoding Head.

A Note on Training Infrastructure: Training spiking language models natively from scratch involves extremely non-trivial surrogate gradient calibration, customized optimizer scheduling, and distributed stabilization techniques. As we are actively advancing toward the 1B+ scale, these proprietary training infrastructures, scripts (train.py), and dynamic datasets are excluded from this initial public codebase.

---

A Next-Generation Neuro-Symbolic Spiking Language Model

SymbolicLight is not just another connectionist deep learning model; it is a true Neuro-Symbolic system natively fused at the lowest architectural level.

We reject the superficial "LLM + external calculator/knowledge graph" paradigm. Instead, we weave continuous connectionist networks with discrete symbolic logic systems at the fundamental computational unit:

• Neuro (The Connectionist Engine): Retains biological generalization and learning capabilities based on synaptic plasticity (STDP), enabling it to process fuzzy, high-dimensional natural language representations and continuously evolve during inference.
• Symbolic (The Logic Controller): Utilizes discrete binary spikes (0/1) as information carriers. This inherent Boolean logic directly triggers deterministic content-addressable memory (SparseTCAM) in the backend—replacing probabilistic attention—and forcibly injects rule-based conditional branching via an EntropyGate and a Bayesian Head.

Through this deep fusion, SymbolicLight delivers profound cognitive reasoning alongside transparent interpretability and a theoretical 100x leap in hardware execution efficiency on edge devices.

Key Innovations

1. SparseTCAM Routing: We completely abandon the $O(n^2)$ probabilistic self-attention of traditional Transformers. Instead, we deploy $O(n \cdot k)$ Sparse Content-Addressable Memory, routing signals deterministically based on explicit Boolean logic.
2. LIF Neurons (Ultra-Sparse Spiking Engine): Event-driven neuron clusters maintain an up to 90% resting rate (zero spikes) during inference, entirely bypassing massive dense matrix multiplications.
3. EntropyGate (White-Box Logic Control): Introduces If-Else conditional branching into the deepest layers of the neural network. Low-entropy tokens trigger rule-based "Early Exits," avoiding redundant computation.
4. Bayesian Head: Replaces traditional blind-guessing Softmax with deterministic statistical inference, utilizing robust prior and posterior confidence boundaries.
5. STDP Online Learning: Achieves maintenance-free lifelong learning at the edge. Neurons naturally reshape weights based on spike timing during inference, completely eliminating the need for backpropagation gradients.

Usage

All commands below should be run from the project root directory (the folder containing model.py, generate.py, etc.).

Prerequisites

pip install torch transformers gradio

Validate Model Performance

Evaluate perplexity and activation sparsity on the TinyStories validation set:

python validate.py --checkpoint best.pt

Interactive Text Generation

Launch the CLI-based autoregressive generator:

python generate.py

Web Demo

Start the Gradio-based web interface for visual, interactive testing:

python web_demo.py

---