dendrite-23m
dendrite-23m is a from-scratch, CPU-optimised small language model built on a hierarchical sparse mixture-of-experts architecture (internally, "HT-LLM — Hierarchical Tree LLM"). It has no self-attention layers: instead, each token embedding is routed through a 3-level fixed routing tree down to one of 512 leaf micro-experts, whose outputs are projected back to the vocabulary.
This is a research / hobby-scale prototype (~23.7M total parameters, of which only a small top-k-selected slice is active per token), written entirely in C with hand-written AVX2/FMA SIMD kernels — no PyTorch/TensorFlow runtime is required for inference.
Model details Developed by: n-AI Research (Musab Küçük) Model type: Sparse hierarchical mixture-of-experts, decoder-only, no attention Language: English License: Apache-2.0 Implementation: pure C, SIMD-accelerated (AVX2 + FMA, with scalar fallback) Architecture tokens → embedding lookup → hierarchical routing tree → top-k micro-experts → output head → logits Component Shape Notes embeddings.weight (16384, 160) vocab=16,384, d_model=160 Routing tree 1 → 8 → 64 → 512 3-level fixed tree (root → 8 domains → 64 sub-domains → 512 leaf experts), branching factor 8 experts.{0..511}.W_in / b_in (160, 112) / (112,) per-expert input projection, 512 experts experts.{0..511}.W_out / b_out (112, 160) / (160,) per-expert output projection output_head.W_proj / b_proj (160, 16384) / (16384,) untied output projection Total parameters: ≈ 23.7M Experts: 512, each a small 2-layer FFN (160 → 112 → 160, GELU-family activation) Routing: tree-structured, dot-product + softmax at each level, top-k selection (default k=2 leaf experts active per token); the routing tree's centroid vectors are not learned — they are generated once from a fixed random seed and kept frozen, so they are not shipped as part of this checkpoint and are reconstructed deterministically at load time. Only the embedding table, the expert FFN weights, and the output head are trained. Context handling: autoregressive, no positional attention — position information (if any) enters only through the routing/embedding path, not through a self-attention mechanism.
Note: because the routing tree is not attention-based and its centroids are frozen rather than learned end-to-end, this model trades the flexibility of learned attention/routing for very cheap, predictable, branch-free CPU inference (O(log₈ N) routing cost).
Training
Trained from scratch (random initialization) using a hand-derived backward pass in C (no autograd). The routing tree is frozen throughout training; only embeddings, expert weights, and the output head receive gradient updates.
Training proceeded in two stages:
Base training on Veelane/TinyStories-small Fine-tuning on Abirate/english_quotes Intended use
This model is a small-scale research/engineering prototype meant to explore attention-free, sparse, CPU-only inference architectures. It is not intended for production text generation, is not instruction-tuned or RLHF-aligned, and has a small (16,384-token) vocabulary trained on narrow, small-scale datasets (short children's stories and short quotes). Expect limited coherence, factuality, and generalization outside of those domains.
Limitations No attention mechanism — long-range dependency modeling is limited to whatever the fixed routing tree and embedding path can express. Routing tree centroids are randomly initialized and frozen (not learned), so expert specialization is not guaranteed to align with semantic categories in any interpretable way. Trained on a small amount of narrow-domain data (short stories + quotes); not suitable for general-purpose or factual use. No safety fine-tuning / alignment has been applied. Hardware / inference
Reference implementation is written in C with AVX2 + FMA SIMD kernels and a scalar fallback path, targeting commodity x86-64 CPUs (no GPU required).