--- license: mit language: en tags: - hyperdimensional-computing - vector-symbolic-architectures - associative-memory - training-free - kuramoto - ensemble-methods - cpu - no-gpu - memory-augmented - bpe library_name: ensemble --- # ENSEMBLE — a training-free AI built from portable experts > **No gradient. No epochs. No GPU.** Turn any dataset into a compressed expert file in one pass, then let a Kuramoto-coupled brain synchronize experts into emergent answers. ENSEMBLE thinks when idle, **grows a persistent central memory**, and safely rewrites its own coupling. It is built on the [Palimpseste](https://huggingface.co/thefinalboss/palimpseste-max) hypervector substrate. ENSEMBLE is **not** a transformer and is not trying to be one. It is an experiment in a different direction: a training-free, ultra-compact, instantly-updateable, composable associative memory that grows smarter over time without retraining. This README is precise about what it is, what it does well, and where it loses to a trained 1B transformer. ## What's new in v0.2 - **Global shared BPE tokenizer** — one tokenizer across all experts lifts the char-level quality ceiling. On TinyStories (150 KB excerpt, D=5000): **next-token accuracy 94.5% → 98.5%**, **query latency 112ms → 28ms (4× faster)**, **RAM 187MB → 89MB (2× less)**. - **Persistent central brain memory** — the brain now accumulates synthetic concepts (from continuous thinking) in a memory that survives `save()` / `load()`. The system grows indefinitely: add experts + let it think, no retraining. - **Brain save/load** — full brain state persistence. - 75 tests passing (was 49). ## What's new in v0.3 — structural generalization - **Structural query encoding** — decompose a question into a PATTERN + a SLOT via explicit templates (`Expert.from_qa_pairs(..., patterns=["what is the capital of {country}"])`). Questions sharing a pattern are encoded consistently, so the answer **format generalizes to unseen slots**. - **Measured generalization lift**: on capitals (train: france/germany/italy/japan/egypt; holdout: spain/portugal/greece/brazil/norway/india/mexico/kenya), the char-level baseline returns an **empty answer 100% of the time** on unseen slots; the structural encoder returns a **non-empty answer 100% of the time**. That is **+100 points** of graceful degradation on the unknown. - **Honest scope**: this guarantees a well-formed answer shape, *not* factual correctness for a fully unseen slot. Without semantic embeddings, "spain" and "france" are orthogonal at the HV level, so the system imitates structure rather than interpolating meaning. The README is explicit about this. - 91 tests passing (was 75). ## What's new in v0.4 — embedding expert (optional semantics) - **Embedding expert** — an optional plugin that brings real **semantic similarity** to the brain. Dense word vectors (fastText, or a local mini-embedding) are projected into HV space via a seeded random-projection-then-sign (the new `projection.py` primitive), so `spain` and `portugal` become **similar hypervectors** (cosine-preserving, JL-style). This is exactly the ingredient the structural encoder needed for analogy to *find the right slot*. - **It's a pure plugin**: `EmbeddingExpert` satisfies the same duck-typed contract as `Expert` (relevance / candidate_hv / answer / signature_hv), so it plugs into the Brain and couples with lexical experts via Kuramoto **with zero changes to the brain**. The training-free / numpy-only core stays pure; semantics is opt-in. - **Two construction paths**: `from_corpus_local` (PPMI+SVD mini-embedding, training-free, numpy only, seconds — lower quality) and `from_fasttext` (load pretrained `.vec`, best quality, one-time download). - **Honest result**: the embedding expert measurably brings semantic similarity (`spain~portugal = 0.18` vs `~0` without), which improves routing. However, with the **mini-embedding** the factual-correctness on the capitals holdout remains ~0% — the slot is found but the answer content is still decoded char-level. The pipeline is ready for fastText to deliver real factual gains; that's the documented next step. - 106 tests passing (was 91). ## What's new in v0.5 — factual generalization via dense analogy - **Dense analogy solving** — `EmbeddingExpert.learn_relation("capital_of", {france: paris, ...})` + `solve_analogy("capital_of", "spain")` implements the classic Mikolov analogy (`paris − france + spain ≈ madrid`) in normalized dense space, averaging the `(answer − slot)` direction over known examples and finding the nearest word — **excluding the query slot** so it doesn't parrot itself. - **Wired into structural answers** — when an embedding expert is attached to a lexical expert (`from_qa_pairs(patterns=..., embedding=emb)`), unseen slots are answered via dense analogy instead of char-level decode, returning the **plain-text** answer. - **Measured factual gains (fastText, end-to-end via the Brain)**: on a capitals holdout (train: france/germany/italy/japan/egypt/russia; holdout: spain/portugal/greece/norway/china/india/brazil/turkey), factual correctness goes from **0% (char) / 0% (structural-only) to ~50–67% (structural + embedding analogy)**. Concretely: `spain → madrid`, `portugal → lisbon`, `greece → athens`, `norway → oslo` are recovered correctly despite never being in training. - **Morphological filtering** (default on): excludes candidates sharing the query slot's stem (filters `indian`, `turkeys`, `brazilian`), which empirically recovers `india → delhi` (50% → 62% on the isolated analogy benchmark). Two scoring methods exposed: `cosadd` (default, Mikolov 3CosAdd) and `cosmul` (Levy & Goldberg 3CosMul) — measured `cosadd+morph` to be the best on this benchmark. - **Honest scope**: fastText analogies are imperfect (china → porcelain on a miss) — that's fastText's distributional ceiling, not the architecture. More known pairs and larger vocab would lift it further. The point stands: the wall is broken. - 109 tests passing (was 106). --- ## The four ideas ### 1. Dataset → Expert (`.exp`) — one pass, no gradient, smaller than the source Any dataset becomes a frozen, portable expert file in a single pass. There is no gradient descent and no epoch — learning is an `O(1)`-per-token write into an append-only memory. The expert file is **smaller than the dataset** (typically 3–9×): hypervectors are never stored; the symbolic token stream is gzipped and HVs are rebuilt on load from a deterministic encoder. ### 2. Kuramoto brain — experts couple and synchronize into emergent answers Load several experts into a **Brain**. Each expert becomes a Kuramoto oscillator (natural frequency = relevance, coupling = signature similarity). The oscillators synchronize by similarity; the emergent attractor is a state **no single expert produced**. The brain routes questions to the right expert or composes answers when two experts know the topic. ### 3. Global shared BPE + persistent central memory — the brain grows A single BPE tokenizer makes all experts token-compatible (so they share the same vocabulary and can be assembled freely). Meanwhile the brain's **central memory** accumulates synthetic concepts discovered during continuous thinking — and unlike a transformer's frozen weights, this memory **persists and grows**. Adding knowledge is Lego: drop in a `.exp`, let the brain think, and its concept store expands. No joint retraining. ### 4. Continuous thought + safe self-modification When idle, the brain **thinks**: it samples remembered queries from one expert, asks all the others, runs the attractor, and writes novel coherent results into the central memory. It can also **rewrite its own coupling** under a **Lyapunov constraint** (ΔE ≤ 0). The acceptance rule is immutable — the recursion is bounded by construction. --- ## How it differs from an LLM | | Transformer LLM | ENSEMBLE | |---|---|---| | Learning | gradient descent, ~1T tokens | one-pass write, no gradient | | Adding knowledge | retrain / fine-tune / RAG | drop in a `.exp` (Lego) + think | | Model size vs data | grows with parameters | expert is **smaller** than its data | | Grows over time | frozen weights | **central memory accumulates concepts** | | Idle behavior | nothing | dreams across experts, writes concepts | | Self-tuning | hyperparameters fixed | rewrites its own coupling (Lyapunov-bound) | | Hardware | GPU | plain CPU | --- ## Quick start This repo is **self-contained**: it vendors the `palimseste` substrate, so a fresh clone runs with only `numpy`. ```bash git clone https://huggingface.co/thefinalboss/ensemble cd ensemble pip install numpy # the only runtime dependency python -c "from ensemble import Brain, Expert; print('ok')" ``` ### Build experts and assemble a brain ```python from ensemble import Expert, Brain # Option A: char-level (backward compatible, simplest) math = Expert.from_qa_pairs( [("what is pi", "pi is approximately three point one four")] * 5, domain="math", D=10000) # Option B: global shared BPE (recommended — higher quality) corpus = open("some_corpus.txt").read() bpe = Expert.build_bpe(corpus, vocab_size=2000, D=10000) # train once math = Expert.from_qa_pairs( [("what is pi", "pi is approximately three point one four")] * 5, domain="math", D=10000, tokenizer=bpe) # share it geo = Expert.from_qa_pairs( [("what is the capital of france", "the capital of france is paris")] * 5, domain="geography", D=10000, tokenizer=bpe) brain = Brain() brain.add_expert(math) brain.add_expert(geo) print(brain.query("what is pi").answer) # -> math print(brain.query("what is the capital of france").answer) # -> geography brain.think(seconds=10) # the brain dreams -> writes to central memory brain.self_modify() # safely retunes its own coupling # the central memory persists brain.save("mybrain") brain2 = Brain.load("mybrain") # concepts survive reload print(f"{brain2.n_concepts} concepts persisted") ``` ### Saving experts (compressed) ```python result = math.save("math.exp") # source -> expert, compressed math2 = Expert.load("math.exp", tokenizer=bpe) # BPE experts need the shared tokenizer ``` ### Structural generalization (unseen slots) ```python # Train on capitals of some countries, WITH a template. # The template lets the expert generalize the *answer format* to unseen slots. geo = Expert.from_qa_pairs( [("what is the capital of france", "the capital of france is paris"), ("what is the capital of germany", "the capital of germany is berlin")] * 4, domain="geo", D=10000, patterns=["what is the capital of {country}"]) # spain was NEVER in training — yet the structural expert answers (format-generalized), # where a char-level expert would return empty. print(geo.answer("what is the capital of spain")) # non-empty, by analogy ``` --- ## Scaling to "1B-equivalent" ENSEMBLE has **no stored parameters**, so "1B" means **capacity** — distinct associations the memory holds without collision, exponential in dimension `D`. The `1b` preset uses `D = 100 000`. | preset | D | use case | |---|---|---| | `tiny` | 2,000 | quick demos | | `small` | 10,000 | laptop default | | `medium` | 30,000 | more capacity | | `large` | 50,000 | large corpora | | `1b` | 100,000 | "1B-equivalent" capacity | --- ## Benchmark results (measured, reproducible) Full methodology in [`RESULTS.md`](./RESULTS.md). ### BPE vs char-level on TinyStories (150 KB excerpt, D=5000) | metric | char | **BPE** | delta | |---|--:|--:|---| | **next-token accuracy** | 94.5% | **98.5%** | **+4.0 pts** | | **query latency** | 112 ms | **28 ms** | **4× faster** | | **RAM** | 187 MB | **89 MB** | **2× less** | | build time | 46s | 107s | one-shot cost | The global BPE lifts the quality ceiling exactly as predicted: higher accuracy, far lower latency (fewer, longer tokens), and half the RAM. The tradeoff is a slower one-time build. ### ENSEMBLE scaling with D (facts corpus) | D | next-token acc | QA recall | .exp/source | |--:|--:|--:|--:| | 2,000 | 30.6% | 0% | 7.0× | | 10,000 | 93.5% | 35% | 7.0× | | 100,000 (1b) | 94.7% | 40% | 7.0× | ### Generalization to unseen slots (capitals benchmark) Train on capitals of 5 countries; hold out 8 unseen countries. Metric: % of holdout returning a non-empty, well-formed answer. | mode | non-empty on unseen slots | |---|--:| | char-level | **0%** (silent failure) | | **structural (patterns)** | **100%** (graceful, format-generalized) | Honest: the structural answers are well-formed guesses by analogy, not factually correct for fully unseen slots (no semantic embeddings). The win is *graceful degradation* — the system says something sensible instead of nothing. ### Honest verdict vs 1B transformers (TinyLlama-1.1B, Pythia-1B) **ENSEMBLE wins on:** zero training (~30s CPU vs a GPU cluster on 1T tokens), footprint (a `1b` expert is **~2 KB** vs **~2 GB**), instant knowledge injection, compositionality, near-perfect memorization of seen data (95–99% with BPE), and **continuous growth** (the central memory never stops accumulating). **A 1B transformer wins on:** broad world knowledge (MMLU, HellaSwag), **generalization** (ENSEMBLE memorizes; holdout QA ≈ 0%), fluency on unseen text, reasoning. **Bottom line:** ENSEMBLE is a different tool. For narrow domains with known data and a CPU-only / tiny-footprint constraint, it is competitive or superior. For general intelligence, it is not — yet. --- ## Architecture ``` Expert dataset -> compressed .exp (one pass, no gradient) └ build_bpe train a global shared BPE tokenizer ExpertOscillator an expert as a Kuramoto oscillator ExpertKuramotoAttractor multi-expert synchronization -> emergent attractor Brain conductor: roster, query, think, self-modify, save/load BrainMemory persistent central memory (grows via thinking) ContinuousThinking cross-expert dreaming -> writes concepts to BrainMemory SelfModifier Lyapunov-bounded retuning of coupling ``` --- ## What this is (and is not) **Is:** a training-free associative memory built on hypervector (VSA) algebra; an expert system where knowledge is portable, compressed, composable, and grows over time; a research artifact exploring Kuramoto coupling + persistent concept memory. CPU-only, numpy-only, auditable. **Is not:** a transformer; competitive on broad-knowledge benchmarks; a generalizer (it memorizes what it is shown). --- ## Run the tests ```bash pip install numpy pytest pytest tests/ -q # 75 tests ``` ## Repository layout ``` ensemble/ the ENSEMBLE package (expert, bpe integration, brain, brain_memory, ...) palimseste/ the vendored hypervector substrate bench/ reproducible benchmarks (incl. BPE vs char comparison) tests/ 75 tests RESULTS.md full benchmark report ``` ## License MIT. Both `ensemble` and the vendored `palimseste` substrate are MIT-licensed.