README.md

---
license: mit
library_name: mlx
tags:
  - hrr
  - vsa
  - holographic-reduced-representations
  - tokenizer
  - morphemes
  - mlx
  - compositional
language: en
pipeline_tag: feature-extraction
---

# morph-hrr

A **compositional morpheme tokenizer built on Holographic Reduced Representations (HRR)**, for Apple MLX.

Each word becomes one fixed-width dense vector by *binding* its morphemes (prefix ⊗ root ⊗ suffix) into a single "holistic" superposition. Because the binding is circular convolution with **unitary role vectors**, you can algebraically pull a morpheme back out: `unbind(word_vec, prefix_role)` recovers the prefix. Words that share morphology land near each other in vector space, and an out-of-vocabulary word built from its pieces still neighbors its root family.

## What this is — and isn't

**It is:** an **input representation / embedding layer** for HRR-native or experimental models. It emits dense vectors, one per word, and exposes the HRR algebra (`bind`, `unbind`, `bundle`, `make_unitary`) behind them.

**It is not:** a HuggingFace text↔ID tokenizer. It does not produce token IDs, has no vocabulary, and will not drop into `transformers`. Think of it as an alternative to a learned embedding table — one whose vectors are *compositional by construction* rather than arbitrary.

**It is not:** a way to make a pretrained Qwen/Gemma smaller or faster. HRR input is a different representation than what pretrained weights expect; you cannot swap it into an existing model without retraining from scratch. (See *Scope & honesty* below.)

## Install

```bash
pip install morph-hrr        # Apple Silicon (mlx is the only dependency)
```

Development:

```bash
git clone <repo> && cd morph-hrr
pip install -e ".[test]"
pytest -q
```

## Quickstart

```python
from morph_hrr import MorphemeTokenizer, unbind, cosine_similarity

tok = MorphemeTokenizer(dim=2048)        # deterministic given seed

# A word is one fixed-width vector, composed from its morphemes by role.
v = tok.word_vector("unhappy")           # mx.array, shape (2048,)
print(tok.segment("unhappy"))            # ('un', 'happy', '')

# Composition is algebraic: recover the prefix filler by unbinding its role.
recovered = unbind(v, tok.prefix_role)
print(cosine_similarity(recovered, tok.bytes_vector("un")))   # ~0.6 (vs ~0 control)

# Shared morphology => shared neighborhood.
print(cosine_similarity(tok.word_vector("unhappy"),
                        tok.word_vector("happy")))            # > 0.2 (vs ~0 unrelated)

# Encode a sentence: one float16 vector per word.
mat = tok.encode("the quick brown fox")   # mx.array, shape (4, 2048), float16
```

## The math (one paragraph)

`bind(a, b)` is circular convolution, computed in the Fourier domain as `real(IFFT(FFT(a) * FFT(b)))`. `unbind(bind(a,b), b)` is the convolution inverse, `real(IFFT(FFT(bound) * conj(FFT(b))))`. A **unitary** vector has all FFT magnitudes equal to 1, so binding with it is a perfect, lossless rotation — unbinding recovers the original to >0.99 cosine. A word vector bundles three role⊗filler pairs — `prefix_role ⊗ bytes(prefix)`, `root_role ⊗ bytes(root)`, `suffix_role ⊗ bytes(suffix)` — into one normalized superposition; unbinding a role pulls its filler back out of the superposition (the other two act as noise, which is why longer dims recover more cleanly).

## Compositionality demo (regression-tested, dim=2048)

| Property | HRR cosine | Random control | Test |
|---|---|---|---|
| Prefix recovery (`unbind(unhappy, prefix_role)` ≈ `bytes("un")`) | > 0.50 | ≈ 0 | `test_prefix_recovery_beats_control` |
| Shared-root clustering (`unhappy` ~ `happy`) | > 0.20 | ≈ 0 | `test_shared_root_clusters` |
| Shared-suffix clustering (`running` ~ `walking`) | > 0.15 | ≈ 0 | `test_shared_suffix_clusters` |
| OOV root family (`unbind(unkind, root_role)` ≈ `bytes("kind")`) | > 0.50 | ≈ 0 | `test_oov_root_family` |
| Role vectors near-orthogonal | < 0.10 | — | `test_roles_are_near_orthogonal` |

Each HRR measurement is paired against a random-vector control of the same dimension, asserting the structured signal is meaningfully stronger than noise.

## Public API

| Name | What |
|---|---|
| `MorphemeTokenizer(dim=2048, seed=0)` | Map text → HRR morpheme vectors |
| `.segment(word)` | `(prefix, root, suffix)` |
| `.word_vector(word)` / `.bytes_vector(text)` | Holistic / byte-fold word vector |
| `.encode(text)` / `.iter_vectors(text)` | Stacked `(n, dim)` float16 / lazy yield |
| `.prefix_role` / `.root_role` / `.suffix_role` | Exposed unitary role vectors (for `unbind`) |
| `bind`, `unbind`, `bundle`, `normalize`, `make_unitary`, `cosine_similarity`, `update_context` | HRR primitives |
| `segment`, `PREFIXES`, `SUFFIXES` | Standalone morphological segmentation |

`HolographicMorphemeTokenizer` is kept as a backwards-compatible alias for `MorphemeTokenizer`.

## Segmentation: deliberately simple

`morphemes.segment` is dependency-free, longest-match affix stripping over curated prefix/suffix lists, with a minimum-root guard. It is **imperfect by design**: it does not undo consonant doubling (`running → runn + ing`, not `run + ing`) and cannot tell a real root from a suffixable tail (`preorder → pre + ord + er`). No dictionary, no learned model, no `nltk`. This keeps the package lightweight; better segmentation is future work and is orthogonal to the HRR representation itself.

## Scope & honesty

- **mlx-only.** Apple Silicon target audience; a numpy/JAX backend is a documented future option, not this release.
- **Representation, not a drop-in tokenizer.** No token IDs, no HF integration.
- **Can't retrofit pretrained models.** Swapping HRR input into a real model throws away its pretrained weights — you must train from scratch. On consumer hardware (e.g. 16 GB) that means small research-scale models, not deployable ones. The value of this package is the *compositional input representation* and the HRR algebra, demonstrated on small models.
- v0.1 ships the tokenizer + primitives + tests. A demo Space, a trained reference model, and a portable backend are future work.

## License

MIT. See `LICENSE`.

## Cite / priority

If you build on this representation in published work, please cite this repository. The compositional HRR morpheme representation (role⊗filler binding of prefix/root/suffix via circular convolution, with unitary roles enabling algebraic morpheme manipulation) is, to our knowledge, novel as a tokenization scheme; we'd appreciate attribution.