morph-hrr

A compositional morpheme tokenizer built on Holographic Reduced Representations (HRR), for Apple MLX.

Each word becomes one fixed-width dense vector by binding its morphemes (prefix ⊗ root ⊗ suffix) into a single "holistic" superposition. Because the binding is circular convolution with unitary role vectors, you can algebraically pull a morpheme back out: unbind(word_vec, prefix_role) recovers the prefix. Words that share morphology land near each other in vector space, and an out-of-vocabulary word built from its pieces still neighbors its root family.

What this is — and isn't

It is: an input representation / embedding layer for HRR-native or experimental models. It emits dense vectors, one per word, and exposes the HRR algebra (bind, unbind, bundle, make_unitary) behind them.

It is not: a HuggingFace text↔ID tokenizer. It does not produce token IDs, has no vocabulary, and will not drop into transformers. Think of it as an alternative to a learned embedding table — one whose vectors are compositional by construction rather than arbitrary.

It is not: a way to make a pretrained Qwen/Gemma smaller or faster. HRR input is a different representation than what pretrained weights expect; you cannot swap it into an existing model without retraining from scratch. (See Scope & honesty below.)

Install

pip install morph-hrr        # Apple Silicon (mlx is the only dependency)

Development:

git clone <repo> && cd morph-hrr
pip install -e ".[test]"
pytest -q

Quickstart

from morph_hrr import MorphemeTokenizer, unbind, cosine_similarity

tok = MorphemeTokenizer(dim=2048)        # deterministic given seed

# A word is one fixed-width vector, composed from its morphemes by role.
v = tok.word_vector("unhappy")           # mx.array, shape (2048,)
print(tok.segment("unhappy"))            # ('un', 'happy', '')

# Composition is algebraic: recover the prefix filler by unbinding its role.
recovered = unbind(v, tok.prefix_role)
print(cosine_similarity(recovered, tok.bytes_vector("un")))   # ~0.6 (vs ~0 control)

# Shared morphology => shared neighborhood.
print(cosine_similarity(tok.word_vector("unhappy"),
                        tok.word_vector("happy")))            # > 0.2 (vs ~0 unrelated)

# Encode a sentence: one float16 vector per word.
mat = tok.encode("the quick brown fox")   # mx.array, shape (4, 2048), float16

The math (one paragraph)

bind(a, b) is circular convolution, computed in the Fourier domain as real(IFFT(FFT(a) * FFT(b))). unbind(bind(a,b), b) is the convolution inverse, real(IFFT(FFT(bound) * conj(FFT(b)))). A unitary vector has all FFT magnitudes equal to 1, so binding with it is a perfect, lossless rotation — unbinding recovers the original to >0.99 cosine. A word vector bundles three role⊗filler pairs — prefix_role ⊗ bytes(prefix), root_role ⊗ bytes(root), suffix_role ⊗ bytes(suffix) — into one normalized superposition; unbinding a role pulls its filler back out of the superposition (the other two act as noise, which is why longer dims recover more cleanly).

Compositionality demo (regression-tested, dim=2048)

Property	HRR cosine	Random control	Test
Prefix recovery (unbind(unhappy, prefix_role) ≈ bytes("un"))	> 0.50	≈ 0	test_prefix_recovery_beats_control
Shared-root clustering (unhappy ~ happy)	> 0.20	≈ 0	test_shared_root_clusters
Shared-suffix clustering (running ~ walking)	> 0.15	≈ 0	test_shared_suffix_clusters
OOV root family (unbind(unkind, root_role) ≈ bytes("kind"))	> 0.50	≈ 0	test_oov_root_family
Role vectors near-orthogonal	< 0.10	—	test_roles_are_near_orthogonal

Each HRR measurement is paired against a random-vector control of the same dimension, asserting the structured signal is meaningfully stronger than noise.

Public API

Name	What
MorphemeTokenizer(dim=2048, seed=0)	Map text → HRR morpheme vectors
.segment(word)	(prefix, root, suffix)
.word_vector(word) / .bytes_vector(text)	Holistic / byte-fold word vector
.encode(text) / .iter_vectors(text)	Stacked (n, dim) float16 / lazy yield
.prefix_role / .root_role / .suffix_role	Exposed unitary role vectors (for unbind)
bind, unbind, bundle, normalize, make_unitary, cosine_similarity, update_context	HRR primitives
segment, PREFIXES, SUFFIXES	Standalone morphological segmentation

HolographicMorphemeTokenizer is kept as a backwards-compatible alias for MorphemeTokenizer.

Segmentation: deliberately simple

morphemes.segment is dependency-free, longest-match affix stripping over curated prefix/suffix lists, with a minimum-root guard. It is imperfect by design: it does not undo consonant doubling (running → runn + ing, not run + ing) and cannot tell a real root from a suffixable tail (preorder → pre + ord + er). No dictionary, no learned model, no nltk. This keeps the package lightweight; better segmentation is future work and is orthogonal to the HRR representation itself.

Scope & honesty

• mlx-only. Apple Silicon target audience; a numpy/JAX backend is a documented future option, not this release.
• Representation, not a drop-in tokenizer. No token IDs, no HF integration.
• Can't retrofit pretrained models. Swapping HRR input into a real model throws away its pretrained weights — you must train from scratch. On consumer hardware (e.g. 16 GB) that means small research-scale models, not deployable ones. The value of this package is the compositional input representation and the HRR algebra, demonstrated on small models.
• v0.1 ships the tokenizer + primitives + tests. A demo Space, a trained reference model, and a portable backend are future work.

License

MIT. See LICENSE.

Cite / priority

If you build on this representation in published work, please cite this repository. The compositional HRR morpheme representation (role⊗filler binding of prefix/root/suffix via circular convolution, with unitary roles enabling algebraic morpheme manipulation) is, to our knowledge, novel as a tokenization scheme; we'd appreciate attribution.