license: mit
library_name: mlx
tags:
- hrr
- vsa
- holographic-reduced-representations
- tokenizer
- morphemes
- mlx
- compositional
language: en
pipeline_tag: feature-extraction
morph-hrr
A compositional morpheme tokenizer built on Holographic Reduced Representations (HRR), for Apple MLX.
Each word becomes one fixed-width dense vector by binding its morphemes (prefix ⊗ root ⊗ suffix) into a single "holistic" superposition. Because the binding is circular convolution with unitary role vectors, you can algebraically pull a morpheme back out: unbind(word_vec, prefix_role) recovers the prefix. Words that share morphology land near each other in vector space, and an out-of-vocabulary word built from its pieces still neighbors its root family.
What this is — and isn't
It is: an input representation / embedding layer for HRR-native or experimental models. It emits dense vectors, one per word, and exposes the HRR algebra (bind, unbind, bundle, make_unitary) behind them.
It is not: a HuggingFace text↔ID tokenizer. It does not produce token IDs, has no vocabulary, and will not drop into transformers. Think of it as an alternative to a learned embedding table — one whose vectors are compositional by construction rather than arbitrary.
It is not: a way to make a pretrained Qwen/Gemma smaller or faster. HRR input is a different representation than what pretrained weights expect; you cannot swap it into an existing model without retraining from scratch. (See Scope & honesty below.)
Install
pip install morph-hrr # Apple Silicon (mlx is the only dependency)
Development:
git clone <repo> && cd morph-hrr
pip install -e ".[test]"
pytest -q
Quickstart
from morph_hrr import MorphemeTokenizer, unbind, cosine_similarity
tok = MorphemeTokenizer(dim=2048) # deterministic given seed
# A word is one fixed-width vector, composed from its morphemes by role.
v = tok.word_vector("unhappy") # mx.array, shape (2048,)
print(tok.segment("unhappy")) # ('un', 'happy', '')
# Composition is algebraic: recover the prefix filler by unbinding its role.
recovered = unbind(v, tok.prefix_role)
print(cosine_similarity(recovered, tok.bytes_vector("un"))) # ~0.6 (vs ~0 control)
# Shared morphology => shared neighborhood.
print(cosine_similarity(tok.word_vector("unhappy"),
tok.word_vector("happy"))) # > 0.2 (vs ~0 unrelated)
# Encode a sentence: one float16 vector per word.
mat = tok.encode("the quick brown fox") # mx.array, shape (4, 2048), float16
The math (one paragraph)
bind(a, b) is circular convolution, computed in the Fourier domain as real(IFFT(FFT(a) * FFT(b))). unbind(bind(a,b), b) is the convolution inverse, real(IFFT(FFT(bound) * conj(FFT(b)))). A unitary vector has all FFT magnitudes equal to 1, so binding with it is a perfect, lossless rotation — unbinding recovers the original to >0.99 cosine. A word vector bundles three role⊗filler pairs — prefix_role ⊗ bytes(prefix), root_role ⊗ bytes(root), suffix_role ⊗ bytes(suffix) — into one normalized superposition; unbinding a role pulls its filler back out of the superposition (the other two act as noise, which is why longer dims recover more cleanly).
Compositionality demo (regression-tested, dim=2048)
| Property | HRR cosine | Random control | Test |
|---|---|---|---|
Prefix recovery (unbind(unhappy, prefix_role) ≈ bytes("un")) |
> 0.50 | ≈ 0 | test_prefix_recovery_beats_control |
Shared-root clustering (unhappy ~ happy) |
> 0.20 | ≈ 0 | test_shared_root_clusters |
Shared-suffix clustering (running ~ walking) |
> 0.15 | ≈ 0 | test_shared_suffix_clusters |
OOV root family (unbind(unkind, root_role) ≈ bytes("kind")) |
> 0.50 | ≈ 0 | test_oov_root_family |
| Role vectors near-orthogonal | < 0.10 | — | test_roles_are_near_orthogonal |
Each HRR measurement is paired against a random-vector control of the same dimension, asserting the structured signal is meaningfully stronger than noise.
Public API
| Name | What |
|---|---|
MorphemeTokenizer(dim=2048, seed=0) |
Map text → HRR morpheme vectors |
.segment(word) |
(prefix, root, suffix) |
.word_vector(word) / .bytes_vector(text) |
Holistic / byte-fold word vector |
.encode(text) / .iter_vectors(text) |
Stacked (n, dim) float16 / lazy yield |
.prefix_role / .root_role / .suffix_role |
Exposed unitary role vectors (for unbind) |
bind, unbind, bundle, normalize, make_unitary, cosine_similarity, update_context |
HRR primitives |
segment, PREFIXES, SUFFIXES |
Standalone morphological segmentation |
HolographicMorphemeTokenizer is kept as a backwards-compatible alias for MorphemeTokenizer.
Segmentation: deliberately simple
morphemes.segment is dependency-free, longest-match affix stripping over curated prefix/suffix lists, with a minimum-root guard. It is imperfect by design: it does not undo consonant doubling (running → runn + ing, not run + ing) and cannot tell a real root from a suffixable tail (preorder → pre + ord + er). No dictionary, no learned model, no nltk. This keeps the package lightweight; better segmentation is future work and is orthogonal to the HRR representation itself.
Scope & honesty
- mlx-only. Apple Silicon target audience; a numpy/JAX backend is a documented future option, not this release.
- Representation, not a drop-in tokenizer. No token IDs, no HF integration.
- Can't retrofit pretrained models. Swapping HRR input into a real model throws away its pretrained weights — you must train from scratch. On consumer hardware (e.g. 16 GB) that means small research-scale models, not deployable ones. The value of this package is the compositional input representation and the HRR algebra, demonstrated on small models.
- v0.1 ships the tokenizer + primitives + tests. A demo Space, a trained reference model, and a portable backend are future work.
License
MIT. See LICENSE.
Cite / priority
If you build on this representation in published work, please cite this repository. The compositional HRR morpheme representation (role⊗filler binding of prefix/root/suffix via circular convolution, with unitary roles enabling algebraic morpheme manipulation) is, to our knowledge, novel as a tokenization scheme; we'd appreciate attribution.