bytelex v1: README — model-free byte-information relational system, artifact contract, loss primitives

README.md

@@ -1,3 +1,89 @@
 ---
 license: mit
+tags:
+  - tokenization
+  - byte-level
+  - distillation
+  - lexicon
 ---
+
+# geolip-bytelex — the universal byte-information relational system
+
+A **model-free translation matrix between tokenizer languages and byte
+language**. Foreign tokens are not the same language as bytes — but any
+real token *is* a byte string underneath, and the statistics of byte
+information itself (not any particular model's internals) decide how
+that token relates to byte-native units. This repo hosts the matrix:
+every token of many tokenizers, projected through corpus-derived byte
+relational structure, so that **any ByteLM derivative** can consume
+foreign-tokenizer supervision (distillation, alignment, evaluation)
+through one reusable interface.
+
+Design rules, per the program's laws:
+
+- **Model-free.** Nothing here depends on model weights. Internal model
+  structure drifts over training; corpus byte statistics do not care.
+- **Alphabet-parametric.** The alphabet (256 byte values today) is a
+  schema parameter — it can grow or shrink; the system regenerates.
+- **Gram-modular.** Relational views are declared, not hardcoded:
+  char n-grams (1–4 exact), hashed large-n (9-gram class), word-grams
+  via a separator predicate — add views as required (quadgram, wordgram
+  combos, whatever the task needs).
+- **Specials are out-of-alphabet.** Control tokens are tabled and
+  flagged, never silently byte-expanded into text statistics.
+
+## Artifacts
+
+| path | contents |
+|---|---|
+| `vocab_<tokenizer>.jsonl` | Normalized token→bytes tables: `{"id", "hex", "text", "n_bytes", "is_special", "continuation"}`. Hex is authoritative; `text` is null when bytes aren't valid UTF-8. |
+| `byte_lexicon_v1/` | The corpus statistics: exact char-gram tables (n=1..4), hashed 9-gram table, word-gram table, schema + provenance in `meta.json`. |
+| `matrix_<tokenizer>.jsonl` | The full lexicon translation matrix: every non-special token's relational profile — `hmax`/`hargmax` (internal boundary-entropy maximum: where byte-language says the token divides), `pmin` (minimum internal PMI: cohesion), `word` (word-gram standing). |
+| `matrix_summary.json` | Cross-tokenizer comparison: word fraction, divides/cohesive fractions, mean internal entropy. |
+
+## Tokenizer roster (v1)
+
+byte-BPE: gpt2 · Qwen3.8 (248,077 vocab) · DeepSeek-V3 · SmolLM2 ·
+Llama-3.1 — tiktoken: cl100k_base · o200k_base — SentencePiece-BPE:
+Mistral v0.3 · XLM-R — SP-unigram: T5 — WordPiece: bert-base-uncased
+(lossy/uncased: flagged) — byte-identity control: ByT5.
+
+All twelve extracted and round-trip spot-checked. Every generative
+tokenizer in the roster attaches whitespace **leading**; byte-native
+segmentations (measured on AlephLM) tend to build **trailing**-space
+units — consumers must normalize the convention before comparing
+boundaries.
+
+## The loss primitives (what consumers do with this)
+
+For distillation from any token-level teacher into any ByteLM student:
+
+1. **Alignment endpoints** = corpus-entropy boundaries (successor
+   branching entropy over the byte corpus), *not* any model's
+   segmentation and *not* the teacher's token boundaries alone.
+2. **Per-token weight** = cohesion (`pmin`, internal PMI): a token the
+   byte-language considers one unit aligns as one endpoint-to-endpoint
+   span.
+3. **Non-cohesive tokens split** at their internal entropy maxima
+   (`hargmax`) before matching — the token was two byte-units wearing
+   one id, and the loss should know.
+
+Teacher logits push forward to byte space along each token's byte
+expansion (the exact-conversion direction of Phan et al., ICLR 2025);
+chunk-level likelihood matching happens between co-boundaries (the ALM
+family, arXiv:2503.20083), with the chunks defined by the corpus, so
+the same matrix serves every teacher and every ByteLM student.
+
+## Regeneration
+
+The consuming/producing code is `geolip.bytelex` in
+[AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm):
+build a `ByteLexicon(GramSchema(...))` over any corpus stream, `feed`
+raw bytes, `save`; project any `vocab_*.jsonl` through
+`ByteLexicon.profile`. New alphabet, new corpus, new gram views, new
+tokenizers — same recipe.
+
+Part of the AlephLLM / Mini-Beatrix program
+([training record](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training)).
+AlephLM is the strongest current consumer of this structure — but the
+matrix belongs to the bytes, not to any one model.