WYRD42 — BPE-32k Tokenizer
A deterministic 32,768-token ByteLevel BPE tokenizer for WYRD42, a seeded, induction-resistant constructed language. It is the shared token id-space for both base-pretraining and chat fine-tuning of a 1.1B WYRD language model.
What is WYRD?
WYRD is a constructed language defined not by a hand-written grammar but by a seeded
deterministic generator: a pure program plus a private 256-bit master seed that coins
its own lexicon, fills every inflectional paradigm, and realises surface strings in
linear forward time. Irregularity across orthography, morphophonology, morphology, and
syntax is keyed to a pseudorandom function of (lexeme, morphosyntactic cell) under the
seed, so the corpus is cheap to generate forward and engineered to be intractable to
invert into a compact grammar. Seed for this corpus: "42" (hence WYRD42).
Determinism & reproducibility
The tokenizer is a byte-verifiable function of the corpus and a pinned trainer:
- Training sample: a fixed, sorted, per-register byte budget of the
wyrdfield of the seed-42 v4 corpus — no sampling, no shuffling (train/train_tokenizer.py). - BPE trainer:
tokenizers==0.22.2,vocab_size=32768,min_frequency=2, ByteLevel. manifest.jsonrecords the sha256 of the sample and ofvocab.json/merges.txt, so a re-run can be byte-compared.
| artefact | sha256 (first 16) |
|---|---|
| training sample (3037 MB) | 9c3364d96a873ae2 |
vocab.json |
22dee61e598829d5 |
merges.txt |
a53b26703e0256f4 |
Special tokens
<|endoftext|> · <|doc|> · <|q|> · <|user|> · <|assistant|> · <|pad|> — document,
query, and chat-role markers, so base bins and chat-SFT data share one id-space.
Fertility
3.38 characters / token (3.92 bytes/token) on native WYRD. WYRD is agglutinative with long words; this fertility sets the ~30B-token training budget for the 1.1B model.
Registers in the training sample
FineWeb-Edu, FineWeb (commerce), Wikipedia, Gutenberg, balanced native-generation, conversations, queries, and a multilingual feature-rich supplement — so the merges cover web, narrative, chat, and the full WYRD feature space (evidentiality, dual/paucal, etc.).
Usage
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
ids = tok.encode("Riga-Kola thüükwúe John Muir").ids # brands pass through as tokens
print(tok.decode(ids)) # -> "Riga-Kola thüükwúe John Muir"
Proper names and brands pass through the language verbatim as a transparent loanword channel, so a brand first seen after training still has a valid token sequence.
License
Research artefact. WYRD is a linguistics research object (forward-easy / invert-hard keyed text transform), not a security primitive.