Upload folder using huggingface_hub

README.md

@@ -0,0 +1,79 @@
+---
+license: other
+license_name: wyrd-research
+language:
+- en
+library_name: tokenizers
+tags:
+- tokenizer
+- bpe
+- byte-level-bpe
+- constructed-language
+- wyrd
+- induction-resistant
+---
+
+# WYRD42 — BPE-32k Tokenizer
+
+A **deterministic** 32,768-token ByteLevel BPE tokenizer for **WYRD42**, a seeded,
+induction-resistant constructed language. It is the shared token id-space for both
+base-pretraining and chat fine-tuning of a 1.1B WYRD language model.
+
+## What is WYRD?
+
+WYRD is a constructed language defined not by a hand-written grammar but by a *seeded
+deterministic generator*: a pure program plus a private 256-bit master seed that coins
+its own lexicon, fills every inflectional paradigm, and realises surface strings in
+linear forward time. Irregularity across orthography, morphophonology, morphology, and
+syntax is keyed to a pseudorandom function of `(lexeme, morphosyntactic cell)` under the
+seed, so the corpus is cheap to generate forward and engineered to be intractable to
+invert into a compact grammar. Seed for this corpus: `"42"` (hence *WYRD42*).
+
+## Determinism & reproducibility
+
+The tokenizer is a byte-verifiable function of the corpus and a pinned trainer:
+
+- Training sample: a **fixed, sorted, per-register byte budget** of the `wyrd` field of
+  the seed-42 v4 corpus — no sampling, no shuffling (`train/train_tokenizer.py`).
+- BPE trainer: `tokenizers==0.22.2`, `vocab_size=32768`, `min_frequency=2`, ByteLevel.
+- `manifest.json` records the sha256 of the sample and of `vocab.json` / `merges.txt`,
+  so a re-run can be byte-compared.
+
+| artefact | sha256 (first 16) |
+|---|---|
+| training sample (3037 MB) | `9c3364d96a873ae2` |
+| `vocab.json` | `22dee61e598829d5` |
+| `merges.txt` | `a53b26703e0256f4` |
+
+## Special tokens
+
+`<|endoftext|>` · `<|doc|>` · `<|q|>` · `<|user|>` · `<|assistant|>` · `<|pad|>` — document,
+query, and chat-role markers, so base bins and chat-SFT data share one id-space.
+
+## Fertility
+
+**3.38 characters / token** (3.92 bytes/token) on native WYRD. WYRD is agglutinative with
+long words; this fertility sets the ~30B-token training budget for the 1.1B model.
+
+## Registers in the training sample
+
+FineWeb-Edu, FineWeb (commerce), Wikipedia, Gutenberg, balanced native-generation,
+conversations, queries, and a multilingual feature-rich supplement — so the merges cover
+web, narrative, chat, and the full WYRD feature space (evidentiality, dual/paucal, etc.).
+
+## Usage
+
+```python
+from tokenizers import Tokenizer
+tok = Tokenizer.from_file("tokenizer.json")
+ids = tok.encode("Riga-Kola thüükwúe John Muir").ids   # brands pass through as tokens
+print(tok.decode(ids))                                 # -> "Riga-Kola thüükwúe John Muir"
+```
+
+Proper names and brands pass through the language verbatim as a transparent loanword
+channel, so a brand first seen *after* training still has a valid token sequence.
+
+## License
+
+Research artefact. WYRD is a linguistics research object (forward-easy / invert-hard keyed
+text transform), **not** a security primitive.

manifest.json

@@ -0,0 +1,68 @@
+{
+  "name": "wyrd42-bpe-32k",
+  "vocab_size": 32768,
+  "min_frequency": 2,
+  "special_tokens": [
+    "<|endoftext|>",
+    "<|doc|>",
+    "<|q|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|pad|>"
+  ],
+  "tokenizers_version": "0.22.2",
+  "registers": [
+    [
+      "web_edu",
+      "data/web-wyrd-edu-v4/*.jsonl",
+      "jsonl",
+      900
+    ],
+    [
+      "web_gen",
+      "data/web-wyrd-gen-v4/*.jsonl",
+      "jsonl",
+      700
+    ],
+    [
+      "wikipedia",
+      "data/wyrd-wikipedia-v4/*.jsonl",
+      "jsonl",
+      500
+    ],
+    [
+      "gutenberg",
+      "data/wyrd-gutenberg-v4/*.jsonl",
+      "jsonl",
+      500
+    ],
+    [
+      "native",
+      "data/native-wyrd/*.txt",
+      "text",
+      400
+    ],
+    [
+      "conv",
+      "data/wyrd-conv-v4/*.jsonl",
+      "jsonl",
+      200
+    ],
+    [
+      "multiling",
+      "data/multiling-wyrd/**/*",
+      "text",
+      100
+    ],
+    [
+      "queries",
+      "data/wyrd-queries-v4/*.jsonl",
+      "jsonl",
+      100
+    ]
+  ],
+  "sample_mb": 3037,
+  "sample_sha256": "9c3364d96a873ae24266de3798ea76472c62ca5b942155fade1002b22301ccde",
+  "vocab_sha256": "22dee61e598829d558e1bd396a54abe87ece5f997bd2d6e3e3e2f623e961cf2c",
+  "merges_sha256": "a53b26703e0256f41b74aa06e57b4abbd9902110f7e8a374f1b4dc55e8691599"
+}