Publish 18 measured scholarly/code BPE tokenizers and research report

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+research/corpus-composition.png filter=lfs diff=lfs merge=lfs -text
+research/resources-and-utilization.png filter=lfs diff=lfs merge=lfs -text
+research/source-comparisons.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,110 @@
+---
+library_name: transformers
+language:
+- en
+tags:
+- tokenizer
+- byte-level-bpe
+- code
+- scientific-text
+- from-scratch
+---
+
+# Scholarly/code tokenizer learning curve
+
+18 byte-level BPE tokenizers trained from scratch on Modal: 64, 128, 256, 512,
+1,024 and 2,048 MiB of text, each at 16,384, 32,768 and 50,000 vocabulary entries.
+This is a tokenizer resource, with no language-model weights.
+
+The default is **32,768 entries trained on 2 GiB**, a practical starting point
+balancing compression and embedding/output size. This is not a demonstrated
+model-quality optimum. All 18 variants are included for reproducible comparisons.
+The existing scholarly-code-v1 tokenizer in GitHub has not been replaced.
+
+## Usage
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("jfrick/scholarly-code-tokenizer")
+text = "def square(x):\n    return x ** 2\n"
+ids = tokenizer.encode(text, add_special_tokens=False)
+assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text
+
+# Choose any measured vocabulary and fitting size:
+tokenizer_50k = AutoTokenizer.from_pretrained(
+    "jfrick/scholarly-code-tokenizer", subfolder="variants/2048mib-50000"
+)
+```
+
+Tested with `transformers==4.57.6` and `tokenizers==0.22.2`. No custom remote code
+is required. The Rust `tokenizers.Tokenizer.from_file("tokenizer.json")` API also
+works. BOS/EOS are not inserted automatically; training pipelines must add document
+boundaries explicitly. Special IDs: PAD=0, BOS=1, EOS=2, UNK=3. Decode with
+`skip_special_tokens=False` when exact round trips including special literals matter.
+Byte coverage avoids unknown tokens for ordinary valid UTF-8 text. No maximum
+model context is implied by this tokenizer.
+
+## Results
+
+![Learning curve](research/learning-curve.png)
+
+All variants were evaluated on the same 32 MiB source-weighted held-out corpus,
+with parent separation and a 16 KiB evaluation parent cap. Higher bytes/token means
+fewer tokens for identical text. The [research report](research/REPORT.md) and
+[PDF](research/research-report.pdf) include per-source curves, paired bootstrap
+intervals, vocabulary utilization, fragmentation probes, fit time and peak memory.
+
+| Fitting MiB | Vocabulary | Bytes/token | Variant |
+|---:|---:|---:|---|
+| 64 | 16,384 | 3.7463 | [`64mib-16384`](variants/64mib-16384) |
+| 64 | 32,768 | 4.0110 | [`64mib-32768`](variants/64mib-32768) |
+| 64 | 50,000 | 4.1392 | [`64mib-50000`](variants/64mib-50000) |
+| 128 | 16,384 | 3.7571 | [`128mib-16384`](variants/128mib-16384) |
+| 128 | 32,768 | 4.0258 | [`128mib-32768`](variants/128mib-32768) |
+| 128 | 50,000 | 4.1549 | [`128mib-50000`](variants/128mib-50000) |
+| 256 | 16,384 | 3.7624 | [`256mib-16384`](variants/256mib-16384) |
+| 256 | 32,768 | 4.0338 | [`256mib-32768`](variants/256mib-32768) |
+| 256 | 50,000 | 4.1655 | [`256mib-50000`](variants/256mib-50000) |
+| 512 | 16,384 | 3.7646 | [`512mib-16384`](variants/512mib-16384) |
+| 512 | 32,768 | 4.0395 | [`512mib-32768`](variants/512mib-32768) |
+| 512 | 50,000 | 4.1729 | [`512mib-50000`](variants/512mib-50000) |
+| 1024 | 16,384 | 3.7659 | [`1024mib-16384`](variants/1024mib-16384) |
+| 1024 | 32,768 | 4.0431 | [`1024mib-32768`](variants/1024mib-32768) |
+| 1024 | 50,000 | 4.1774 | [`1024mib-50000`](variants/1024mib-50000) |
+| 2048 | 16,384 | 3.7660 | [`2048mib-16384`](variants/2048mib-16384) |
+| 2048 | 32,768 | 4.0438 | [`2048mib-32768`](variants/2048mib-32768) |
+| 2048 | 50,000 | 4.1786 | [`2048mib-50000`](variants/2048mib-50000) |
+
+## Data and method
+
+All 12 pinned source inventories in [recipe.json](recipe.json) were eligible:
+FineWeb-Edu 10%, Python code 30%, FineMath 20%, peS2o 10%, PubMed 3%, arXiv 2%,
+Pressbooks 3%, LibreTexts 3%, DOAB 4%, Gutenberg 5%, pre-1929 books 7%, and
+Library of Congress 3%. Percentages apply to UTF-8 bytes. This uses samples from
+all sources, not every record in those datasets. Source revisions and full shard
+inventories are pinned in the manifests. Raw source text is not distributed here;
+source datasets retain their respective licenses and provenance.
+
+Fitting uses nested samples, a seeded shard permutation and sequential records,
+64 KiB per document and 256 KiB per parent. Evaluation initially had sparse book
+coverage and was rebuilt with a 16 KiB parent cap before interpreting curves;
+all unchanged tokenizer weights were rescored. Normalized exact-chunk deduplication
+is applied, including fitting/evaluation overlap checks. Byte-level BPE uses the
+same fixed pretokenization rules, a full byte alphabet, min frequency 2 and maximum
+learned token length 32. No normalization or automatic prefix space is applied.
+
+## Limitations and reproducibility
+
+One sampling seed; no global reservoir sampling, fuzzy deduplication guarantee,
+or cross-source parent reconciliation. Bootstrap intervals measure evaluation
+uncertainty, not training-seed variance. Compression is not language-model quality.
+Compare downstream bits per byte under matched parameter and compute budgets
+before selecting a final vocabulary. At width 768, 32K adds 12.58M tied embedding
+parameters over 16K; 50K adds 25.82M. Untied input/output matrices double these deltas.
+
+[Implementation and PR](https://github.com/koogle/lm-exploration/pull/8).
+[Publication manifest](publication-manifest.json) pins every tokenizer hash.
+The `research/` manifests and measurements record corpus hashes, source coverage,
+correctness, resource measurements and original result hashes. Candidate tokenizers
+and full parent-level results are also preserved on the original Modal volumes.

local-verification.json

@@ -0,0 +1,17 @@
+{
+  "location": "/workspace/scratch/tokenizer-hf-release",
+  "revision": null,
+  "status": "passed",
+  "default_and_variants_checked": 19,
+  "probes_per_variant": 6,
+  "checks": [
+    "artifact_hashes",
+    "special_ids",
+    "vocabulary_size",
+    "token_id_equivalence",
+    "exact_roundtrips",
+    "no_automatic_boundaries"
+  ],
+  "transformers_version": "4.57.6",
+  "tokenizers_version": "0.22.2"
+}

publication-manifest.json

@@ -0,0 +1,25 @@
+{
+  "default_variant": "2048mib-32768",
+  "selection": "32K is a practical starting point balancing compression and vocabulary parameter overhead; no downstream optimum has been established.",
+  "variants": {
+    "64mib-16384": "97fa4177ad5a038cc18df87a2acb76c3ed3da89a9b7848517442bcbd7029d83d",
+    "64mib-32768": "cd9109d658834401ef05b4de231cdfd789385bb0d9f294e343e19b447d94b52a",
+    "64mib-50000": "86de4d58e6eb5417264afe227857e98c0b0431eb7adbfaf6a9ebacac37d77726",
+    "128mib-16384": "64356327337221b75b445cc7c761d655a40c1af2bccd8f8ab5453fd1fa1035c0",
+    "128mib-32768": "ade4e945319da6e9b010ad7f48f19d9489f7ac583448318a91cf302c21aaf321",
+    "128mib-50000": "db13f817c797466237e07a17ca3d62b65dafe334745d6c72714073fa47d36380",
+    "256mib-16384": "e76d8e8a71db29a3e1ed036d8e6813ce180ac568b0310d0cdc01a2c8974c92e6",
+    "256mib-32768": "3839415fcdd736bb9322b958a45ba82ba01310e2007a1b139f77e3486568c1b6",
+    "256mib-50000": "eb4b69846169b0aedd86c72eb2bc7e85b96b074519f7bba8563423cd200baf81",
+    "512mib-16384": "c3bae921b15a294391e45fe3bf2ce0a2f3ac1b6678c0f32541613106f322078e",
+    "512mib-32768": "ee276c0fba28e5d4f555f85971e9d99b5cee6b53aa5181bc1c4dd33417572829",
+    "512mib-50000": "1f4b70d34b608d97a3ad7a9bcd7b0094fa82c652281d39bcc12325c1f16865e2",
+    "1024mib-16384": "020928562c47ea392514da50b9c4886b4e3e24fd01cfdc30501599042ffb1359",
+    "1024mib-32768": "17ce952fb54679634b5a723756a217bb536ad1d2f6616917ac0172b7a6ef255c",
+    "1024mib-50000": "ce5e83a52cab742ea5edc2440c8defd26c946d05ca2f1d3f64ab3dedc26b744d",
+    "2048mib-16384": "f95ec3899180fd6aef876f92e8fa1e6ec9cd96c180e22b23ead4e88b3b002561",
+    "2048mib-32768": "6f8742023692929139e450a3a64802e355649742e7051f9d8a6e6db47fbc8e96",
+    "2048mib-50000": "5e3de8f77cfa5a5820a96bf0205e7cceee7f27c43e16afdc16d452df7a37f724"
+  },
+  "evaluation_manifest_sha256": "e4e66512ca51226f56844d61a4d0c065064900b6a6d0bd0243e9e64b8698024f"
+}

recipe.json

@@ -0,0 +1,489 @@
+{
+  "id": "scholarly-code-bpe-v1",
+  "library": "tokenizers==0.22.2",
+  "max_chunk_characters": 4096,
+  "max_chunks_per_document": 4,
+  "max_document_characters": 4000000,
+  "max_scanned_documents_per_source": 100000,
+  "model_target": "About150Mtotalparameters; embeddings included; model training not part of tokenizer job",
+  "sampling_weight_unit": "UTF-8 bytes before tokenizer exists; actual resulting token shares reported",
+  "seed": 20261016,
+  "selection_rule": "Prefer16K unless32K reduces total held-out tokens by at least10% and no individual source needs over2%moretokens; all roundtrip and bytecoverage checks must pass",
+  "sources": [
+    {
+      "files": [
+        "sample/10BT/000_00000.parquet",
+        "sample/10BT/001_00000.parquet",
+        "sample/10BT/002_00000.parquet",
+        "sample/10BT/003_00000.parquet",
+        "sample/10BT/004_00000.parquet",
+        "sample/10BT/005_00000.parquet",
+        "sample/10BT/006_00000.parquet",
+        "sample/10BT/007_00000.parquet",
+        "sample/10BT/008_00000.parquet",
+        "sample/10BT/009_00000.parquet",
+        "sample/10BT/010_00000.parquet",
+        "sample/10BT/011_00000.parquet",
+        "sample/10BT/012_00000.parquet",
+        "sample/10BT/013_00000.parquet"
+      ],
+      "format": "parquet",
+      "name": "fineweb_edu",
+      "repo": "HuggingFaceFW/fineweb-edu",
+      "revision": "87f09149ef4734204d70ed1d046ddc9ca3f2b8f9",
+      "source_url": "https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu",
+      "synthetic": false,
+      "weight_percent": 10
+    },
+    {
+      "adapter": "codeparrot",
+      "allowed_spdx_licenses": [
+        "MIT",
+        "Apache-2.0",
+        "BSD-2-Clause",
+        "BSD-3-Clause",
+        "ISC"
+      ],
+      "files": [
+        "file-000000000001.json.gz",
+        "file-000000000002.json.gz",
+        "file-000000000003.json.gz",
+        "file-000000000004.json.gz",
+        "file-000000000005.json.gz",
+        "file-000000000006.json.gz",
+        "file-000000000007.json.gz",
+        "file-000000000008.json.gz",
+        "file-000000000009.json.gz",
+        "file-000000000010.json.gz",
+        "file-000000000011.json.gz",
+        "file-000000000012.json.gz",
+        "file-000000000013.json.gz",
+        "file-000000000014.json.gz",
+        "file-000000000015.json.gz",
+        "file-000000000016.json.gz",
+        "file-000000000017.json.gz",
+        "file-000000000018.json.gz",
+        "file-000000000019.json.gz",
+        "file-000000000020.json.gz",
+        "file-000000000021.json.gz",
+        "file-000000000022.json.gz",
+        "file-000000000023.json.gz",
+        "file-000000000024.json.gz",
+        "file-000000000025.json.gz",
+        "file-000000000026.json.gz",
+        "file-000000000027.json.gz",
+        "file-000000000028.json.gz",
+        "file-000000000029.json.gz",
+        "file-000000000030.json.gz",
+        "file-000000000031.json.gz",
+        "file-000000000032.json.gz",
+        "file-000000000033.json.gz",
+        "file-000000000034.json.gz",
+        "file-000000000035.json.gz",
+        "file-000000000036.json.gz",
+        "file-000000000037.json.gz",
+        "file-000000000038.json.gz",
+        "file-000000000039.json.gz",
+        "file-000000000040.json.gz",
+        "file-000000000041.json.gz",
+        "file-000000000042.json.gz",
+        "file-000000000043.json.gz",
+        "file-000000000044.json.gz",
+        "file-000000000045.json.gz",
+        "file-000000000046.json.gz",
+        "file-000000000047.json.gz",
+        "file-000000000048.json.gz",
+        "file-000000000049.json.gz",
+        "file-000000000050.json.gz",
+        "file-000000000051.json.gz",
+        "file-000000000052.json.gz",
+        "file-000000000053.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "code_python",
+      "repo": "codeparrot/codeparrot-clean-train",
+      "revision": "3e6ab65f2864931e041f6a82db9b5a6ec2b71ab4",
+      "source_url": "https://huggingface.co/datasets/codeparrot/codeparrot-clean-train",
+      "synthetic": false,
+      "weight_percent": 30
+    },
+    {
+      "files": [
+        "finemath-4plus/train-00000-of-00064.parquet",
+        "finemath-4plus/train-00001-of-00064.parquet",
+        "finemath-4plus/train-00002-of-00064.parquet",
+        "finemath-4plus/train-00003-of-00064.parquet",
+        "finemath-4plus/train-00004-of-00064.parquet",
+        "finemath-4plus/train-00005-of-00064.parquet",
+        "finemath-4plus/train-00006-of-00064.parquet",
+        "finemath-4plus/train-00007-of-00064.parquet",
+        "finemath-4plus/train-00008-of-00064.parquet",
+        "finemath-4plus/train-00009-of-00064.parquet",
+        "finemath-4plus/train-00010-of-00064.parquet",
+        "finemath-4plus/train-00011-of-00064.parquet",
+        "finemath-4plus/train-00012-of-00064.parquet",
+        "finemath-4plus/train-00013-of-00064.parquet",
+        "finemath-4plus/train-00014-of-00064.parquet",
+        "finemath-4plus/train-00015-of-00064.parquet",
+        "finemath-4plus/train-00016-of-00064.parquet",
+        "finemath-4plus/train-00017-of-00064.parquet",
+        "finemath-4plus/train-00018-of-00064.parquet",
+        "finemath-4plus/train-00019-of-00064.parquet",
+        "finemath-4plus/train-00020-of-00064.parquet",
+        "finemath-4plus/train-00021-of-00064.parquet",
+        "finemath-4plus/train-00022-of-00064.parquet",
+        "finemath-4plus/train-00023-of-00064.parquet",
+        "finemath-4plus/train-00024-of-00064.parquet",
+        "finemath-4plus/train-00025-of-00064.parquet",
+        "finemath-4plus/train-00026-of-00064.parquet",
+        "finemath-4plus/train-00027-of-00064.parquet",
+        "finemath-4plus/train-00028-of-00064.parquet",
+        "finemath-4plus/train-00029-of-00064.parquet",
+        "finemath-4plus/train-00030-of-00064.parquet",
+        "finemath-4plus/train-00031-of-00064.parquet",
+        "finemath-4plus/train-00032-of-00064.parquet",
+        "finemath-4plus/train-00033-of-00064.parquet",
+        "finemath-4plus/train-00034-of-00064.parquet",
+        "finemath-4plus/train-00035-of-00064.parquet",
+        "finemath-4plus/train-00036-of-00064.parquet",
+        "finemath-4plus/train-00037-of-00064.parquet",
+        "finemath-4plus/train-00038-of-00064.parquet",
+        "finemath-4plus/train-00039-of-00064.parquet",
+        "finemath-4plus/train-00040-of-00064.parquet",
+        "finemath-4plus/train-00041-of-00064.parquet",
+        "finemath-4plus/train-00042-of-00064.parquet",
+        "finemath-4plus/train-00043-of-00064.parquet",
+        "finemath-4plus/train-00044-of-00064.parquet",
+        "finemath-4plus/train-00045-of-00064.parquet",
+        "finemath-4plus/train-00046-of-00064.parquet",
+        "finemath-4plus/train-00047-of-00064.parquet",
+        "finemath-4plus/train-00048-of-00064.parquet",
+        "finemath-4plus/train-00049-of-00064.parquet",
+        "finemath-4plus/train-00050-of-00064.parquet",
+        "finemath-4plus/train-00051-of-00064.parquet",
+        "finemath-4plus/train-00052-of-00064.parquet",
+        "finemath-4plus/train-00053-of-00064.parquet",
+        "finemath-4plus/train-00054-of-00064.parquet",
+        "finemath-4plus/train-00055-of-00064.parquet",
+        "finemath-4plus/train-00056-of-00064.parquet",
+        "finemath-4plus/train-00057-of-00064.parquet",
+        "finemath-4plus/train-00058-of-00064.parquet",
+        "finemath-4plus/train-00059-of-00064.parquet",
+        "finemath-4plus/train-00060-of-00064.parquet",
+        "finemath-4plus/train-00061-of-00064.parquet",
+        "finemath-4plus/train-00062-of-00064.parquet",
+        "finemath-4plus/train-00063-of-00064.parquet"
+      ],
+      "format": "parquet",
+      "name": "finemath_4plus",
+      "repo": "HuggingFaceTB/finemath",
+      "revision": "e92b25a616738fe95dc186b64dfb19f9c8525594",
+      "synthetic": false,
+      "weight_percent": 20
+    },
+    {
+      "files": [
+        "peS2o-0000.json.gz",
+        "peS2o-0001.json.gz",
+        "peS2o-0002.json.gz",
+        "peS2o-0003.json.gz",
+        "peS2o-0004.json.gz",
+        "peS2o-0005.json.gz",
+        "peS2o-0006.json.gz",
+        "peS2o-0007.json.gz",
+        "peS2o-0008.json.gz",
+        "peS2o-0009.json.gz",
+        "peS2o-0010.json.gz",
+        "peS2o-0011.json.gz",
+        "peS2o-0012.json.gz",
+        "peS2o-0013.json.gz",
+        "peS2o-0014.json.gz",
+        "peS2o-0015.json.gz",
+        "peS2o-0016.json.gz",
+        "peS2o-0017.json.gz",
+        "peS2o-0018.json.gz",
+        "peS2o-0019.json.gz",
+        "peS2o-0020.json.gz",
+        "peS2o-0021.json.gz",
+        "peS2o-0022.json.gz",
+        "peS2o-0023.json.gz",
+        "peS2o-0024.json.gz",
+        "peS2o-0025.json.gz",
+        "peS2o-0026.json.gz",
+        "peS2o-0027.json.gz",
+        "peS2o-0028.json.gz",
+        "peS2o-0029.json.gz",
+        "peS2o-0030.json.gz",
+        "peS2o-0031.json.gz",
+        "peS2o-0032.json.gz",
+        "peS2o-0033.json.gz",
+        "peS2o-0034.json.gz",
+        "peS2o-0035.json.gz",
+        "peS2o-0036.json.gz",
+        "peS2o-0037.json.gz",
+        "peS2o-0038.json.gz",
+        "peS2o-0039.json.gz",
+        "peS2o-0040.json.gz",
+        "peS2o-0041.json.gz",
+        "peS2o-0042.json.gz",
+        "peS2o-0043.json.gz",
+        "peS2o-0044.json.gz",
+        "peS2o-0045.json.gz",
+        "peS2o-0046.json.gz",
+        "peS2o-0047.json.gz",
+        "peS2o-0048.json.gz",
+        "peS2o-0049.json.gz",
+        "peS2o-0050.json.gz",
+        "peS2o-0051.json.gz",
+        "peS2o-0052.json.gz",
+        "peS2o-0053.json.gz",
+        "peS2o-0054.json.gz",
+        "peS2o-0055.json.gz",
+        "peS2o-0056.json.gz",
+        "peS2o-0057.json.gz",
+        "peS2o-0058.json.gz",
+        "peS2o-0059.json.gz",
+        "peS2o-0060.json.gz",
+        "peS2o-0061.json.gz",
+        "peS2o-0062.json.gz",
+        "peS2o-0063.json.gz",
+        "peS2o-0064.json.gz",
+        "peS2o-0065.json.gz",
+        "peS2o-0066.json.gz",
+        "peS2o-0067.json.gz",
+        "peS2o-0068.json.gz",
+        "peS2o-0069.json.gz",
+        "peS2o-0070.json.gz",
+        "peS2o-0071.json.gz",
+        "peS2o-0072.json.gz",
+        "peS2o-0073.json.gz",
+        "peS2o-0074.json.gz",
+        "peS2o-0075.json.gz",
+        "peS2o-0076.json.gz",
+        "peS2o-0077.json.gz",
+        "peS2o-0078.json.gz",
+        "peS2o-0079.json.gz",
+        "peS2o-0080.json.gz",
+        "peS2o-0081.json.gz",
+        "peS2o-0082.json.gz",
+        "peS2o-0083.json.gz",
+        "peS2o-0084.json.gz",
+        "peS2o-0085.json.gz",
+        "peS2o-0086.json.gz",
+        "peS2o-0087.json.gz",
+        "peS2o-0088.json.gz",
+        "peS2o-0089.json.gz",
+        "peS2o-0090.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "pes2o",
+      "repo": "common-pile/peS2o_filtered",
+      "revision": "297747513bfb0ff1fbf61ddad3b03319d0f04597",
+      "source_url": "https://huggingface.co/datasets/common-pile/peS2o_filtered",
+      "synthetic": false,
+      "weight_percent": 10
+    },
+    {
+      "files": [
+        "licensed_pubmed-0000.json.gz",
+        "licensed_pubmed-0001.json.gz",
+        "licensed_pubmed-0002.json.gz",
+        "licensed_pubmed-0003.json.gz",
+        "licensed_pubmed-0004.json.gz",
+        "licensed_pubmed-0005.json.gz",
+        "licensed_pubmed-0006.json.gz",
+        "licensed_pubmed-0007.json.gz",
+        "licensed_pubmed-0008.json.gz",
+        "licensed_pubmed-0009.json.gz",
+        "licensed_pubmed-0010.json.gz",
+        "licensed_pubmed-0011.json.gz",
+        "licensed_pubmed-0012.json.gz",
+        "licensed_pubmed-0013.json.gz",
+        "licensed_pubmed-0014.json.gz",
+        "licensed_pubmed-0015.json.gz",
+        "licensed_pubmed-0016.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "pubmed",
+      "repo": "common-pile/pubmed_filtered",
+      "revision": "c156f0569a92d8f2edc33cebe1f72f7d3e1cae84",
+      "source_url": "https://huggingface.co/datasets/common-pile/pubmed_filtered",
+      "synthetic": false,
+      "weight_percent": 3
+    },
+    {
+      "files": [
+        "arxiv-papers-0000.json.gz",
+        "arxiv-papers-0001.json.gz",
+        "arxiv-papers-0002.json.gz",
+        "arxiv-papers-0003.json.gz",
+        "arxiv-papers-0004.json.gz",
+        "arxiv-papers-0005.json.gz",
+        "arxiv-papers-0006.json.gz",
+        "arxiv-papers-0007.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "arxiv",
+      "repo": "common-pile/arxiv_papers_filtered",
+      "revision": "033cf7f53f9b348deec868c1a5a48484f3ee9e52",
+      "source_url": "https://huggingface.co/datasets/common-pile/arxiv_papers_filtered",
+      "synthetic": false,
+      "weight_percent": 2
+    },
+    {
+      "files": [
+        "pressbooks-0000.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "pressbooks",
+      "repo": "common-pile/pressbooks_filtered",
+      "revision": "1a1d3b50d77f834370f8eb4c0d174668dd1676bb",
+      "source_url": "https://huggingface.co/datasets/common-pile/pressbooks_filtered",
+      "synthetic": false,
+      "weight_percent": 3
+    },
+    {
+      "files": [
+        "libretexts-0000.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "libretexts",
+      "repo": "common-pile/libretexts_filtered",
+      "revision": "70388bca52b4a93515e14b1d56618fd7944988fd",
+      "source_url": "https://huggingface.co/datasets/common-pile/libretexts_filtered",
+      "synthetic": false,
+      "weight_percent": 3
+    },
+    {
+      "files": [
+        "doab-0000.json.gz",
+        "doab-0001.json.gz",
+        "doab-0002.json.gz",
+        "doab-0003.json.gz",
+        "doab-0004.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "doab",
+      "repo": "common-pile/doab_filtered",
+      "revision": "defb24ca72ef6aba6ce0228b669eec06dcfbffbc",
+      "source_url": "https://huggingface.co/datasets/common-pile/doab_filtered",
+      "synthetic": false,
+      "weight_percent": 4
+    },
+    {
+      "files": [
+        "project_gutenberg-dolma-0000.json.gz",
+        "project_gutenberg-dolma-0001.json.gz",
+        "project_gutenberg-dolma-0002.json.gz",
+        "project_gutenberg-dolma-0003.json.gz",
+        "project_gutenberg-dolma-0004.json.gz",
+        "project_gutenberg-dolma-0005.json.gz",
+        "project_gutenberg-dolma-0006.json.gz",
+        "project_gutenberg-dolma-0007.json.gz",
+        "project_gutenberg-dolma-0008.json.gz",
+        "project_gutenberg-dolma-0009.json.gz",
+        "project_gutenberg-dolma-0010.json.gz",
+        "project_gutenberg-dolma-0011.json.gz",
+        "project_gutenberg-dolma-0012.json.gz",
+        "project_gutenberg-dolma-0013.json.gz",
+        "project_gutenberg-dolma-0014.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "gutenberg",
+      "repo": "common-pile/project_gutenberg_filtered",
+      "revision": "3cdf6879c807f4e4e063f2ceb23bc268d8c29ab7",
+      "source_url": "https://huggingface.co/datasets/common-pile/project_gutenberg_filtered",
+      "synthetic": false,
+      "weight_percent": 5
+    },
+    {
+      "files": [
+        "public_library_1929_dolma-0000.json.gz",
+        "public_library_1929_dolma-0001.json.gz",
+        "public_library_1929_dolma-0002.json.gz",
+        "public_library_1929_dolma-0003.json.gz",
+        "public_library_1929_dolma-0004.json.gz",
+        "public_library_1929_dolma-0005.json.gz",
+        "public_library_1929_dolma-0006.json.gz",
+        "public_library_1929_dolma-0007.json.gz",
+        "public_library_1929_dolma-0008.json.gz",
+        "public_library_1929_dolma-0009.json.gz",
+        "public_library_1929_dolma-0010.json.gz",
+        "public_library_1929_dolma-0011.json.gz",
+        "public_library_1929_dolma-0012.json.gz",
+        "public_library_1929_dolma-0013.json.gz",
+        "public_library_1929_dolma-0014.json.gz",
+        "public_library_1929_dolma-0015.json.gz",
+        "public_library_1929_dolma-0016.json.gz",
+        "public_library_1929_dolma-0017.json.gz",
+        "public_library_1929_dolma-0018.json.gz",
+        "public_library_1929_dolma-0019.json.gz",
+        "public_library_1929_dolma-0020.json.gz",
+        "public_library_1929_dolma-0021.json.gz",
+        "public_library_1929_dolma-0022.json.gz",
+        "public_library_1929_dolma-0023.json.gz",
+        "public_library_1929_dolma-0024.json.gz",
+        "public_library_1929_dolma-0025.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "pre_1929_books",
+      "repo": "common-pile/pre_1929_books_filtered",
+      "revision": "23f9d96dbb1db3324bbc9fbfe1f8299cc799c4d1",
+      "synthetic": false,
+      "weight_percent": 7
+    },
+    {
+      "files": [
+        "loc_books_dolma-0000.json.gz",
+        "loc_books_dolma-0001.json.gz",
+        "loc_books_dolma-0002.json.gz",
+        "loc_books_dolma-0003.json.gz",
+        "loc_books_dolma-0004.json.gz",
+        "loc_books_dolma-0005.json.gz",
+        "loc_books_dolma-0006.json.gz",
+        "loc_books_dolma-0007.json.gz",
+        "loc_books_dolma-0008.json.gz",
+        "loc_books_dolma-0009.json.gz",
+        "loc_books_dolma-0010.json.gz",
+        "loc_books_dolma-0011.json.gz",
+        "loc_books_dolma-0012.json.gz",
+        "loc_books_dolma-0013.json.gz",
+        "loc_books_dolma-0014.json.gz",
+        "loc_books_dolma-0015.json.gz",
+        "loc_books_dolma-0016.json.gz",
+        "loc_books_dolma-0017.json.gz",
+        "loc_books_dolma-0018.json.gz",
+        "loc_books_dolma-0019.json.gz",
+        "loc_books_dolma-0020.json.gz",
+        "loc_books_dolma-0021.json.gz",
+        "loc_books_dolma-0022.json.gz",
+        "loc_books_dolma-0023.json.gz",
+        "loc_books_dolma-0024.json.gz"
+      ],
+      "format": "json.gz",
+      "name": "library_of_congress",
+      "repo": "common-pile/library_of_congress_filtered",
+      "revision": "56725c7aa1bb320703e22eb5f42903173d5bac3d",
+      "synthetic": false,
+      "weight_percent": 3
+    }
+  ],
+  "special_tokens": [
+    "<|pad|>",
+    "<|bos|>",
+    "<|eos|>",
+    "<|unk|>"
+  ],
+  "split_policy": "Same mixture-split-v1 parent hash: buckets2..9 training, bucket0 validation, bucket1 excluded; no held-out fitting",
+  "tokenizer": "Byte-level BPE with complete256bytealphabet; no normalization; digit groups up to3; explicit whitespace; no automatic BOS/EOS",
+  "trainer": {
+    "max_token_length": 32,
+    "min_frequency": 2
+  },
+  "training_utf8_bytes": 67108864,
+  "validation_utf8_bytes": 8388608,
+  "vocab_sizes": [
+    16384,
+    32768
+  ]
+}

research/REPORT.md

@@ -0,0 +1,138 @@
+# Tokenizer learning-curve research report
+
+## Overview
+
+18 of 18 requested candidates have measured Modal results. The grid doubles fitting text from 64 MiB through 2 GiB and compares 16,384, 32,768 and 50,000 vocabulary entries. All fresh candidates use the same source-weighted, parent-held-out evaluation corpus. A higher bytes-per-token value means fewer tokens for the same text; it does not establish better language-model quality. The strongest observed compression is 4.1786 bytes/token at 2048 MiB and 50,000 entries. This identifies a compression result, not a final model-optimal vocabulary.
+
+**Status: complete.**
+
+## Main findings
+
+- The strongest observed compression is 4.1786 bytes/token at 2048 MiB and 50,000 entries. This identifies a compression result, not a final model-optimal vocabulary.
+- At 16,384 entries, 64 MiB → 2 GiB reduces held-out token count by 0.52%; the final doubling contributes 0.003%. The smallest sample within 1% of this vocabulary's best observed token count is 64 MiB. The 1% tolerance is an analysis heuristic.
+- At 32,768 entries, 64 MiB → 2 GiB reduces held-out token count by 0.81%; the final doubling contributes 0.017%. The smallest sample within 1% of this vocabulary's best observed token count is 64 MiB. The 1% tolerance is an analysis heuristic.
+- At 50,000 entries, 64 MiB → 2 GiB reduces held-out token count by 0.94%; the final doubling contributes 0.031%. The smallest sample within 1% of this vocabulary's best observed token count is 64 MiB. The 1% tolerance is an analysis heuristic.
+- At 2 GiB, 32K uses 6.87% fewer tokens than 16K; 50K uses 9.88% fewer. At width 768, tied vocabulary parameters increase by 12.58M and 25.82M respectively. Untied input/output matrices double these deltas.
+- The published default is the 2 GiB / 32K candidate, a practical starting point balancing compression and vocabulary parameter overhead. All 18 variants are published for comparison. This choice has not been validated by matched-compute language-model training.
+
+
+## Experimental design
+
+- All 12 pinned source inventories from the scholarly/code plan are eligible; the experiment samples them rather than claiming to process every upstream record.
+- Nested per-source UTF-8 byte prefixes: 64, 128, 256, 512, 1,024 and 2,048 MiB. 16K means 16,384 entries, 32K means 32,768, and 50K means exactly 50,000.
+- Source weights follow the plan, applied to bytes for tokenizer fitting. This is distinct from the eventual model-training token mixture.
+- Shared 32 MiB evaluation corpus; parent-level train/evaluation/test separation before chunking. Test parents are excluded. The final evaluation caps each document and parent at 16 KiB to broaden coverage; fitting data are unchanged.
+- Byte-level BPE, full byte alphabet, no normalization or automatic boundaries; the frozen v1 pretokenization rules are held fixed. The original tokenizer is unchanged.
+- Seeded shard permutation, sequential records within shards, a 64 KiB document cap and 256 KiB parent cap for fitting. Exact whitespace-normalized duplicate chunks are removed; cross-split conflicts are quarantined before freezing.
+- CPU-only Modal: eight-CPU and 64 GiB memory requests, with a two-hour candidate timeout; at most three candidates concurrently. Peak RSS includes fitting and evaluation. CPU/memory requests are not hard caps; Rayon uses eight threads. Final rescoring uses two CPUs and 8 GiB RAM (requests and hard caps), at most three workers and a 20-minute timeout. Throughput includes encoding, decoding, counting and checks; peak RSS refers to the original fitting/evaluation process.
+
+## Evaluation coverage audit
+
+The initial evaluation had only five LibreTexts parent books. Before interpreting the candidate curves, evaluation was rebuilt with a 16 KiB parent cap and the same 32 MiB source quotas. Every candidate was rescored on that shared corpus. An initial 8 KiB cap could not fill the LibreTexts quota (636,017 bytes after exhausting the pinned inventory), so a fresh evaluation version used 16 KiB. Fitting text and tokenizer weights were preserved. Original fitting chunks were reserved in the dedup index to exclude exact normalized chunk overlap. Original result.json files remain on Modal; result-balanced-evaluation.json contains final metrics.
+
+## Frozen corpus composition
+
+| Source | Weight % | Fit MiB | Fit parents | Eval MiB | Eval parents |
+|---|---:|---:|---:|---:|---:|
+| fineweb_edu | 10 | 204.80 | 46886 | 3.20 | 845 |
+| code_python | 30 | 614.40 | 39130 | 9.60 | 1394 |
+| finemath_4plus | 20 | 409.60 | 87795 | 6.40 | 1573 |
+| pes2o | 10 | 204.80 | 7302 | 3.20 | 214 |
+| pubmed | 3 | 61.44 | 1689 | 0.96 | 66 |
+| arxiv | 2 | 40.96 | 859 | 0.64 | 42 |
+| pressbooks | 3 | 61.44 | 859 | 0.96 | 67 |
+| libretexts | 3 | 61.44 | 367 | 0.96 | 67 |
+| doab | 4 | 81.92 | 661 | 1.28 | 83 |
+| gutenberg | 5 | 102.40 | 1886 | 1.60 | 108 |
+| pre_1929_books | 7 | 143.36 | 2672 | 2.24 | 149 |
+| library_of_congress | 3 | 61.44 | 1211 | 0.96 | 65 |
+
+Parent counts use the adapter's source-scoped identifiers; equivalent works across sources are not fully reconciled.
+
+
+## Aggregate learning curve
+
+![Learning curve](learning-curve.png)
+
+Shading is a 95% parent-bootstrap interval, stratified by source with fixed source-byte weights (256 paired replicates). It describes evaluation-sample uncertainty, not uncertainty across training seeds.
+
+| MiB | Vocabulary requested / actual | Bytes/token | Token reduction vs 64 MiB | Fit seconds | Peak RSS MiB | Correctness |
+|---:|---:|---:|---:|---:|---:|---|
+| 64 | 16,384 / 16,384 | 3.7463 | 0.00% | 8.8 | 409 | pass |
+| 64 | 32,768 / 32,768 | 4.0110 | 0.00% | 9.8 | 422 | pass |
+| 64 | 50,000 / 50,000 | 4.1392 | 0.00% | 10.4 | 427 | pass |
+| 128 | 16,384 / 16,384 | 3.7571 | 0.29% | 13.8 | 618 | pass |
+| 128 | 32,768 / 32,768 | 4.0258 | 0.37% | 15.6 | 655 | pass |
+| 128 | 50,000 / 50,000 | 4.1549 | 0.38% | 14.7 | 646 | pass |
+| 256 | 16,384 / 16,384 | 3.7624 | 0.43% | 22.8 | 908 | pass |
+| 256 | 32,768 / 32,768 | 4.0338 | 0.57% | 26.7 | 975 | pass |
+| 256 | 50,000 / 50,000 | 4.1655 | 0.63% | 28.2 | 977 | pass |
+| 512 | 16,384 / 16,384 | 3.7646 | 0.49% | 40.4 | 1469 | pass |
+| 512 | 32,768 / 32,768 | 4.0395 | 0.71% | 46.0 | 1472 | pass |
+| 512 | 50,000 / 50,000 | 4.1729 | 0.81% | 41.7 | 1463 | pass |
+| 1024 | 16,384 / 16,384 | 3.7659 | 0.52% | 69.9 | 2281 | pass |
+| 1024 | 32,768 / 32,768 | 4.0431 | 0.79% | 72.4 | 2347 | pass |
+| 1024 | 50,000 / 50,000 | 4.1774 | 0.91% | 85.3 | 2406 | pass |
+| 2048 | 16,384 / 16,384 | 3.7660 | 0.52% | 125.6 | 3352 | pass |
+| 2048 | 32,768 / 32,768 | 4.0438 | 0.81% | 136.5 | 3510 | pass |
+| 2048 | 50,000 / 50,000 | 4.1786 | 0.94% | 159.3 | 3532 | pass |
+
+## Source-level comparisons
+
+![Source comparisons](source-comparisons.png)
+
+## Resource use and vocabulary utilization
+
+![Resource comparisons](resources-and-utilization.png)
+
+Unseen vocabulary includes special/control entries. A low utilization fraction alone is not evidence of a poor tokenizer on this finite evaluation set.
+
+## Gains from more fitting data
+
+![Data-size gains](data-size-gains.png)
+
+## Vocabulary-size tradeoffs
+
+| MiB | Vocabulary | Token reduction vs 16K | Paired 95% interval |
+|---:|---:|---:|---|
+| 64 | 16,384 | 0.00% | 0.00% to 0.00% |
+| 64 | 32,768 | 6.60% | 6.53% to 6.68% |
+| 64 | 50,000 | 9.49% | 9.41% to 9.61% |
+| 128 | 16,384 | 0.00% | 0.00% to 0.00% |
+| 128 | 32,768 | 6.67% | 6.60% to 6.76% |
+| 128 | 50,000 | 9.58% | 9.49% to 9.69% |
+| 256 | 16,384 | 0.00% | 0.00% to 0.00% |
+| 256 | 32,768 | 6.73% | 6.66% to 6.82% |
+| 256 | 50,000 | 9.68% | 9.58% to 9.79% |
+| 512 | 16,384 | 0.00% | 0.00% to 0.00% |
+| 512 | 32,768 | 6.80% | 6.74% to 6.89% |
+| 512 | 50,000 | 9.78% | 9.69% to 9.90% |
+| 1024 | 16,384 | 0.00% | 0.00% to 0.00% |
+| 1024 | 32,768 | 6.86% | 6.79% to 6.94% |
+| 1024 | 50,000 | 9.85% | 9.75% to 9.96% |
+| 2048 | 16,384 | 0.00% | 0.00% to 0.00% |
+| 2048 | 32,768 | 6.87% | 6.80% to 6.95% |
+| 2048 | 50,000 | 9.88% | 9.77% to 9.99% |
+
+Larger vocabularies reduce sequence length but add embedding/output parameters. At width 768 with tied embeddings, 32K adds 12.58M parameters over 16K; 50K adds 25.82M. Untied input/output weights double those increases. A final choice needs matched model-compute experiments.
+
+## Fixed-term fragmentation
+
+![Fragmentation probes](fragmentation-probes.png)
+
+These are diagnostic examples, not a representative task benchmark. Fewer pieces are not automatically better, particularly for numbers.
+
+## Interpretation and limitations
+
+Do not select a winner from an incomplete matrix. Once complete, examine same-vocabulary gains from each doubling, source regressions, fragmentation probes and resource use together. A compression plateau is evidence of diminishing tokenizer-fitting returns, not proof of downstream optimality.
+
+This experiment uses one corpus sampling seed. It does not establish stability across independent corpus samples. Near duplicates with changed wording or shifted chunk boundaries and equivalent books/papers across sources may remain. These limitations must be resolved for strong generalization claims. The frozen v1 collector split is not reproducible from its handoff, so its historical results are not mixed into this matched learning curve.
+
+Token-level language-model perplexity is not compared here. Downstream evaluation should use the same held-out text and bits per byte under matched model size and compute. No model quality or billing-cost improvement is claimed.
+
+## Reproduction and evidence
+
+- Published tokenizer variants: [jfrick/scholarly-code-tokenizer](https://huggingface.co/jfrick/scholarly-code-tokenizer).
+- `measurements.json` contains compact metrics, per-source comparisons, fragmentation probes and raw-result hashes. Full parent totals and candidate tokenizers remain on the Modal results volume.
+- `corpus-manifest.json` records source pins, sampling settings, corpus hashes and collection decisions; `evaluation-corpus-manifest.json` pins the broader held-out corpus. `modal-smoke.json` records remote test execution.
+- Run `python tokenizer_report.py <results-folder>` to regenerate this report and Matplotlib PNG/SVG/PDF figures from downloaded raw result JSONs.

research/corpus-manifest.json

@@ -0,0 +1,1046 @@
+{
+  "identity": {
+    "adapters": "34a07ec99175e0f1fb1df19c75b0e380a75179df3d516c1350ed68b672a28f9f",
+    "implementation": "5d992fec42941a33951b22ffb7468ae58811f6de79364cd9f5bcfd714ee772aa",
+    "recipe": {
+      "id": "scholarly-code-bpe-v1",
+      "library": "tokenizers==0.22.2",
+      "max_chunk_characters": 4096,
+      "max_chunks_per_document": 4,
+      "max_document_characters": 4000000,
+      "max_scanned_documents_per_source": 100000,
+      "model_target": "About150Mtotalparameters; embeddings included; model training not part of tokenizer job",
+      "sampling_weight_unit": "UTF-8 bytes before tokenizer exists; actual resulting token shares reported",
+      "seed": 20261016,
+      "selection_rule": "Prefer16K unless32K reduces total held-out tokens by at least10% and no individual source needs over2%moretokens; all roundtrip and bytecoverage checks must pass",
+      "sources": [
+        {
+          "files": [
+            "sample/10BT/000_00000.parquet",
+            "sample/10BT/001_00000.parquet",
+            "sample/10BT/002_00000.parquet",
+            "sample/10BT/003_00000.parquet",
+            "sample/10BT/004_00000.parquet",
+            "sample/10BT/005_00000.parquet",
+            "sample/10BT/006_00000.parquet",
+            "sample/10BT/007_00000.parquet",
+            "sample/10BT/008_00000.parquet",
+            "sample/10BT/009_00000.parquet",
+            "sample/10BT/010_00000.parquet",
+            "sample/10BT/011_00000.parquet",
+            "sample/10BT/012_00000.parquet",
+            "sample/10BT/013_00000.parquet"
+          ],
+          "format": "parquet",
+          "name": "fineweb_edu",
+          "repo": "HuggingFaceFW/fineweb-edu",
+          "revision": "87f09149ef4734204d70ed1d046ddc9ca3f2b8f9",
+          "source_url": "https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu",
+          "synthetic": false,
+          "weight_percent": 10
+        },
+        {
+          "adapter": "codeparrot",
+          "allowed_spdx_licenses": [
+            "MIT",
+            "Apache-2.0",
+            "BSD-2-Clause",
+            "BSD-3-Clause",
+            "ISC"
+          ],
+          "files": [
+            "file-000000000001.json.gz",
+            "file-000000000002.json.gz",
+            "file-000000000003.json.gz",
+            "file-000000000004.json.gz",
+            "file-000000000005.json.gz",
+            "file-000000000006.json.gz",
+            "file-000000000007.json.gz",
+            "file-000000000008.json.gz",
+            "file-000000000009.json.gz",
+            "file-000000000010.json.gz",
+            "file-000000000011.json.gz",
+            "file-000000000012.json.gz",
+            "file-000000000013.json.gz",
+            "file-000000000014.json.gz",
+            "file-000000000015.json.gz",
+            "file-000000000016.json.gz",
+            "file-000000000017.json.gz",
+            "file-000000000018.json.gz",
+            "file-000000000019.json.gz",
+            "file-000000000020.json.gz",
+            "file-000000000021.json.gz",
+            "file-000000000022.json.gz",
+            "file-000000000023.json.gz",
+            "file-000000000024.json.gz",
+            "file-000000000025.json.gz",
+            "file-000000000026.json.gz",
+            "file-000000000027.json.gz",
+            "file-000000000028.json.gz",
+            "file-000000000029.json.gz",
+            "file-000000000030.json.gz",
+            "file-000000000031.json.gz",
+            "file-000000000032.json.gz",
+            "file-000000000033.json.gz",
+            "file-000000000034.json.gz",
+            "file-000000000035.json.gz",
+            "file-000000000036.json.gz",
+            "file-000000000037.json.gz",
+            "file-000000000038.json.gz",
+            "file-000000000039.json.gz",
+            "file-000000000040.json.gz",
+            "file-000000000041.json.gz",
+            "file-000000000042.json.gz",
+            "file-000000000043.json.gz",
+            "file-000000000044.json.gz",
+            "file-000000000045.json.gz",
+            "file-000000000046.json.gz",
+            "file-000000000047.json.gz",
+            "file-000000000048.json.gz",
+            "file-000000000049.json.gz",
+            "file-000000000050.json.gz",
+            "file-000000000051.json.gz",
+            "file-000000000052.json.gz",
+            "file-000000000053.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "code_python",
+          "repo": "codeparrot/codeparrot-clean-train",
+          "revision": "3e6ab65f2864931e041f6a82db9b5a6ec2b71ab4",
+          "source_url": "https://huggingface.co/datasets/codeparrot/codeparrot-clean-train",
+          "synthetic": false,
+          "weight_percent": 30
+        },
+        {
+          "files": [
+            "finemath-4plus/train-00000-of-00064.parquet",
+            "finemath-4plus/train-00001-of-00064.parquet",
+            "finemath-4plus/train-00002-of-00064.parquet",
+            "finemath-4plus/train-00003-of-00064.parquet",
+            "finemath-4plus/train-00004-of-00064.parquet",
+            "finemath-4plus/train-00005-of-00064.parquet",
+            "finemath-4plus/train-00006-of-00064.parquet",
+            "finemath-4plus/train-00007-of-00064.parquet",
+            "finemath-4plus/train-00008-of-00064.parquet",
+            "finemath-4plus/train-00009-of-00064.parquet",
+            "finemath-4plus/train-00010-of-00064.parquet",
+            "finemath-4plus/train-00011-of-00064.parquet",
+            "finemath-4plus/train-00012-of-00064.parquet",
+            "finemath-4plus/train-00013-of-00064.parquet",
+            "finemath-4plus/train-00014-of-00064.parquet",
+            "finemath-4plus/train-00015-of-00064.parquet",
+            "finemath-4plus/train-00016-of-00064.parquet",
+            "finemath-4plus/train-00017-of-00064.parquet",
+            "finemath-4plus/train-00018-of-00064.parquet",
+            "finemath-4plus/train-00019-of-00064.parquet",
+            "finemath-4plus/train-00020-of-00064.parquet",
+            "finemath-4plus/train-00021-of-00064.parquet",
+            "finemath-4plus/train-00022-of-00064.parquet",
+            "finemath-4plus/train-00023-of-00064.parquet",
+            "finemath-4plus/train-00024-of-00064.parquet",
+            "finemath-4plus/train-00025-of-00064.parquet",
+            "finemath-4plus/train-00026-of-00064.parquet",
+            "finemath-4plus/train-00027-of-00064.parquet",
+            "finemath-4plus/train-00028-of-00064.parquet",
+            "finemath-4plus/train-00029-of-00064.parquet",
+            "finemath-4plus/train-00030-of-00064.parquet",
+            "finemath-4plus/train-00031-of-00064.parquet",
+            "finemath-4plus/train-00032-of-00064.parquet",
+            "finemath-4plus/train-00033-of-00064.parquet",
+            "finemath-4plus/train-00034-of-00064.parquet",
+            "finemath-4plus/train-00035-of-00064.parquet",
+            "finemath-4plus/train-00036-of-00064.parquet",
+            "finemath-4plus/train-00037-of-00064.parquet",
+            "finemath-4plus/train-00038-of-00064.parquet",
+            "finemath-4plus/train-00039-of-00064.parquet",
+            "finemath-4plus/train-00040-of-00064.parquet",
+            "finemath-4plus/train-00041-of-00064.parquet",
+            "finemath-4plus/train-00042-of-00064.parquet",
+            "finemath-4plus/train-00043-of-00064.parquet",
+            "finemath-4plus/train-00044-of-00064.parquet",
+            "finemath-4plus/train-00045-of-00064.parquet",
+            "finemath-4plus/train-00046-of-00064.parquet",
+            "finemath-4plus/train-00047-of-00064.parquet",
+            "finemath-4plus/train-00048-of-00064.parquet",
+            "finemath-4plus/train-00049-of-00064.parquet",
+            "finemath-4plus/train-00050-of-00064.parquet",
+            "finemath-4plus/train-00051-of-00064.parquet",
+            "finemath-4plus/train-00052-of-00064.parquet",
+            "finemath-4plus/train-00053-of-00064.parquet",
+            "finemath-4plus/train-00054-of-00064.parquet",
+            "finemath-4plus/train-00055-of-00064.parquet",
+            "finemath-4plus/train-00056-of-00064.parquet",
+            "finemath-4plus/train-00057-of-00064.parquet",
+            "finemath-4plus/train-00058-of-00064.parquet",
+            "finemath-4plus/train-00059-of-00064.parquet",
+            "finemath-4plus/train-00060-of-00064.parquet",
+            "finemath-4plus/train-00061-of-00064.parquet",
+            "finemath-4plus/train-00062-of-00064.parquet",
+            "finemath-4plus/train-00063-of-00064.parquet"
+          ],
+          "format": "parquet",
+          "name": "finemath_4plus",
+          "repo": "HuggingFaceTB/finemath",
+          "revision": "e92b25a616738fe95dc186b64dfb19f9c8525594",
+          "synthetic": false,
+          "weight_percent": 20
+        },
+        {
+          "files": [
+            "peS2o-0000.json.gz",
+            "peS2o-0001.json.gz",
+            "peS2o-0002.json.gz",
+            "peS2o-0003.json.gz",
+            "peS2o-0004.json.gz",
+            "peS2o-0005.json.gz",
+            "peS2o-0006.json.gz",
+            "peS2o-0007.json.gz",
+            "peS2o-0008.json.gz",
+            "peS2o-0009.json.gz",
+            "peS2o-0010.json.gz",
+            "peS2o-0011.json.gz",
+            "peS2o-0012.json.gz",
+            "peS2o-0013.json.gz",
+            "peS2o-0014.json.gz",
+            "peS2o-0015.json.gz",
+            "peS2o-0016.json.gz",
+            "peS2o-0017.json.gz",
+            "peS2o-0018.json.gz",
+            "peS2o-0019.json.gz",
+            "peS2o-0020.json.gz",
+            "peS2o-0021.json.gz",
+            "peS2o-0022.json.gz",
+            "peS2o-0023.json.gz",
+            "peS2o-0024.json.gz",
+            "peS2o-0025.json.gz",
+            "peS2o-0026.json.gz",
+            "peS2o-0027.json.gz",
+            "peS2o-0028.json.gz",
+            "peS2o-0029.json.gz",
+            "peS2o-0030.json.gz",
+            "peS2o-0031.json.gz",
+            "peS2o-0032.json.gz",
+            "peS2o-0033.json.gz",
+            "peS2o-0034.json.gz",
+            "peS2o-0035.json.gz",
+            "peS2o-0036.json.gz",
+            "peS2o-0037.json.gz",
+            "peS2o-0038.json.gz",
+            "peS2o-0039.json.gz",
+            "peS2o-0040.json.gz",
+            "peS2o-0041.json.gz",
+            "peS2o-0042.json.gz",
+            "peS2o-0043.json.gz",
+            "peS2o-0044.json.gz",
+            "peS2o-0045.json.gz",
+            "peS2o-0046.json.gz",
+            "peS2o-0047.json.gz",
+            "peS2o-0048.json.gz",
+            "peS2o-0049.json.gz",
+            "peS2o-0050.json.gz",
+            "peS2o-0051.json.gz",
+            "peS2o-0052.json.gz",
+            "peS2o-0053.json.gz",
+            "peS2o-0054.json.gz",
+            "peS2o-0055.json.gz",
+            "peS2o-0056.json.gz",
+            "peS2o-0057.json.gz",
+            "peS2o-0058.json.gz",
+            "peS2o-0059.json.gz",
+            "peS2o-0060.json.gz",
+            "peS2o-0061.json.gz",
+            "peS2o-0062.json.gz",
+            "peS2o-0063.json.gz",
+            "peS2o-0064.json.gz",
+            "peS2o-0065.json.gz",
+            "peS2o-0066.json.gz",
+            "peS2o-0067.json.gz",
+            "peS2o-0068.json.gz",
+            "peS2o-0069.json.gz",
+            "peS2o-0070.json.gz",
+            "peS2o-0071.json.gz",
+            "peS2o-0072.json.gz",
+            "peS2o-0073.json.gz",
+            "peS2o-0074.json.gz",
+            "peS2o-0075.json.gz",
+            "peS2o-0076.json.gz",
+            "peS2o-0077.json.gz",
+            "peS2o-0078.json.gz",
+            "peS2o-0079.json.gz",
+            "peS2o-0080.json.gz",
+            "peS2o-0081.json.gz",
+            "peS2o-0082.json.gz",
+            "peS2o-0083.json.gz",
+            "peS2o-0084.json.gz",
+            "peS2o-0085.json.gz",
+            "peS2o-0086.json.gz",
+            "peS2o-0087.json.gz",
+            "peS2o-0088.json.gz",
+            "peS2o-0089.json.gz",
+            "peS2o-0090.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pes2o",
+          "repo": "common-pile/peS2o_filtered",
+          "revision": "297747513bfb0ff1fbf61ddad3b03319d0f04597",
+          "source_url": "https://huggingface.co/datasets/common-pile/peS2o_filtered",
+          "synthetic": false,
+          "weight_percent": 10
+        },
+        {
+          "files": [
+            "licensed_pubmed-0000.json.gz",
+            "licensed_pubmed-0001.json.gz",
+            "licensed_pubmed-0002.json.gz",
+            "licensed_pubmed-0003.json.gz",
+            "licensed_pubmed-0004.json.gz",
+            "licensed_pubmed-0005.json.gz",
+            "licensed_pubmed-0006.json.gz",
+            "licensed_pubmed-0007.json.gz",
+            "licensed_pubmed-0008.json.gz",
+            "licensed_pubmed-0009.json.gz",
+            "licensed_pubmed-0010.json.gz",
+            "licensed_pubmed-0011.json.gz",
+            "licensed_pubmed-0012.json.gz",
+            "licensed_pubmed-0013.json.gz",
+            "licensed_pubmed-0014.json.gz",
+            "licensed_pubmed-0015.json.gz",
+            "licensed_pubmed-0016.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pubmed",
+          "repo": "common-pile/pubmed_filtered",
+          "revision": "c156f0569a92d8f2edc33cebe1f72f7d3e1cae84",
+          "source_url": "https://huggingface.co/datasets/common-pile/pubmed_filtered",
+          "synthetic": false,
+          "weight_percent": 3
+        },
+        {
+          "files": [
+            "arxiv-papers-0000.json.gz",
+            "arxiv-papers-0001.json.gz",
+            "arxiv-papers-0002.json.gz",
+            "arxiv-papers-0003.json.gz",
+            "arxiv-papers-0004.json.gz",
+            "arxiv-papers-0005.json.gz",
+            "arxiv-papers-0006.json.gz",
+            "arxiv-papers-0007.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "arxiv",
+          "repo": "common-pile/arxiv_papers_filtered",
+          "revision": "033cf7f53f9b348deec868c1a5a48484f3ee9e52",
+          "source_url": "https://huggingface.co/datasets/common-pile/arxiv_papers_filtered",
+          "synthetic": false,
+          "weight_percent": 2
+        },
+        {
+          "files": [
+            "pressbooks-0000.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pressbooks",
+          "repo": "common-pile/pressbooks_filtered",
+          "revision": "1a1d3b50d77f834370f8eb4c0d174668dd1676bb",
+          "source_url": "https://huggingface.co/datasets/common-pile/pressbooks_filtered",
+          "synthetic": false,
+          "weight_percent": 3
+        },
+        {
+          "files": [
+            "libretexts-0000.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "libretexts",
+          "repo": "common-pile/libretexts_filtered",
+          "revision": "70388bca52b4a93515e14b1d56618fd7944988fd",
+          "source_url": "https://huggingface.co/datasets/common-pile/libretexts_filtered",
+          "synthetic": false,
+          "weight_percent": 3
+        },
+        {
+          "files": [
+            "doab-0000.json.gz",
+            "doab-0001.json.gz",
+            "doab-0002.json.gz",
+            "doab-0003.json.gz",
+            "doab-0004.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "doab",
+          "repo": "common-pile/doab_filtered",
+          "revision": "defb24ca72ef6aba6ce0228b669eec06dcfbffbc",
+          "source_url": "https://huggingface.co/datasets/common-pile/doab_filtered",
+          "synthetic": false,
+          "weight_percent": 4
+        },
+        {
+          "files": [
+            "project_gutenberg-dolma-0000.json.gz",
+            "project_gutenberg-dolma-0001.json.gz",
+            "project_gutenberg-dolma-0002.json.gz",
+            "project_gutenberg-dolma-0003.json.gz",
+            "project_gutenberg-dolma-0004.json.gz",
+            "project_gutenberg-dolma-0005.json.gz",
+            "project_gutenberg-dolma-0006.json.gz",
+            "project_gutenberg-dolma-0007.json.gz",
+            "project_gutenberg-dolma-0008.json.gz",
+            "project_gutenberg-dolma-0009.json.gz",
+            "project_gutenberg-dolma-0010.json.gz",
+            "project_gutenberg-dolma-0011.json.gz",
+            "project_gutenberg-dolma-0012.json.gz",
+            "project_gutenberg-dolma-0013.json.gz",
+            "project_gutenberg-dolma-0014.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "gutenberg",
+          "repo": "common-pile/project_gutenberg_filtered",
+          "revision": "3cdf6879c807f4e4e063f2ceb23bc268d8c29ab7",
+          "source_url": "https://huggingface.co/datasets/common-pile/project_gutenberg_filtered",
+          "synthetic": false,
+          "weight_percent": 5
+        },
+        {
+          "files": [
+            "public_library_1929_dolma-0000.json.gz",
+            "public_library_1929_dolma-0001.json.gz",
+            "public_library_1929_dolma-0002.json.gz",
+            "public_library_1929_dolma-0003.json.gz",
+            "public_library_1929_dolma-0004.json.gz",
+            "public_library_1929_dolma-0005.json.gz",
+            "public_library_1929_dolma-0006.json.gz",
+            "public_library_1929_dolma-0007.json.gz",
+            "public_library_1929_dolma-0008.json.gz",
+            "public_library_1929_dolma-0009.json.gz",
+            "public_library_1929_dolma-0010.json.gz",
+            "public_library_1929_dolma-0011.json.gz",
+            "public_library_1929_dolma-0012.json.gz",
+            "public_library_1929_dolma-0013.json.gz",
+            "public_library_1929_dolma-0014.json.gz",
+            "public_library_1929_dolma-0015.json.gz",
+            "public_library_1929_dolma-0016.json.gz",
+            "public_library_1929_dolma-0017.json.gz",
+            "public_library_1929_dolma-0018.json.gz",
+            "public_library_1929_dolma-0019.json.gz",
+            "public_library_1929_dolma-0020.json.gz",
+            "public_library_1929_dolma-0021.json.gz",
+            "public_library_1929_dolma-0022.json.gz",
+            "public_library_1929_dolma-0023.json.gz",
+            "public_library_1929_dolma-0024.json.gz",
+            "public_library_1929_dolma-0025.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pre_1929_books",
+          "repo": "common-pile/pre_1929_books_filtered",
+          "revision": "23f9d96dbb1db3324bbc9fbfe1f8299cc799c4d1",
+          "synthetic": false,
+          "weight_percent": 7
+        },
+        {
+          "files": [
+            "loc_books_dolma-0000.json.gz",
+            "loc_books_dolma-0001.json.gz",
+            "loc_books_dolma-0002.json.gz",
+            "loc_books_dolma-0003.json.gz",
+            "loc_books_dolma-0004.json.gz",
+            "loc_books_dolma-0005.json.gz",
+            "loc_books_dolma-0006.json.gz",
+            "loc_books_dolma-0007.json.gz",
+            "loc_books_dolma-0008.json.gz",
+            "loc_books_dolma-0009.json.gz",
+            "loc_books_dolma-0010.json.gz",
+            "loc_books_dolma-0011.json.gz",
+            "loc_books_dolma-0012.json.gz",
+            "loc_books_dolma-0013.json.gz",
+            "loc_books_dolma-0014.json.gz",
+            "loc_books_dolma-0015.json.gz",
+            "loc_books_dolma-0016.json.gz",
+            "loc_books_dolma-0017.json.gz",
+            "loc_books_dolma-0018.json.gz",
+            "loc_books_dolma-0019.json.gz",
+            "loc_books_dolma-0020.json.gz",
+            "loc_books_dolma-0021.json.gz",
+            "loc_books_dolma-0022.json.gz",
+            "loc_books_dolma-0023.json.gz",
+            "loc_books_dolma-0024.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "library_of_congress",
+          "repo": "common-pile/library_of_congress_filtered",
+          "revision": "56725c7aa1bb320703e22eb5f42903173d5bac3d",
+          "synthetic": false,
+          "weight_percent": 3
+        }
+      ],
+      "special_tokens": [
+        "<|pad|>",
+        "<|bos|>",
+        "<|eos|>",
+        "<|unk|>"
+      ],
+      "split_policy": "Same mixture-split-v1 parent hash: buckets2..9 training, bucket0 validation, bucket1 excluded; no held-out fitting",
+      "tokenizer": "Byte-level BPE with complete256bytealphabet; no normalization; digit groups up to3; explicit whitespace; no automatic BOS/EOS",
+      "trainer": {
+        "max_token_length": 32,
+        "min_frequency": 2
+      },
+      "training_utf8_bytes": 67108864,
+      "validation_utf8_bytes": 8388608,
+      "vocab_sizes": [
+        16384,
+        32768
+      ]
+    },
+    "spec": {
+      "chunk_bytes": 8192,
+      "cpu": 8,
+      "evaluation_mib": 32,
+      "id": "scholarly-tokenizer-learning-curve-v1",
+      "library": "tokenizers==0.22.2",
+      "max_document_bytes": 65536,
+      "max_parallel_candidates": 3,
+      "max_parent_bytes": 262144,
+      "max_token_length": 32,
+      "memory_mib": 65536,
+      "min_frequency": 2,
+      "sampling": "Seeded permutation of every pinned shard per source; source-weighted UTF-8 byte quotas; nested per-source prefixes",
+      "seed": 20261016,
+      "sizes_mib": [
+        64,
+        128,
+        256,
+        512,
+        1024,
+        2048
+      ],
+      "source_recipe": "tokenizer/recipe-v1.json",
+      "split_policy": "scholarly-parent-v2; bucket 0 evaluation, bucket 1 excluded, buckets 2-9 fitting",
+      "timeout_seconds": 7200,
+      "vocab_sizes": [
+        16384,
+        32768,
+        50000
+      ]
+    }
+  },
+  "sources": {
+    "fineweb_edu": {
+      "weight_percent": 10,
+      "train": {
+        "path": "fineweb_edu-train.jsonl",
+        "sha256": "d50f84b59d36c8cbb81ab7d6e40d285d0464bb6f7d73131e8786603828ae8462",
+        "bytes": 214748365,
+        "chunks": 55847,
+        "parents": 46886,
+        "target_bytes": 214748365
+      },
+      "validation": {
+        "path": "fineweb_edu-validation.jsonl",
+        "sha256": "87c64308527569fd2917e3d1d2fcd971c791bf35de3ac66f219a85fdf6038bae",
+        "bytes": 3355444,
+        "chunks": 895,
+        "parents": 762,
+        "target_bytes": 3355444
+      }
+    },
+    "code_python": {
+      "weight_percent": 30,
+      "train": {
+        "path": "code_python-train.jsonl",
+        "sha256": "5e1c04e310375d9922e008142e7d9813f56e541918b729dd7764eaec3ffbadea",
+        "bytes": 644245095,
+        "chunks": 120955,
+        "parents": 39130,
+        "target_bytes": 644245095
+      },
+      "validation": {
+        "path": "code_python-validation.jsonl",
+        "sha256": "73d44fdab095e9ea0483993698d9d657630489aa5e38b4d1e205b956d1390178",
+        "bytes": 10066330,
+        "chunks": 1836,
+        "parents": 885,
+        "target_bytes": 10066330
+      }
+    },
+    "finemath_4plus": {
+      "weight_percent": 20,
+      "train": {
+        "path": "finemath_4plus-train.jsonl",
+        "sha256": "d496fb1ff323a5abecfbc7554372eb1ee64871ffe09b93a815acef264fdfe450",
+        "bytes": 429496730,
+        "chunks": 108405,
+        "parents": 87795,
+        "target_bytes": 429496730
+      },
+      "validation": {
+        "path": "finemath_4plus-validation.jsonl",
+        "sha256": "a5ce0255f2f3564287979f71ffe5d779fd4779bb7c4fec90e04b2a1d573dc249",
+        "bytes": 6710887,
+        "chunks": 1706,
+        "parents": 1387,
+        "target_bytes": 6710887
+      }
+    },
+    "pes2o": {
+      "weight_percent": 10,
+      "train": {
+        "path": "pes2o-train.jsonl",
+        "sha256": "ad20f8a9b28a5eae7f9e85f0cacfd599d8b4aedb3a013e4635f8460155425669",
+        "bytes": 214748365,
+        "chunks": 29712,
+        "parents": 7302,
+        "target_bytes": 214748365
+      },
+      "validation": {
+        "path": "pes2o-validation.jsonl",
+        "sha256": "2a159c9e0d6f943659961deba809ca2e4c0fbe142891d0401a1074eaa95ea693",
+        "bytes": 3355444,
+        "chunks": 463,
+        "parents": 104,
+        "target_bytes": 3355444
+      }
+    },
+    "pubmed": {
+      "weight_percent": 3,
+      "train": {
+        "path": "pubmed-train.jsonl",
+        "sha256": "dfaf74b557b8f20ad922e646176041a5ae55adcb679ed52d8b918e8a4d6d780f",
+        "bytes": 64424510,
+        "chunks": 8628,
+        "parents": 1689,
+        "target_bytes": 64424510
+      },
+      "validation": {
+        "path": "pubmed-validation.jsonl",
+        "sha256": "78ddd383a8a5b305b2669971f323af6a29b862e30838036c3b02371e47c14f19",
+        "bytes": 1006633,
+        "chunks": 137,
+        "parents": 28,
+        "target_bytes": 1006633
+      }
+    },
+    "arxiv": {
+      "weight_percent": 2,
+      "train": {
+        "path": "arxiv-train.jsonl",
+        "sha256": "6f9056d49ccd62e3a0ea7eb602f49863ab2c9717c861e2b5bbbbea56cb05cbd3",
+        "bytes": 42949673,
+        "chunks": 5500,
+        "parents": 859,
+        "target_bytes": 42949673
+      },
+      "validation": {
+        "path": "arxiv-validation.jsonl",
+        "sha256": "70adc71f486d26203877e1d074f87c24c2970132c1eac946be4c395af0a365c5",
+        "bytes": 671089,
+        "chunks": 84,
+        "parents": 12,
+        "target_bytes": 671089
+      }
+    },
+    "pressbooks": {
+      "weight_percent": 3,
+      "train": {
+        "path": "pressbooks-train.jsonl",
+        "sha256": "8aa0e00775c0706e948b9db49679b8a970d00d5d93734bb247d4764b3638f0ec",
+        "bytes": 64424510,
+        "chunks": 15879,
+        "parents": 859,
+        "target_bytes": 64424510
+      },
+      "validation": {
+        "path": "pressbooks-validation.jsonl",
+        "sha256": "0fbe94b2da8e62e1775d4b92e0a14a8f729264033c700e502cf0ba0ca5d027f8",
+        "bytes": 1006633,
+        "chunks": 346,
+        "parents": 36,
+        "target_bytes": 1006633
+      }
+    },
+    "libretexts": {
+      "weight_percent": 3,
+      "train": {
+        "path": "libretexts-train.jsonl",
+        "sha256": "9b2ab410d4f99b402504f822c99a5b3ca5869be500d9d5708c9f7532d853053b",
+        "bytes": 64424509,
+        "chunks": 12650,
+        "parents": 367,
+        "target_bytes": 64424509
+      },
+      "validation": {
+        "path": "libretexts-validation.jsonl",
+        "sha256": "5f267258cf1bb816af8aaab9bf0b41d05733f192ebd566309e1babc4c5dded29",
+        "bytes": 1006632,
+        "chunks": 237,
+        "parents": 5,
+        "target_bytes": 1006632
+      }
+    },
+    "doab": {
+      "weight_percent": 4,
+      "train": {
+        "path": "doab-train.jsonl",
+        "sha256": "83cdfce4c8eda6c73a890d924af9d42659cd1acdf442feff07b1a9a1d36fc3eb",
+        "bytes": 85899345,
+        "chunks": 14622,
+        "parents": 661,
+        "target_bytes": 85899345
+      },
+      "validation": {
+        "path": "doab-validation.jsonl",
+        "sha256": "b87c6c74be3c3f66310015957cb7b391d5bae0bd89ebcec0e75427dad94b46a7",
+        "bytes": 1342177,
+        "chunks": 219,
+        "parents": 13,
+        "target_bytes": 1342177
+      }
+    },
+    "gutenberg": {
+      "weight_percent": 5,
+      "train": {
+        "path": "gutenberg-train.jsonl",
+        "sha256": "4c8d6979e49b27ef5d83edfe75ce6f2c1c98c8a3eadd9c6815feef0707bfa9d2",
+        "bytes": 107374182,
+        "chunks": 13371,
+        "parents": 1886,
+        "target_bytes": 107374182
+      },
+      "validation": {
+        "path": "gutenberg-validation.jsonl",
+        "sha256": "6f60664c0e3a3fe4251984bb599e1b921d4ffc0df6bf3220a32db38a8861efff",
+        "bytes": 1677721,
+        "chunks": 209,
+        "parents": 28,
+        "target_bytes": 1677721
+      }
+    },
+    "pre_1929_books": {
+      "weight_percent": 7,
+      "train": {
+        "path": "pre_1929_books-train.jsonl",
+        "sha256": "a53d4c3fda6c816f93f492469ca7be2f87d56698e298fe5b0b7e497e2e793169",
+        "bytes": 150323855,
+        "chunks": 18706,
+        "parents": 2672,
+        "target_bytes": 150323855
+      },
+      "validation": {
+        "path": "pre_1929_books-validation.jsonl",
+        "sha256": "0a2f43f51a940cb55e6f131770a4cbc217b79cee63fc8ce148b5a02366dfa5c9",
+        "bytes": 2348810,
+        "chunks": 290,
+        "parents": 38,
+        "target_bytes": 2348810
+      }
+    },
+    "library_of_congress": {
+      "weight_percent": 3,
+      "train": {
+        "path": "library_of_congress-train.jsonl",
+        "sha256": "59edbc730be46a94d37c7f461cb714a9609460579566d14fc1cb495559dba75b",
+        "bytes": 64424509,
+        "chunks": 8071,
+        "parents": 1211,
+        "target_bytes": 64424509
+      },
+      "validation": {
+        "path": "library_of_congress-validation.jsonl",
+        "sha256": "c981b6ffb7ceebf8e57cd360b98c246df7d011648e67fc0419d0f9591ff006a5",
+        "bytes": 1006632,
+        "chunks": 130,
+        "parents": 23,
+        "target_bytes": 1006632
+      }
+    }
+  },
+  "sampling_unit": "UTF-8 bytes, not model-training token proportions",
+  "limitations": [
+    "Seeded shard order with sequential records, not a global reservoir",
+    "Parent identity not reconciled across different datasets",
+    "Normalized exact-chunk deduplication; no fuzzy near-duplicate guarantee",
+    "One sampling seed; candidate differences do not measure seed variance",
+    "Fresh candidates share a held-out split; frozen v1 split overlap unknown"
+  ],
+  "decisions": [
+    [
+      "arxiv",
+      "duplicate_chunk",
+      4
+    ],
+    [
+      "arxiv",
+      "excluded_test",
+      114
+    ],
+    [
+      "arxiv",
+      "sampled",
+      993
+    ],
+    [
+      "code_python",
+      "code_license",
+      59739
+    ],
+    [
+      "code_python",
+      "cross_split_quarantine",
+      33
+    ],
+    [
+      "code_python",
+      "duplicate_chunk",
+      311
+    ],
+    [
+      "code_python",
+      "excluded_test",
+      9590
+    ],
+    [
+      "code_python",
+      "parent_cap",
+      1718
+    ],
+    [
+      "code_python",
+      "sampled",
+      82830
+    ],
+    [
+      "doab",
+      "cross_split_quarantine",
+      1
+    ],
+    [
+      "doab",
+      "duplicate_chunk",
+      6
+    ],
+    [
+      "doab",
+      "empty",
+      3
+    ],
+    [
+      "doab",
+      "excluded_test",
+      1433
+    ],
+    [
+      "doab",
+      "parent_cap",
+      6159
+    ],
+    [
+      "doab",
+      "sampled",
+      8987
+    ],
+    [
+      "finemath_4plus",
+      "cross_split_quarantine",
+      1
+    ],
+    [
+      "finemath_4plus",
+      "duplicate_chunk",
+      13
+    ],
+    [
+      "finemath_4plus",
+      "excluded_test",
+      10858
+    ],
+    [
+      "finemath_4plus",
+      "sampled",
+      98732
+    ],
+    [
+      "fineweb_edu",
+      "cross_split_quarantine",
+      6
+    ],
+    [
+      "fineweb_edu",
+      "duplicate_chunk",
+      11
+    ],
+    [
+      "fineweb_edu",
+      "excluded_test",
+      5872
+    ],
+    [
+      "fineweb_edu",
+      "sampled",
+      52930
+    ],
+    [
+      "gutenberg",
+      "cross_split_quarantine",
+      15
+    ],
+    [
+      "gutenberg",
+      "duplicate_chunk",
+      62
+    ],
+    [
+      "gutenberg",
+      "excluded_test",
+      237
+    ],
+    [
+      "gutenberg",
+      "oversize",
+      4
+    ],
+    [
+      "gutenberg",
+      "sampled",
+      2133
+    ],
+    [
+      "library_of_congress",
+      "cross_split_quarantine",
+      1
+    ],
+    [
+      "library_of_congress",
+      "duplicate_chunk",
+      15
+    ],
+    [
+      "library_of_congress",
+      "empty",
+      4
+    ],
+    [
+      "library_of_congress",
+      "excluded_test",
+      150
+    ],
+    [
+      "library_of_congress",
+      "oversize",
+      2
+    ],
+    [
+      "library_of_congress",
+      "sampled",
+      1367
+    ],
+    [
+      "libretexts",
+      "duplicate_chunk",
+      1
+    ],
+    [
+      "libretexts",
+      "empty",
+      3
+    ],
+    [
+      "libretexts",
+      "excluded_test",
+      2401
+    ],
+    [
+      "libretexts",
+      "parent_cap",
+      10276
+    ],
+    [
+      "libretexts",
+      "sampled",
+      11516
+    ],
+    [
+      "pes2o",
+      "duplicate_chunk",
+      4
+    ],
+    [
+      "pes2o",
+      "excluded_test",
+      909
+    ],
+    [
+      "pes2o",
+      "sampled",
+      8236
+    ],
+    [
+      "pre_1929_books",
+      "cross_split_quarantine",
+      7
+    ],
+    [
+      "pre_1929_books",
+      "duplicate_chunk",
+      28
+    ],
+    [
+      "pre_1929_books",
+      "empty",
+      11
+    ],
+    [
+      "pre_1929_books",
+      "excluded_test",
+      350
+    ],
+    [
+      "pre_1929_books",
+      "oversize",
+      7
+    ],
+    [
+      "pre_1929_books",
+      "sampled",
+      3026
+    ],
+    [
+      "pressbooks",
+      "duplicate_chunk",
+      2
+    ],
+    [
+      "pressbooks",
+      "empty",
+      3
+    ],
+    [
+      "pressbooks",
+      "excluded_test",
+      1195
+    ],
+    [
+      "pressbooks",
+      "parent_cap",
+      387
+    ],
+    [
+      "pressbooks",
+      "sampled",
+      14151
+    ],
+    [
+      "pubmed",
+      "duplicate_chunk",
+      4
+    ],
+    [
+      "pubmed",
+      "excluded_test",
+      182
+    ],
+    [
+      "pubmed",
+      "sampled",
+      1898
+    ]
+  ]
+}

research/evaluation-corpus-manifest.json

@@ -0,0 +1,885 @@
+{
+  "identity": {
+    "adapters": "34a07ec99175e0f1fb1df19c75b0e380a75179df3d516c1350ed68b672a28f9f",
+    "implementation": "5d992fec42941a33951b22ffb7468ae58811f6de79364cd9f5bcfd714ee772aa",
+    "recipe": {
+      "id": "scholarly-code-bpe-v1",
+      "library": "tokenizers==0.22.2",
+      "max_chunk_characters": 4096,
+      "max_chunks_per_document": 4,
+      "max_document_characters": 4000000,
+      "max_scanned_documents_per_source": 100000,
+      "model_target": "About150Mtotalparameters; embeddings included; model training not part of tokenizer job",
+      "sampling_weight_unit": "UTF-8 bytes before tokenizer exists; actual resulting token shares reported",
+      "seed": 20261016,
+      "selection_rule": "Prefer16K unless32K reduces total held-out tokens by at least10% and no individual source needs over2%moretokens; all roundtrip and bytecoverage checks must pass",
+      "sources": [
+        {
+          "files": [
+            "sample/10BT/000_00000.parquet",
+            "sample/10BT/001_00000.parquet",
+            "sample/10BT/002_00000.parquet",
+            "sample/10BT/003_00000.parquet",
+            "sample/10BT/004_00000.parquet",
+            "sample/10BT/005_00000.parquet",
+            "sample/10BT/006_00000.parquet",
+            "sample/10BT/007_00000.parquet",
+            "sample/10BT/008_00000.parquet",
+            "sample/10BT/009_00000.parquet",
+            "sample/10BT/010_00000.parquet",
+            "sample/10BT/011_00000.parquet",
+            "sample/10BT/012_00000.parquet",
+            "sample/10BT/013_00000.parquet"
+          ],
+          "format": "parquet",
+          "name": "fineweb_edu",
+          "repo": "HuggingFaceFW/fineweb-edu",
+          "revision": "87f09149ef4734204d70ed1d046ddc9ca3f2b8f9",
+          "source_url": "https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu",
+          "synthetic": false,
+          "weight_percent": 10
+        },
+        {
+          "adapter": "codeparrot",
+          "allowed_spdx_licenses": [
+            "MIT",
+            "Apache-2.0",
+            "BSD-2-Clause",
+            "BSD-3-Clause",
+            "ISC"
+          ],
+          "files": [
+            "file-000000000001.json.gz",
+            "file-000000000002.json.gz",
+            "file-000000000003.json.gz",
+            "file-000000000004.json.gz",
+            "file-000000000005.json.gz",
+            "file-000000000006.json.gz",
+            "file-000000000007.json.gz",
+            "file-000000000008.json.gz",
+            "file-000000000009.json.gz",
+            "file-000000000010.json.gz",
+            "file-000000000011.json.gz",
+            "file-000000000012.json.gz",
+            "file-000000000013.json.gz",
+            "file-000000000014.json.gz",
+            "file-000000000015.json.gz",
+            "file-000000000016.json.gz",
+            "file-000000000017.json.gz",
+            "file-000000000018.json.gz",
+            "file-000000000019.json.gz",
+            "file-000000000020.json.gz",
+            "file-000000000021.json.gz",
+            "file-000000000022.json.gz",
+            "file-000000000023.json.gz",
+            "file-000000000024.json.gz",
+            "file-000000000025.json.gz",
+            "file-000000000026.json.gz",
+            "file-000000000027.json.gz",
+            "file-000000000028.json.gz",
+            "file-000000000029.json.gz",
+            "file-000000000030.json.gz",
+            "file-000000000031.json.gz",
+            "file-000000000032.json.gz",
+            "file-000000000033.json.gz",
+            "file-000000000034.json.gz",
+            "file-000000000035.json.gz",
+            "file-000000000036.json.gz",
+            "file-000000000037.json.gz",
+            "file-000000000038.json.gz",
+            "file-000000000039.json.gz",
+            "file-000000000040.json.gz",
+            "file-000000000041.json.gz",
+            "file-000000000042.json.gz",
+            "file-000000000043.json.gz",
+            "file-000000000044.json.gz",
+            "file-000000000045.json.gz",
+            "file-000000000046.json.gz",
+            "file-000000000047.json.gz",
+            "file-000000000048.json.gz",
+            "file-000000000049.json.gz",
+            "file-000000000050.json.gz",
+            "file-000000000051.json.gz",
+            "file-000000000052.json.gz",
+            "file-000000000053.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "code_python",
+          "repo": "codeparrot/codeparrot-clean-train",
+          "revision": "3e6ab65f2864931e041f6a82db9b5a6ec2b71ab4",
+          "source_url": "https://huggingface.co/datasets/codeparrot/codeparrot-clean-train",
+          "synthetic": false,
+          "weight_percent": 30
+        },
+        {
+          "files": [
+            "finemath-4plus/train-00000-of-00064.parquet",
+            "finemath-4plus/train-00001-of-00064.parquet",
+            "finemath-4plus/train-00002-of-00064.parquet",
+            "finemath-4plus/train-00003-of-00064.parquet",
+            "finemath-4plus/train-00004-of-00064.parquet",
+            "finemath-4plus/train-00005-of-00064.parquet",
+            "finemath-4plus/train-00006-of-00064.parquet",
+            "finemath-4plus/train-00007-of-00064.parquet",
+            "finemath-4plus/train-00008-of-00064.parquet",
+            "finemath-4plus/train-00009-of-00064.parquet",
+            "finemath-4plus/train-00010-of-00064.parquet",
+            "finemath-4plus/train-00011-of-00064.parquet",
+            "finemath-4plus/train-00012-of-00064.parquet",
+            "finemath-4plus/train-00013-of-00064.parquet",
+            "finemath-4plus/train-00014-of-00064.parquet",
+            "finemath-4plus/train-00015-of-00064.parquet",
+            "finemath-4plus/train-00016-of-00064.parquet",
+            "finemath-4plus/train-00017-of-00064.parquet",
+            "finemath-4plus/train-00018-of-00064.parquet",
+            "finemath-4plus/train-00019-of-00064.parquet",
+            "finemath-4plus/train-00020-of-00064.parquet",
+            "finemath-4plus/train-00021-of-00064.parquet",
+            "finemath-4plus/train-00022-of-00064.parquet",
+            "finemath-4plus/train-00023-of-00064.parquet",
+            "finemath-4plus/train-00024-of-00064.parquet",
+            "finemath-4plus/train-00025-of-00064.parquet",
+            "finemath-4plus/train-00026-of-00064.parquet",
+            "finemath-4plus/train-00027-of-00064.parquet",
+            "finemath-4plus/train-00028-of-00064.parquet",
+            "finemath-4plus/train-00029-of-00064.parquet",
+            "finemath-4plus/train-00030-of-00064.parquet",
+            "finemath-4plus/train-00031-of-00064.parquet",
+            "finemath-4plus/train-00032-of-00064.parquet",
+            "finemath-4plus/train-00033-of-00064.parquet",
+            "finemath-4plus/train-00034-of-00064.parquet",
+            "finemath-4plus/train-00035-of-00064.parquet",
+            "finemath-4plus/train-00036-of-00064.parquet",
+            "finemath-4plus/train-00037-of-00064.parquet",
+            "finemath-4plus/train-00038-of-00064.parquet",
+            "finemath-4plus/train-00039-of-00064.parquet",
+            "finemath-4plus/train-00040-of-00064.parquet",
+            "finemath-4plus/train-00041-of-00064.parquet",
+            "finemath-4plus/train-00042-of-00064.parquet",
+            "finemath-4plus/train-00043-of-00064.parquet",
+            "finemath-4plus/train-00044-of-00064.parquet",
+            "finemath-4plus/train-00045-of-00064.parquet",
+            "finemath-4plus/train-00046-of-00064.parquet",
+            "finemath-4plus/train-00047-of-00064.parquet",
+            "finemath-4plus/train-00048-of-00064.parquet",
+            "finemath-4plus/train-00049-of-00064.parquet",
+            "finemath-4plus/train-00050-of-00064.parquet",
+            "finemath-4plus/train-00051-of-00064.parquet",
+            "finemath-4plus/train-00052-of-00064.parquet",
+            "finemath-4plus/train-00053-of-00064.parquet",
+            "finemath-4plus/train-00054-of-00064.parquet",
+            "finemath-4plus/train-00055-of-00064.parquet",
+            "finemath-4plus/train-00056-of-00064.parquet",
+            "finemath-4plus/train-00057-of-00064.parquet",
+            "finemath-4plus/train-00058-of-00064.parquet",
+            "finemath-4plus/train-00059-of-00064.parquet",
+            "finemath-4plus/train-00060-of-00064.parquet",
+            "finemath-4plus/train-00061-of-00064.parquet",
+            "finemath-4plus/train-00062-of-00064.parquet",
+            "finemath-4plus/train-00063-of-00064.parquet"
+          ],
+          "format": "parquet",
+          "name": "finemath_4plus",
+          "repo": "HuggingFaceTB/finemath",
+          "revision": "e92b25a616738fe95dc186b64dfb19f9c8525594",
+          "synthetic": false,
+          "weight_percent": 20
+        },
+        {
+          "files": [
+            "peS2o-0000.json.gz",
+            "peS2o-0001.json.gz",
+            "peS2o-0002.json.gz",
+            "peS2o-0003.json.gz",
+            "peS2o-0004.json.gz",
+            "peS2o-0005.json.gz",
+            "peS2o-0006.json.gz",
+            "peS2o-0007.json.gz",
+            "peS2o-0008.json.gz",
+            "peS2o-0009.json.gz",
+            "peS2o-0010.json.gz",
+            "peS2o-0011.json.gz",
+            "peS2o-0012.json.gz",
+            "peS2o-0013.json.gz",
+            "peS2o-0014.json.gz",
+            "peS2o-0015.json.gz",
+            "peS2o-0016.json.gz",
+            "peS2o-0017.json.gz",
+            "peS2o-0018.json.gz",
+            "peS2o-0019.json.gz",
+            "peS2o-0020.json.gz",
+            "peS2o-0021.json.gz",
+            "peS2o-0022.json.gz",
+            "peS2o-0023.json.gz",
+            "peS2o-0024.json.gz",
+            "peS2o-0025.json.gz",
+            "peS2o-0026.json.gz",
+            "peS2o-0027.json.gz",
+            "peS2o-0028.json.gz",
+            "peS2o-0029.json.gz",
+            "peS2o-0030.json.gz",
+            "peS2o-0031.json.gz",
+            "peS2o-0032.json.gz",
+            "peS2o-0033.json.gz",
+            "peS2o-0034.json.gz",
+            "peS2o-0035.json.gz",
+            "peS2o-0036.json.gz",
+            "peS2o-0037.json.gz",
+            "peS2o-0038.json.gz",
+            "peS2o-0039.json.gz",
+            "peS2o-0040.json.gz",
+            "peS2o-0041.json.gz",
+            "peS2o-0042.json.gz",
+            "peS2o-0043.json.gz",
+            "peS2o-0044.json.gz",
+            "peS2o-0045.json.gz",
+            "peS2o-0046.json.gz",
+            "peS2o-0047.json.gz",
+            "peS2o-0048.json.gz",
+            "peS2o-0049.json.gz",
+            "peS2o-0050.json.gz",
+            "peS2o-0051.json.gz",
+            "peS2o-0052.json.gz",
+            "peS2o-0053.json.gz",
+            "peS2o-0054.json.gz",
+            "peS2o-0055.json.gz",
+            "peS2o-0056.json.gz",
+            "peS2o-0057.json.gz",
+            "peS2o-0058.json.gz",
+            "peS2o-0059.json.gz",
+            "peS2o-0060.json.gz",
+            "peS2o-0061.json.gz",
+            "peS2o-0062.json.gz",
+            "peS2o-0063.json.gz",
+            "peS2o-0064.json.gz",
+            "peS2o-0065.json.gz",
+            "peS2o-0066.json.gz",
+            "peS2o-0067.json.gz",
+            "peS2o-0068.json.gz",
+            "peS2o-0069.json.gz",
+            "peS2o-0070.json.gz",
+            "peS2o-0071.json.gz",
+            "peS2o-0072.json.gz",
+            "peS2o-0073.json.gz",
+            "peS2o-0074.json.gz",
+            "peS2o-0075.json.gz",
+            "peS2o-0076.json.gz",
+            "peS2o-0077.json.gz",
+            "peS2o-0078.json.gz",
+            "peS2o-0079.json.gz",
+            "peS2o-0080.json.gz",
+            "peS2o-0081.json.gz",
+            "peS2o-0082.json.gz",
+            "peS2o-0083.json.gz",
+            "peS2o-0084.json.gz",
+            "peS2o-0085.json.gz",
+            "peS2o-0086.json.gz",
+            "peS2o-0087.json.gz",
+            "peS2o-0088.json.gz",
+            "peS2o-0089.json.gz",
+            "peS2o-0090.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pes2o",
+          "repo": "common-pile/peS2o_filtered",
+          "revision": "297747513bfb0ff1fbf61ddad3b03319d0f04597",
+          "source_url": "https://huggingface.co/datasets/common-pile/peS2o_filtered",
+          "synthetic": false,
+          "weight_percent": 10
+        },
+        {
+          "files": [
+            "licensed_pubmed-0000.json.gz",
+            "licensed_pubmed-0001.json.gz",
+            "licensed_pubmed-0002.json.gz",
+            "licensed_pubmed-0003.json.gz",
+            "licensed_pubmed-0004.json.gz",
+            "licensed_pubmed-0005.json.gz",
+            "licensed_pubmed-0006.json.gz",
+            "licensed_pubmed-0007.json.gz",
+            "licensed_pubmed-0008.json.gz",
+            "licensed_pubmed-0009.json.gz",
+            "licensed_pubmed-0010.json.gz",
+            "licensed_pubmed-0011.json.gz",
+            "licensed_pubmed-0012.json.gz",
+            "licensed_pubmed-0013.json.gz",
+            "licensed_pubmed-0014.json.gz",
+            "licensed_pubmed-0015.json.gz",
+            "licensed_pubmed-0016.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pubmed",
+          "repo": "common-pile/pubmed_filtered",
+          "revision": "c156f0569a92d8f2edc33cebe1f72f7d3e1cae84",
+          "source_url": "https://huggingface.co/datasets/common-pile/pubmed_filtered",
+          "synthetic": false,
+          "weight_percent": 3
+        },
+        {
+          "files": [
+            "arxiv-papers-0000.json.gz",
+            "arxiv-papers-0001.json.gz",
+            "arxiv-papers-0002.json.gz",
+            "arxiv-papers-0003.json.gz",
+            "arxiv-papers-0004.json.gz",
+            "arxiv-papers-0005.json.gz",
+            "arxiv-papers-0006.json.gz",
+            "arxiv-papers-0007.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "arxiv",
+          "repo": "common-pile/arxiv_papers_filtered",
+          "revision": "033cf7f53f9b348deec868c1a5a48484f3ee9e52",
+          "source_url": "https://huggingface.co/datasets/common-pile/arxiv_papers_filtered",
+          "synthetic": false,
+          "weight_percent": 2
+        },
+        {
+          "files": [
+            "pressbooks-0000.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pressbooks",
+          "repo": "common-pile/pressbooks_filtered",
+          "revision": "1a1d3b50d77f834370f8eb4c0d174668dd1676bb",
+          "source_url": "https://huggingface.co/datasets/common-pile/pressbooks_filtered",
+          "synthetic": false,
+          "weight_percent": 3
+        },
+        {
+          "files": [
+            "libretexts-0000.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "libretexts",
+          "repo": "common-pile/libretexts_filtered",
+          "revision": "70388bca52b4a93515e14b1d56618fd7944988fd",
+          "source_url": "https://huggingface.co/datasets/common-pile/libretexts_filtered",
+          "synthetic": false,
+          "weight_percent": 3
+        },
+        {
+          "files": [
+            "doab-0000.json.gz",
+            "doab-0001.json.gz",
+            "doab-0002.json.gz",
+            "doab-0003.json.gz",
+            "doab-0004.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "doab",
+          "repo": "common-pile/doab_filtered",
+          "revision": "defb24ca72ef6aba6ce0228b669eec06dcfbffbc",
+          "source_url": "https://huggingface.co/datasets/common-pile/doab_filtered",
+          "synthetic": false,
+          "weight_percent": 4
+        },
+        {
+          "files": [
+            "project_gutenberg-dolma-0000.json.gz",
+            "project_gutenberg-dolma-0001.json.gz",
+            "project_gutenberg-dolma-0002.json.gz",
+            "project_gutenberg-dolma-0003.json.gz",
+            "project_gutenberg-dolma-0004.json.gz",
+            "project_gutenberg-dolma-0005.json.gz",
+            "project_gutenberg-dolma-0006.json.gz",
+            "project_gutenberg-dolma-0007.json.gz",
+            "project_gutenberg-dolma-0008.json.gz",
+            "project_gutenberg-dolma-0009.json.gz",
+            "project_gutenberg-dolma-0010.json.gz",
+            "project_gutenberg-dolma-0011.json.gz",
+            "project_gutenberg-dolma-0012.json.gz",
+            "project_gutenberg-dolma-0013.json.gz",
+            "project_gutenberg-dolma-0014.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "gutenberg",
+          "repo": "common-pile/project_gutenberg_filtered",
+          "revision": "3cdf6879c807f4e4e063f2ceb23bc268d8c29ab7",
+          "source_url": "https://huggingface.co/datasets/common-pile/project_gutenberg_filtered",
+          "synthetic": false,
+          "weight_percent": 5
+        },
+        {
+          "files": [
+            "public_library_1929_dolma-0000.json.gz",
+            "public_library_1929_dolma-0001.json.gz",
+            "public_library_1929_dolma-0002.json.gz",
+            "public_library_1929_dolma-0003.json.gz",
+            "public_library_1929_dolma-0004.json.gz",
+            "public_library_1929_dolma-0005.json.gz",
+            "public_library_1929_dolma-0006.json.gz",
+            "public_library_1929_dolma-0007.json.gz",
+            "public_library_1929_dolma-0008.json.gz",
+            "public_library_1929_dolma-0009.json.gz",
+            "public_library_1929_dolma-0010.json.gz",
+            "public_library_1929_dolma-0011.json.gz",
+            "public_library_1929_dolma-0012.json.gz",
+            "public_library_1929_dolma-0013.json.gz",
+            "public_library_1929_dolma-0014.json.gz",
+            "public_library_1929_dolma-0015.json.gz",
+            "public_library_1929_dolma-0016.json.gz",
+            "public_library_1929_dolma-0017.json.gz",
+            "public_library_1929_dolma-0018.json.gz",
+            "public_library_1929_dolma-0019.json.gz",
+            "public_library_1929_dolma-0020.json.gz",
+            "public_library_1929_dolma-0021.json.gz",
+            "public_library_1929_dolma-0022.json.gz",
+            "public_library_1929_dolma-0023.json.gz",
+            "public_library_1929_dolma-0024.json.gz",
+            "public_library_1929_dolma-0025.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "pre_1929_books",
+          "repo": "common-pile/pre_1929_books_filtered",
+          "revision": "23f9d96dbb1db3324bbc9fbfe1f8299cc799c4d1",
+          "synthetic": false,
+          "weight_percent": 7
+        },
+        {
+          "files": [
+            "loc_books_dolma-0000.json.gz",
+            "loc_books_dolma-0001.json.gz",
+            "loc_books_dolma-0002.json.gz",
+            "loc_books_dolma-0003.json.gz",
+            "loc_books_dolma-0004.json.gz",
+            "loc_books_dolma-0005.json.gz",
+            "loc_books_dolma-0006.json.gz",
+            "loc_books_dolma-0007.json.gz",
+            "loc_books_dolma-0008.json.gz",
+            "loc_books_dolma-0009.json.gz",
+            "loc_books_dolma-0010.json.gz",
+            "loc_books_dolma-0011.json.gz",
+            "loc_books_dolma-0012.json.gz",
+            "loc_books_dolma-0013.json.gz",
+            "loc_books_dolma-0014.json.gz",
+            "loc_books_dolma-0015.json.gz",
+            "loc_books_dolma-0016.json.gz",
+            "loc_books_dolma-0017.json.gz",
+            "loc_books_dolma-0018.json.gz",
+            "loc_books_dolma-0019.json.gz",
+            "loc_books_dolma-0020.json.gz",
+            "loc_books_dolma-0021.json.gz",
+            "loc_books_dolma-0022.json.gz",
+            "loc_books_dolma-0023.json.gz",
+            "loc_books_dolma-0024.json.gz"
+          ],
+          "format": "json.gz",
+          "name": "library_of_congress",
+          "repo": "common-pile/library_of_congress_filtered",
+          "revision": "56725c7aa1bb320703e22eb5f42903173d5bac3d",
+          "synthetic": false,
+          "weight_percent": 3
+        }
+      ],
+      "special_tokens": [
+        "<|pad|>",
+        "<|bos|>",
+        "<|eos|>",
+        "<|unk|>"
+      ],
+      "split_policy": "Same mixture-split-v1 parent hash: buckets2..9 training, bucket0 validation, bucket1 excluded; no held-out fitting",
+      "tokenizer": "Byte-level BPE with complete256bytealphabet; no normalization; digit groups up to3; explicit whitespace; no automatic BOS/EOS",
+      "trainer": {
+        "max_token_length": 32,
+        "min_frequency": 2
+      },
+      "training_utf8_bytes": 67108864,
+      "validation_utf8_bytes": 8388608,
+      "vocab_sizes": [
+        16384,
+        32768
+      ]
+    },
+    "spec": {
+      "chunk_bytes": 8192,
+      "cpu": 8,
+      "evaluation_builder_sha256": "43b518effaffbde7f77ad293f2fafec2407b716bbc3c8b126896d93993f43e95",
+      "evaluation_mib": 32,
+      "id": "scholarly-tokenizer-learning-curve-v1",
+      "library": "tokenizers==0.22.2",
+      "max_document_bytes": 16384,
+      "max_parallel_candidates": 3,
+      "max_parent_bytes": 16384,
+      "max_token_length": 32,
+      "memory_mib": 65536,
+      "min_frequency": 2,
+      "purpose": "evaluation_only",
+      "sampling": "Seeded permutation of every pinned shard per source; source-weighted UTF-8 byte quotas; nested per-source prefixes",
+      "seed": 20261016,
+      "sizes_mib": [
+        64,
+        128,
+        256,
+        512,
+        1024,
+        2048
+      ],
+      "source_recipe": "tokenizer/recipe-v1.json",
+      "split_policy": "scholarly-parent-v2; bucket 0 evaluation, bucket 1 excluded, buckets 2-9 fitting",
+      "timeout_seconds": 7200,
+      "training_bytes_override": 0,
+      "training_manifest_sha256": "be5d52967413adaa08e74fc0cec028305c9d1297f9c07425efb3631b744fe39a",
+      "vocab_sizes": [
+        16384,
+        32768,
+        50000
+      ]
+    }
+  },
+  "sources": {
+    "fineweb_edu": {
+      "weight_percent": 10,
+      "train": {
+        "path": "fineweb_edu-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "fineweb_edu-validation.jsonl",
+        "sha256": "b74453ddb191b017b62b12affe8c3f33e475ad59cfbcac4033744b09cf09cb85",
+        "bytes": 3355444,
+        "chunks": 927,
+        "parents": 845,
+        "target_bytes": 3355444
+      }
+    },
+    "code_python": {
+      "weight_percent": 30,
+      "train": {
+        "path": "code_python-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "code_python-validation.jsonl",
+        "sha256": "611010954c9c48ccad9b9a28f7377d56ab180373b5e7eb76fe87ab723b2b8040",
+        "bytes": 10066330,
+        "chunks": 2092,
+        "parents": 1394,
+        "target_bytes": 10066330
+      }
+    },
+    "finemath_4plus": {
+      "weight_percent": 20,
+      "train": {
+        "path": "finemath_4plus-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "finemath_4plus-validation.jsonl",
+        "sha256": "5e4900e466d88b6020e11edc11c3efc06260e17331501051d606c8842ed9ae04",
+        "bytes": 6710887,
+        "chunks": 1789,
+        "parents": 1573,
+        "target_bytes": 6710887
+      }
+    },
+    "pes2o": {
+      "weight_percent": 10,
+      "train": {
+        "path": "pes2o-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "pes2o-validation.jsonl",
+        "sha256": "15fcd5792eef7622cdc9d3dfa43ca2e4b4fb862fff38fb25f1968f73777275d5",
+        "bytes": 3355444,
+        "chunks": 422,
+        "parents": 214,
+        "target_bytes": 3355444
+      }
+    },
+    "pubmed": {
+      "weight_percent": 3,
+      "train": {
+        "path": "pubmed-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "pubmed-validation.jsonl",
+        "sha256": "bc9307825f7107ce91a55a7bcaf4c463f9a6b1181287d2b64f693d8384305f6d",
+        "bytes": 1006633,
+        "chunks": 128,
+        "parents": 66,
+        "target_bytes": 1006633
+      }
+    },
+    "arxiv": {
+      "weight_percent": 2,
+      "train": {
+        "path": "arxiv-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "arxiv-validation.jsonl",
+        "sha256": "c4c613c23bf9bc0f2f911cd85b9d0140d7362aeb4bfcbbe04243ed2c5ba52322",
+        "bytes": 671089,
+        "chunks": 83,
+        "parents": 42,
+        "target_bytes": 671089
+      }
+    },
+    "pressbooks": {
+      "weight_percent": 3,
+      "train": {
+        "path": "pressbooks-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "pressbooks-validation.jsonl",
+        "sha256": "b5133de3edfe0193752574439f8db3174f2e22347fe03942ce64dd7f80a6c86f",
+        "bytes": 1006633,
+        "chunks": 359,
+        "parents": 67,
+        "target_bytes": 1006633
+      }
+    },
+    "libretexts": {
+      "weight_percent": 3,
+      "train": {
+        "path": "libretexts-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "libretexts-validation.jsonl",
+        "sha256": "4cf44ccf05237f507a46743ca2e40cf5fa51036fd0981167335d132a4f816811",
+        "bytes": 1006632,
+        "chunks": 285,
+        "parents": 67,
+        "target_bytes": 1006632
+      }
+    },
+    "doab": {
+      "weight_percent": 4,
+      "train": {
+        "path": "doab-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "doab-validation.jsonl",
+        "sha256": "ff90e9e8aa284ae5db82e45baff8adea9c916b5e59daaa021b027b60dcc228ac",
+        "bytes": 1342177,
+        "chunks": 315,
+        "parents": 83,
+        "target_bytes": 1342177
+      }
+    },
+    "gutenberg": {
+      "weight_percent": 5,
+      "train": {
+        "path": "gutenberg-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "gutenberg-validation.jsonl",
+        "sha256": "f916d780fe05136c3f4c8ee8efd659dc8fa01e21354e03ce9810613fd71f69ea",
+        "bytes": 1677721,
+        "chunks": 210,
+        "parents": 108,
+        "target_bytes": 1677721
+      }
+    },
+    "pre_1929_books": {
+      "weight_percent": 7,
+      "train": {
+        "path": "pre_1929_books-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "pre_1929_books-validation.jsonl",
+        "sha256": "bdd135393f90737d82814df31a22d032288bbcda758b815b029ea39f4bf93128",
+        "bytes": 2348810,
+        "chunks": 291,
+        "parents": 149,
+        "target_bytes": 2348810
+      }
+    },
+    "library_of_congress": {
+      "weight_percent": 3,
+      "train": {
+        "path": "library_of_congress-train.jsonl",
+        "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "bytes": 0,
+        "chunks": 0,
+        "parents": 0,
+        "target_bytes": 0
+      },
+      "validation": {
+        "path": "library_of_congress-validation.jsonl",
+        "sha256": "2a6194aa8cd4dfd08a2af10c3a52f8284248b02d8b19acef4643a8586555198c",
+        "bytes": 1006632,
+        "chunks": 127,
+        "parents": 65,
+        "target_bytes": 1006632
+      }
+    }
+  },
+  "sampling_unit": "UTF-8 bytes, not model-training token proportions",
+  "limitations": [
+    "Seeded shard order with sequential records, not a global reservoir",
+    "Parent identity not reconciled across different datasets",
+    "Normalized exact-chunk deduplication; no fuzzy near-duplicate guarantee",
+    "One sampling seed; candidate differences do not measure seed variance",
+    "Fresh candidates share a held-out split; frozen v1 split overlap unknown"
+  ],
+  "decisions": [
+    [
+      "arxiv",
+      "duplicate_chunk",
+      1
+    ],
+    [
+      "arxiv",
+      "sampled",
+      42
+    ],
+    [
+      "code_python",
+      "cross_split_quarantine",
+      2
+    ],
+    [
+      "code_python",
+      "duplicate_chunk",
+      2
+    ],
+    [
+      "code_python",
+      "parent_cap",
+      177
+    ],
+    [
+      "code_python",
+      "sampled",
+      1654
+    ],
+    [
+      "doab",
+      "parent_cap",
+      1226
+    ],
+    [
+      "doab",
+      "sampled",
+      252
+    ],
+    [
+      "finemath_4plus",
+      "sampled",
+      1573
+    ],
+    [
+      "fineweb_edu",
+      "sampled",
+      845
+    ],
+    [
+      "gutenberg",
+      "sampled",
+      108
+    ],
+    [
+      "library_of_congress",
+      "sampled",
+      65
+    ],
+    [
+      "libretexts",
+      "parent_cap",
+      3340
+    ],
+    [
+      "libretexts",
+      "sampled",
+      250
+    ],
+    [
+      "pes2o",
+      "cross_split_quarantine",
+      1
+    ],
+    [
+      "pes2o",
+      "duplicate_chunk",
+      1
+    ],
+    [
+      "pes2o",
+      "sampled",
+      214
+    ],
+    [
+      "pre_1929_books",
+      "cross_split_quarantine",
+      1
+    ],
+    [
+      "pre_1929_books",
+      "duplicate_chunk",
+      1
+    ],
+    [
+      "pre_1929_books",
+      "sampled",
+      149
+    ],
+    [
+      "pressbooks",
+      "parent_cap",
+      482
+    ],
+    [
+      "pressbooks",
+      "sampled",
+      339
+    ],
+    [
+      "pubmed",
+      "sampled",
+      66
+    ]
+  ]
+}

research/execution-history.json

@@ -0,0 +1,65 @@
+{
+  "date": "2026-10-16",
+  "active_run_id": "learning-curve-20261016-v2",
+  "status": "completed",
+  "modal_app_url": "https://modal.com/apps/radiantai/main/ap-A86MK9NsJAHwye2lcVuxyv",
+  "local_tests_passed": 21,
+  "remote_research_tests_passed": 5,
+  "attempts": [
+    {
+      "app_id": "ap-zrRin23bx0E5h11kA5Pxec",
+      "outcome": "image_configuration_failed_before_worker_execution",
+      "resolution": "Moved image environment configuration before add_local_* mounts."
+    },
+    {
+      "app_id": "ap-RkN0bYeRHWNVTdndcNZ7pR",
+      "outcome": "remote_smoke_failed_before_collection",
+      "resolution": "Resolved adapter source hash from the imported function rather than a checkout-specific filesystem path."
+    },
+    {
+      "app_id": "ap-5NdqpPYSaOsIIAbQdHc80V",
+      "outcome": "remote_tests_passed_collection_stopped_for_throughput_improvement",
+      "resolution": "Batched transactional writes; preserved partial v1 corpus and started a fresh v2 version. No tokenizer candidates fitted in this attempt."
+    },
+    {
+      "app_id": "ap-GSsPP18jVU2lZFhFnYy8Vw",
+      "outcome": "collection_checkpoint_preserved_then_resumed"
+    },
+    {
+      "app_id": "ap-A86MK9NsJAHwye2lcVuxyv",
+      "outcome": "corpus_frozen_all_18_candidates_completed",
+      "resolution": "Upload checkpoints at one-minute intervals; same sampling code, spec and corpus identity."
+    },
+    {
+      "app_id": "ap-S2OL0auRAcvbx1kCqH7Njd",
+      "outcome": "evaluation_8k_parent_cap_exhausted_libretexts_inventory",
+      "resolution": "Preserved failed evaluation staging; fresh balanced-evaluation-v2 uses a 16 KiB parent/document cap and unchanged 32 MiB source quotas. No tokenizer was retrained."
+    },
+    {
+      "app_id": "ap-F9Q0Rx6GiCDPRcyHzGCaFo",
+      "outcome": "broader_evaluation_frozen",
+      "evaluation_bytes": 33554432,
+      "source_scoped_parents": 4673,
+      "libretexts_parents": 67
+    },
+    {
+      "app_id": "ap-gOCdUYX2WO6VMVzaDuLZVX",
+      "outcome": "all_18_candidates_rescored_on_shared_broader_evaluation",
+      "correctness_passed": 18
+    }
+  ],
+  "authorization": "User explicitly authorized Modal measurements and removed the restored-ledger prerequisite. Resource and time bounds remain enforced.",
+  "remote_test_app_url": "https://modal.com/apps/radiantai/main/ap-CrETuNnj4IkJF8aPh4mqmq",
+  "evaluation_audit": {
+    "original_libretexts_parents": 5,
+    "initial_broader_cap_bytes": 8192,
+    "libretexts_bytes_available_at_8k_cap": 636017,
+    "final_broader_cap_bytes": 16384,
+    "decision_basis": "Parent-coverage audit before interpretation of candidate learning curves."
+  },
+  "fitted_candidates": 18,
+  "rescored_candidates": 18,
+  "evaluation_app_url": "https://modal.com/apps/radiantai/main/ap-gOCdUYX2WO6VMVzaDuLZVX",
+  "sum_fitting_seconds": 927.888957187,
+  "max_peak_process_rss_mib": 3531.80859375
+}

research/modal-smoke.json

@@ -0,0 +1,8 @@
+{
+  "returncode": 0,
+  "stdout": "",
+  "stderr": "test_corruption_and_unrequested_candidate_rejected (test_tokenizer_research.LearningCurveTests.test_corruption_and_unrequested_candidate_rejected) ... ok\ntest_cross_split_duplicate_quarantine (test_tokenizer_research.LearningCurveTests.test_cross_split_duplicate_quarantine) ... ok\ntest_interrupted_uncommitted_batch_replays_from_durable_cursor (test_tokenizer_research.LearningCurveTests.test_interrupted_uncommitted_batch_replays_from_durable_cursor) ... ok\ntest_nested_utf8_samples_and_shared_evaluation (test_tokenizer_research.LearningCurveTests.test_nested_utf8_samples_and_shared_evaluation) ... ok\ntest_resume_and_identity_guard (test_tokenizer_research.LearningCurveTests.test_resume_and_identity_guard) ... ok\n\n----------------------------------------------------------------------\nRan 5 tests in 0.915s\n\nOK\n",
+  "execution": "modal-cpu",
+  "cpu": 2,
+  "memory_mib": 4096
+}

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/1024mib-16384/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/1024mib-16384/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/1024mib-32768/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/1024mib-32768/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/1024mib-50000/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/1024mib-50000/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/128mib-16384/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/128mib-16384/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/128mib-32768/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/128mib-32768/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/128mib-50000/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/128mib-50000/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/2048mib-16384/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/2048mib-16384/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/2048mib-32768/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}