Publish code-focused 32K tokenizer and promote it to the default

.gitattributes

@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 research/corpus-composition.png filter=lfs diff=lfs merge=lfs -text
 research/resources-and-utilization.png filter=lfs diff=lfs merge=lfs -text
 research/source-comparisons.png filter=lfs diff=lfs merge=lfs -text
+research/structured-code-v1/structured-curves.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -6,105 +6,68 @@ tags:
 - tokenizer
 - byte-level-bpe
 - code
-- scientific-text
-- from-scratch
+- tool-calling
 ---
 
-# Scholarly/code tokenizer learning curve
+# Code and tool-calling tokenizer
 
-18 byte-level BPE tokenizers trained from scratch on Modal: 64, 128, 256, 512,
-1,024 and 2,048 MiB of text, each at 16,384, 32,768 and 50,000 vocabulary entries.
-This is a tokenizer resource, with no language-model weights.
-
-The default is **32,768 entries trained on 2 GiB**, a practical starting point
-balancing compression and embedding/output size. This is not a demonstrated
-model-quality optimum. All 18 variants are included for reproducible comparisons.
-The existing scholarly-code-v1 tokenizer in GitHub has not been replaced.
-
-## Usage
+The default is **structured-code-v1: 32,768 entries, fitted on approximately 64 MiB**
+on Modal. It was explicitly selected for code and tool-calling workloads.
+This repository contains tokenizer assets, not language-model weights.
 
 ```python
 from transformers import AutoTokenizer
 
 tokenizer = AutoTokenizer.from_pretrained("jfrick/scholarly-code-tokenizer")
-text = "def square(x):\n    return x ** 2\n"
+text = '{"name":"read_file","arguments":{"path":"src/main.py"}}'
 ids = tokenizer.encode(text, add_special_tokens=False)
 assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text
 
-# Choose any measured vocabulary and fitting size:
-tokenizer_50k = AutoTokenizer.from_pretrained(
-    "jfrick/scholarly-code-tokenizer", subfolder="variants/2048mib-50000"
+# The previous balanced default remains available:
+previous = AutoTokenizer.from_pretrained(
+    "jfrick/scholarly-code-tokenizer", subfolder="variants/2048mib-32768"
 )
 ```
 
-Tested with `transformers==4.57.6` and `tokenizers==0.22.2`. No custom remote code
-is required. The Rust `tokenizers.Tokenizer.from_file("tokenizer.json")` API also
-works. BOS/EOS are not inserted automatically; training pipelines must add document
-boundaries explicitly. Special IDs: PAD=0, BOS=1, EOS=2, UNK=3. Decode with
-`skip_special_tokens=False` when exact round trips including special literals matter.
-Byte coverage avoids unknown tokens for ordinary valid UTF-8 text. No maximum
-model context is implied by this tokenizer.
-
-## Results
+The default is also available in `variants/structured-code-v1-32768`. All 18 original
+learning-curve variants remain available under their existing names. Pin a commit
+revision when training models; changing a tokenizer requires retokenizing data.
+Old tokenized shards and model checkpoints must not be relabeled with the new IDs.
 
-![Learning curve](research/learning-curve.png)
+## Fitting and measurements
 
-All variants were evaluated on the same 32 MiB source-weighted held-out corpus,
-with parent separation and a 16 KiB evaluation parent cap. Higher bytes/token means
-fewer tokens for identical text. The [research report](research/REPORT.md) and
-[PDF](research/research-report.pdf) include per-source curves, paired bootstrap
-intervals, vocabulary utilization, fragmentation probes, fit time and peak memory.
+The focused fitting mixture is 40% real Python excerpts, 20% JSON, 20% Git diffs and
+20% tool-call formats by UTF-8 bytes. JSON, diffs and tool formats are generated
+controlled fixtures derived from Python excerpts. Fitting and evaluation source
+parents are disjoint, but generator templates are shared. They are not production
+traces. Source Python comes from the pinned, license-filtered CodeParrot inventory
+in the original [source recipe](recipe.json); raw source text is not distributed.
+Source datasets retain their respective licenses and provenance.
 
-| Fitting MiB | Vocabulary | Bytes/token | Variant |
-|---:|---:|---:|---|
-| 64 | 16,384 | 3.7463 | [`64mib-16384`](variants/64mib-16384) |
-| 64 | 32,768 | 4.0110 | [`64mib-32768`](variants/64mib-32768) |
-| 64 | 50,000 | 4.1392 | [`64mib-50000`](variants/64mib-50000) |
-| 128 | 16,384 | 3.7571 | [`128mib-16384`](variants/128mib-16384) |
-| 128 | 32,768 | 4.0258 | [`128mib-32768`](variants/128mib-32768) |
-| 128 | 50,000 | 4.1549 | [`128mib-50000`](variants/128mib-50000) |
-| 256 | 16,384 | 3.7624 | [`256mib-16384`](variants/256mib-16384) |
-| 256 | 32,768 | 4.0338 | [`256mib-32768`](variants/256mib-32768) |
-| 256 | 50,000 | 4.1655 | [`256mib-50000`](variants/256mib-50000) |
-| 512 | 16,384 | 3.7646 | [`512mib-16384`](variants/512mib-16384) |
-| 512 | 32,768 | 4.0395 | [`512mib-32768`](variants/512mib-32768) |
-| 512 | 50,000 | 4.1729 | [`512mib-50000`](variants/512mib-50000) |
-| 1024 | 16,384 | 3.7659 | [`1024mib-16384`](variants/1024mib-16384) |
-| 1024 | 32,768 | 4.0431 | [`1024mib-32768`](variants/1024mib-32768) |
-| 1024 | 50,000 | 4.1774 | [`1024mib-50000`](variants/1024mib-50000) |
-| 2048 | 16,384 | 3.7660 | [`2048mib-16384`](variants/2048mib-16384) |
-| 2048 | 32,768 | 4.0438 | [`2048mib-32768`](variants/2048mib-32768) |
-| 2048 | 50,000 | 4.1786 | [`2048mib-50000`](variants/2048mib-50000) |
+Compared with the previous 2 GiB / 32K balanced default, this tokenizer uses 2.81%
+fewer tokens on Python, 8.21% fewer on JSON, 11.83% fewer on Git diffs, and 13.97%
+fewer on tool-format fixtures. It uses **13.99% more tokens on the broader scholarly
+mixture**. This is an explicit specialization trade-off, not a claim of better
+language-model quality or valid tool execution.
 
-## Data and method
+[Structured-output report](research/structured-code-v1/REPORT.md) ·
+[PDF](research/structured-code-v1/research-report.pdf) ·
+[Original learning-curve report](research/REPORT.md)
 
-All 12 pinned source inventories in [recipe.json](recipe.json) were eligible:
-FineWeb-Edu 10%, Python code 30%, FineMath 20%, peS2o 10%, PubMed 3%, arXiv 2%,
-Pressbooks 3%, LibreTexts 3%, DOAB 4%, Gutenberg 5%, pre-1929 books 7%, and
-Library of Congress 3%. Percentages apply to UTF-8 bytes. This uses samples from
-all sources, not every record in those datasets. Source revisions and full shard
-inventories are pinned in the manifests. Raw source text is not distributed here;
-source datasets retain their respective licenses and provenance.
+The reports preserve their original experimental conclusions; this model card
+records the subsequent user decision to promote the focused tokenizer.
 
-Fitting uses nested samples, a seeded shard permutation and sequential records,
-64 KiB per document and 256 KiB per parent. Evaluation initially had sparse book
-coverage and was rebuilt with a 16 KiB parent cap before interpreting curves;
-all unchanged tokenizer weights were rescored. Normalized exact-chunk deduplication
-is applied, including fitting/evaluation overlap checks. Byte-level BPE uses the
-same fixed pretokenization rules, a full byte alphabet, min frequency 2 and maximum
-learned token length 32. No normalization or automatic prefix space is applied.
+## Contract and reproducibility
 
-## Limitations and reproducibility
+Byte-level BPE; full 256-byte alphabet; no normalization, prefix insertion or
+automatic BOS/EOS. PAD=0, BOS=1, EOS=2, UNK=3. Add document boundaries explicitly.
+No model context length is implied. All held-out examples pass exact round-trip
+and zero-UNK checks. JSON validity and tool-call success require model evaluation.
 
-One sampling seed; no global reservoir sampling, fuzzy deduplication guarantee,
-or cross-source parent reconciliation. Bootstrap intervals measure evaluation
-uncertainty, not training-seed variance. Compression is not language-model quality.
-Compare downstream bits per byte under matched parameter and compute budgets
-before selecting a final vocabulary. At width 768, 32K adds 12.58M tied embedding
-parameters over 16K; 50K adds 25.82M. Untied input/output matrices double these deltas.
+Standard loading is verified with `transformers==4.57.6`, `tokenizers==0.22.2`;
+no custom remote code is required. Decode with `skip_special_tokens=False` to
+preserve special literals. The Rust `tokenizers.Tokenizer.from_file` API also works.
 
-[Implementation and PR](https://github.com/koogle/lm-exploration/pull/8).
-[Publication manifest](publication-manifest.json) pins every tokenizer hash.
-The `research/` manifests and measurements record corpus hashes, source coverage,
-correctness, resource measurements and original result hashes. Candidate tokenizers
-and full parent-level results are also preserved on the original Modal volumes.
+[Publication manifest](publication-manifest.json) pins all tokenizer hashes.
+The default variant includes its immutable freeze manifest, fitting corpus metadata
+and evaluation metrics. [Implementation PR](https://github.com/koogle/lm-exploration/pull/8).

local-verification.json

@@ -1,8 +1,8 @@
 {
-  "location": "/workspace/scratch/tokenizer-hf-release",
+  "location": "/workspace/scratch/tokenizer-hf-focused-release",
   "revision": null,
   "status": "passed",
-  "default_and_variants_checked": 19,
+  "default_and_variants_checked": 20,
   "probes_per_variant": 6,
   "checks": [
     "artifact_hashes",

publication-manifest.json

@@ -1,6 +1,6 @@
 {
-  "default_variant": "2048mib-32768",
-  "selection": "32K is a practical starting point balancing compression and vocabulary parameter overhead; no downstream optimum has been established.",
+  "default_variant": "structured-code-v1-32768",
+  "selection": "User-selected code/tool-call-focused 32K tokenizer. Measured scholarly token-count regression accepted; model-quality improvements not established.",
   "variants": {
     "64mib-16384": "97fa4177ad5a038cc18df87a2acb76c3ed3da89a9b7848517442bcbd7029d83d",
     "64mib-32768": "cd9109d658834401ef05b4de231cdfd789385bb0d9f294e343e19b447d94b52a",
@@ -19,7 +19,9 @@
     "1024mib-50000": "ce5e83a52cab742ea5edc2440c8defd26c946d05ca2f1d3f64ab3dedc26b744d",
     "2048mib-16384": "f95ec3899180fd6aef876f92e8fa1e6ec9cd96c180e22b23ead4e88b3b002561",
     "2048mib-32768": "6f8742023692929139e450a3a64802e355649742e7051f9d8a6e6db47fbc8e96",
-    "2048mib-50000": "5e3de8f77cfa5a5820a96bf0205e7cceee7f27c43e16afdc16d452df7a37f724"
+    "2048mib-50000": "5e3de8f77cfa5a5820a96bf0205e7cceee7f27c43e16afdc16d452df7a37f724",
+    "structured-code-v1-32768": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9"
   },
-  "evaluation_manifest_sha256": "e4e66512ca51226f56844d61a4d0c065064900b6a6d0bd0243e9e64b8698024f"
+  "evaluation_manifest_sha256": "e4e66512ca51226f56844d61a4d0c065064900b6a6d0bd0243e9e64b8698024f",
+  "previous_default_variant": "2048mib-32768"
 }

research/structured-code-v1/REPORT.md

@@ -0,0 +1,90 @@
+# JSON, code, Git diff and tool-call tokenizer study
+
+## Overview
+
+All 18 original tokenizers and three fresh code-focused tokenizers were scored on Modal. The new fits use approximately 64 MiB each at 16,384, 32,768 and 50,000 vocabulary entries. Existing tokenizers and the published default are unchanged.
+
+## Method
+
+- 512 held-out Python parent identifiers, one source excerpt per parent, sampled with seed 20261016 from the prior frozen evaluation. Each supplies four benchmark examples (2,048 total).
+- Python uses real code excerpts. JSON, Git diffs and tool calls are generated controlled fixtures derived from those excerpts. They are not a representative collection of production tool traces.
+- JSON includes nested metadata, booleans/null/numbers, compact/pretty formatting, escaped code and Unicode. Tool formats include schemas, argument objects, JSON-encoded arguments, calls and results.
+- Unified patches include insertions, deletions and identifier replacements. Generated fixture patches pass git apply checks in tests; binary patches, renames and multi-file edits are not covered.
+- The focused fitting mixture is 40% Python, 20% JSON, 20% diffs, 20% tool formats by UTF-8 bytes. Whole examples may slightly exceed each quota. Fitting parents are disjoint from evaluation parents; generator templates are shared.
+- Same byte-level BPE, pretokenization, full byte alphabet, four special tokens, min frequency 2 and maximum learned token length 32 as the original experiment. No 120K experiment.
+- Paired 95% intervals use 1,000 bootstrap samples of source parents. They quantify variation within this fixture set, not production generalization or training-seed variance.
+- Actual fitting bytes: 67,116,295; fitting source parents: 14,911.
+- All 21 structured evaluations passed exact round-trip and zero-UNK checks. The three new fits were also evaluated on the original shared 32 MiB scholarly mixture.
+
+## Focused fits versus matched 64 MiB originals
+
+| Vocabulary | Python token savings | JSON | Git diff | Tool call |
+|---:|---:|---:|---:|---:|
+| 16384 | 5.16% | 11.50% | 14.38% | 17.04% |
+| 32768 | 3.72% | 8.84% | 12.52% | 14.70% |
+| 50000 | 2.76% | 7.22% | 7.74% | 13.15% |
+
+Positive savings mean fewer tokens; negative values mean more. Comparisons are within each workload, not an aggregate dominated by long code strings.
+
+![Gains](focused-gains.png)
+
+## All 21 candidates
+
+![Curves](structured-curves.png)
+
+| Candidate | Python bytes/token | JSON | Git diff | Tool call |
+|---|---:|---:|---:|---:|
+| focused-64mib-16384 | 3.8580 | 3.6581 | 3.3957 | 3.6610 |
+| focused-64mib-32768 | 4.0817 | 3.8494 | 3.5404 | 3.8366 |
+| focused-64mib-50000 | 4.1800 | 3.9329 | 3.6137 | 3.9169 |
+| original-64mib-16384 | 3.6588 | 3.2375 | 2.9073 | 3.0371 |
+| original-64mib-32768 | 3.9300 | 3.5093 | 3.0970 | 3.2725 |
+| original-64mib-50000 | 4.0644 | 3.6491 | 3.3339 | 3.4020 |
+| original-128mib-16384 | 3.6701 | 3.2780 | 2.9325 | 3.0461 |
+| original-128mib-32768 | 3.9446 | 3.5153 | 3.1085 | 3.2862 |
+| original-128mib-50000 | 4.0822 | 3.6533 | 3.3384 | 3.3995 |
+| original-256mib-16384 | 3.6757 | 3.2855 | 2.9116 | 3.0472 |
+| original-256mib-32768 | 3.9539 | 3.5274 | 3.1219 | 3.2930 |
+| original-256mib-50000 | 4.0939 | 3.6599 | 3.3433 | 3.4107 |
+| original-512mib-16384 | 3.6794 | 3.2926 | 2.9111 | 3.0234 |
+| original-512mib-32768 | 3.9614 | 3.5260 | 3.2274 | 3.3029 |
+| original-512mib-50000 | 4.1036 | 3.6648 | 3.3536 | 3.4177 |
+| original-1024mib-16384 | 3.6838 | 3.2938 | 2.9131 | 3.0432 |
+| original-1024mib-32768 | 3.9681 | 3.5338 | 3.2407 | 3.3068 |
+| original-1024mib-50000 | 4.1099 | 3.6733 | 3.3565 | 3.4321 |
+| original-2048mib-16384 | 3.6823 | 3.2906 | 2.9141 | 3.0471 |
+| original-2048mib-32768 | 3.9672 | 3.5332 | 3.1214 | 3.3007 |
+| original-2048mib-50000 | 4.1117 | 3.6512 | 3.3561 | 3.4196 |
+
+## General-text regression check
+
+![General regression](general-regression.png)
+
+| Vocabulary | New bytes/token | Extra tokens vs original 64 MiB | Extra tokens vs original 2 GiB |
+|---:|---:|---:|---:|
+| 16384 | 3.2895 | 13.88% | 14.48% |
+| 32768 | 3.5474 | 13.07% | 13.99% |
+| 50000 | 3.6963 | 11.98% | 13.05% |
+
+## Tool-format breakdown: focused 32K vs published 2 GiB / 32K
+
+Long code-bearing tool results dominate aggregate bytes. This separate breakdown checks the smaller calls and schemas as well.
+
+| Format | Examples | Original mean tokens | Focused mean tokens | Token savings |
+|---|---:|---:|---:|---:|
+| read_file_call | 138 | 61.18 | 51.86 | 15.23% |
+| apply_patch_call | 120 | 307.64 | 257.26 | 16.38% |
+| tool_result | 134 | 1748.65 | 1520.01 | 13.08% |
+| schema_and_search_call | 120 | 107.39 | 83.28 | 22.45% |
+
+## Interpretation and limits
+
+At 32K, the focused fit saves 14.70% of tokens on tool fixtures versus the matched 64 MiB original, but costs 13.07% extra tokens on the general mixture. Against the published 2 GiB / 32K default, tool-fixture savings are 13.97% and the general-mixture cost is 13.99%. A structured-output slice in a balanced mixture is a more promising next experiment than wholesale replacement with this code-only mixture. This is a controlled specialization experiment. Better encoding of generated tool formats does not demonstrate valid tool selection, schema adherence, applicable generated patches or functional code. Repeated shared schema/template structure can favor the focused candidates. Confirm findings with independent real tool traces before promoting a tokenizer or changing the model-training mixture.
+
+The original 32K tokenizer remains the default. Treat the new fits as research candidates; their general-text regressions and practical model-level results must inform any replacement.
+
+## Evidence
+
+`measurements.json` includes byte/token counts, mean and p95 tokens per example, exact round-trip/UNK checks and fixed-fragment token counts. `paired-comparisons.json` retains both matched-64-MiB and original-2-GiB contrasts with intervals. Corpus, source files and tokenizers are hashed. Full per-parent measurements and new tokenizer artifacts remain on Modal under `tokenizer-research/structured-code-v1/`. The local report regenerates from `runs/structured-code-v1/scores.json`.
+
+Run: `modal run --detach modal_structured_tokenizer.py`, then `python structured_tokenizer_report.py`. A fresh corpus version is required for a rerun; existing artifact paths refuse overwrite. Resource bounds and actual test outcomes are in `execution.json`.

research/structured-code-v1/corpus-manifest.json

@@ -0,0 +1,37 @@
+{
+  "seed": 20261016,
+  "target_bytes": 67108864,
+  "weights_percent": {
+    "python": 40,
+    "json": 20,
+    "git_diff": 20,
+    "tool_call": 20
+  },
+  "actual_bytes": {
+    "python": 26847131,
+    "json": 13424847,
+    "git_diff": 13421882,
+    "tool_call": 13422435
+  },
+  "training_examples": {
+    "python": 5059,
+    "json": 2290,
+    "git_diff": 33758,
+    "tool_call": 6555
+  },
+  "fitting_parents": 14911,
+  "evaluation_parents_per_category": 512,
+  "train_sha256": "cdc34355495f1c21bba4742ac11e9bcc0a0d0df836ab49eed9a1e01f97ab9a09",
+  "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+  "source_train_sha256": "5e1c04e310375d9922e008142e7d9813f56e541918b729dd7764eaec3ffbadea",
+  "source_validation_sha256": "611010954c9c48ccad9b9a28f7377d56ab180373b5e7eb76fe87ab723b2b8040",
+  "implementation_sha256": "cd1f6e5351aad5acce1a1021af649c4dd8a099c2b0d506e7cc67cd5aee230163",
+  "limitations": [
+    "Python excerpts from the existing pinned codeparrot corpus, not necessarily complete parseable files",
+    "JSON, tool calls and patches are generated controlled fixtures, not independent production traces",
+    "Fitting/evaluation parents are disjoint; generator templates are shared",
+    "Each category quota includes its final whole example, so 64 MiB is approximate",
+    "Unified diffs cover insertions, deletions and identifier replacements, but not binary patches or renames",
+    "One fitting seed; no model-level tool-call or code-generation correctness measured"
+  ]
+}

research/structured-code-v1/execution.json

@@ -0,0 +1,32 @@
+{
+  "execution": "modal-cpu",
+  "remote_tests_passed": 2,
+  "candidates_scored": 21,
+  "focused_fits": 3,
+  "correctness_passed": true,
+  "prepare": {
+    "cpu": 4,
+    "memory_mib": 8192,
+    "timeout_seconds": 1800
+  },
+  "fit": {
+    "cpu": 4,
+    "memory_mib": 8192,
+    "timeout_seconds": 1800,
+    "max_concurrency": 3
+  },
+  "score": {
+    "cpu": 2,
+    "memory_mib": 4096,
+    "timeout_seconds": 1200,
+    "max_concurrency": 3
+  },
+  "resource_settings": "CPU/memory requests and hard caps",
+  "modal_app_url": "https://modal.com/apps/radiantai/main/ap-g5niWKfDfEsYHQ1NEra6Ao",
+  "local_tests_passed": 4,
+  "tokenizers_version": "0.22.2",
+  "generator_implementation_sha256": "cd1f6e5351aad5acce1a1021af649c4dd8a099c2b0d506e7cc67cd5aee230163",
+  "tokenizer_core_sha256": "5d992fec42941a33951b22ffb7468ae58811f6de79364cd9f5bcfd714ee772aa",
+  "tokenizer_config_sha256": "78620acfac5a23364592a2216d4048828b5d416e43baef5762f09930ac3f98f4",
+  "general_evaluation_correctness_passed": true
+}

research/structured-code-v1/focused-fits.json

@@ -0,0 +1,539 @@
+[
+  {
+    "vocab": 16384,
+    "training_seconds": 9.156001359,
+    "tokenizer_sha256": "e63d7d483dde1aa536251a2342d6809ff6d327bf958304ead50fd817e72081d5",
+    "corpus_manifest_sha256": "a4328e99a51bee64daaee66efe25e80ce01a263d568a69a130b94f687edef407",
+    "execution": "modal-cpu",
+    "general_metrics": {
+      "sources": {
+        "fineweb_edu": {
+          "bytes": 3355444,
+          "tokens": 1020901,
+          "chunks": 927,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.286747686602325,
+          "chunk_tokens_p95": 2532
+        },
+        "code_python": {
+          "bytes": 10066330,
+          "tokens": 2609022,
+          "chunks": 2092,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.8582771628602597,
+          "chunk_tokens_p95": 2427
+        },
+        "finemath_4plus": {
+          "bytes": 6710887,
+          "tokens": 2335747,
+          "chunks": 1789,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 2.8731223886833632,
+          "chunk_tokens_p95": 3059
+        },
+        "pes2o": {
+          "bytes": 3355444,
+          "tokens": 1011474,
+          "chunks": 422,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.317380377548014,
+          "chunk_tokens_p95": 2800
+        },
+        "pubmed": {
+          "bytes": 1006633,
+          "tokens": 337939,
+          "chunks": 128,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 2.978741725577693,
+          "chunk_tokens_p95": 3348
+        },
+        "arxiv": {
+          "bytes": 671089,
+          "tokens": 231007,
+          "chunks": 83,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 2.905059154051609,
+          "chunk_tokens_p95": 3729
+        },
+        "pressbooks": {
+          "bytes": 1006633,
+          "tokens": 298450,
+          "chunks": 359,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.3728698274417828,
+          "chunk_tokens_p95": 2266
+        },
+        "libretexts": {
+          "bytes": 1006632,
+          "tokens": 302996,
+          "chunks": 285,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.3222616800221787,
+          "chunk_tokens_p95": 2508
+        },
+        "doab": {
+          "bytes": 1342177,
+          "tokens": 421327,
+          "chunks": 315,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.1855945619435735,
+          "chunk_tokens_p95": 2817
+        },
+        "gutenberg": {
+          "bytes": 1677721,
+          "tokens": 561130,
+          "chunks": 210,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 2.989897171778376,
+          "chunk_tokens_p95": 3154
+        },
+        "pre_1929_books": {
+          "bytes": 2348810,
+          "tokens": 743797,
+          "chunks": 291,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.157864309751182,
+          "chunk_tokens_p95": 2958
+        },
+        "library_of_congress": {
+          "bytes": 1006632,
+          "tokens": 326527,
+          "chunks": 127,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.0828446039684314,
+          "chunk_tokens_p95": 3107
+        }
+      },
+      "bytes": 33554432,
+      "tokens": 10200317,
+      "bytes_per_token": 3.2895479620878447,
+      "evaluation_seconds": 17.576802950999998,
+      "evaluation_mib_per_second": 1.8205813701848108,
+      "vocabulary_used": 15530,
+      "vocabulary_use_fraction": 0.9478759765625,
+      "vocabulary_unseen": 854,
+      "vocabulary_used_at_most_10_times": 1325,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "probes": [
+        {
+          "text": "def square(x):\r\n\treturn x ** 2\r\n",
+          "tokens": 10,
+          "roundtrip": true
+        },
+        {
+          "text": "\u222b\u2080\u00b9 x\u00b2 dx = \u2153",
+          "tokens": 17,
+          "roundtrip": true
+        },
+        {
+          "text": "12345678901234567890",
+          "tokens": 7,
+          "roundtrip": true
+        },
+        {
+          "text": "na\u00efve caf\u00e9 \u4e2d\u6587 \u0627\u0644\u0639\u0631\u0628\u064a\u0629 \ud83d\ude00",
+          "tokens": 31,
+          "roundtrip": true
+        },
+        {
+          "text": "\u0000\t \r\n",
+          "tokens": 3,
+          "roundtrip": true
+        },
+        {
+          "text": "HTTPRequestHandler snake_case_identifier",
+          "tokens": 6,
+          "roundtrip": true
+        },
+        {
+          "text": "e\u0301 \u2260 \u00e9",
+          "tokens": 9,
+          "roundtrip": true
+        }
+      ],
+      "fragmentation": {
+        "electromagnetism": 6,
+        "deoxyribonucleic": 9,
+        "Schr\u00f6dinger": 6,
+        "HTTPRequestHandler": 2,
+        "get_user_configuration": 3,
+        "np.linalg.eigh": 4,
+        "123456789": 3,
+        "0.000001": 4,
+        "\\frac{a}{b}": 7
+      }
+    },
+    "general_evaluation_sha256": "e4e66512ca51226f56844d61a4d0c065064900b6a6d0bd0243e9e64b8698024f"
+  },
+  {
+    "vocab": 32768,
+    "training_seconds": 11.375954247,
+    "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+    "corpus_manifest_sha256": "a4328e99a51bee64daaee66efe25e80ce01a263d568a69a130b94f687edef407",
+    "execution": "modal-cpu",
+    "general_metrics": {
+      "sources": {
+        "fineweb_edu": {
+          "bytes": 3355444,
+          "tokens": 936322,
+          "chunks": 927,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.583643233844767,
+          "chunk_tokens_p95": 2327
+        },
+        "code_python": {
+          "bytes": 10066330,
+          "tokens": 2469754,
+          "chunks": 2092,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 4.075843181142737,
+          "chunk_tokens_p95": 2286
+        },
+        "finemath_4plus": {
+          "bytes": 6710887,
+          "tokens": 2164157,
+          "chunks": 1789,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.10092428599219,
+          "chunk_tokens_p95": 2854
+        },
+        "pes2o": {
+          "bytes": 3355444,
+          "tokens": 923008,
+          "chunks": 422,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.635335771737623,
+          "chunk_tokens_p95": 2595
+        },
+        "pubmed": {
+          "bytes": 1006633,
+          "tokens": 311772,
+          "chunks": 128,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.2287472896860527,
+          "chunk_tokens_p95": 3130
+        },
+        "arxiv": {
+          "bytes": 671089,
+          "tokens": 210824,
+          "chunks": 83,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.1831717451523547,
+          "chunk_tokens_p95": 3373
+        },
+        "pressbooks": {
+          "bytes": 1006633,
+          "tokens": 272469,
+          "chunks": 359,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.694486345235605,
+          "chunk_tokens_p95": 2091
+        },
+        "libretexts": {
+          "bytes": 1006632,
+          "tokens": 278042,
+          "chunks": 285,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.620431445608937,
+          "chunk_tokens_p95": 2344
+        },
+        "doab": {
+          "bytes": 1342177,
+          "tokens": 383761,
+          "chunks": 315,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.49742938964616,
+          "chunk_tokens_p95": 2570
+        },
+        "gutenberg": {
+          "bytes": 1677721,
+          "tokens": 520584,
+          "chunks": 210,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.2227671230771593,
+          "chunk_tokens_p95": 2958
+        },
+        "pre_1929_books": {
+          "bytes": 2348810,
+          "tokens": 686411,
+          "chunks": 291,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.4218711529972565,
+          "chunk_tokens_p95": 2744
+        },
+        "library_of_congress": {
+          "bytes": 1006632,
+          "tokens": 301653,
+          "chunks": 127,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.337052838857893,
+          "chunk_tokens_p95": 2890
+        }
+      },
+      "bytes": 33554432,
+      "tokens": 9458757,
+      "bytes_per_token": 3.5474462447867094,
+      "evaluation_seconds": 22.144963078000004,
+      "evaluation_mib_per_second": 1.4450238588020279,
+      "vocabulary_used": 28593,
+      "vocabulary_use_fraction": 0.872589111328125,
+      "vocabulary_unseen": 4175,
+      "vocabulary_used_at_most_10_times": 6736,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "probes": [
+        {
+          "text": "def square(x):\r\n\treturn x ** 2\r\n",
+          "tokens": 10,
+          "roundtrip": true
+        },
+        {
+          "text": "\u222b\u2080\u00b9 x\u00b2 dx = \u2153",
+          "tokens": 17,
+          "roundtrip": true
+        },
+        {
+          "text": "12345678901234567890",
+          "tokens": 7,
+          "roundtrip": true
+        },
+        {
+          "text": "na\u00efve caf\u00e9 \u4e2d\u6587 \u0627\u0644\u0639\u0631\u0628\u064a\u0629 \ud83d\ude00",
+          "tokens": 30,
+          "roundtrip": true
+        },
+        {
+          "text": "\u0000\t \r\n",
+          "tokens": 3,
+          "roundtrip": true
+        },
+        {
+          "text": "HTTPRequestHandler snake_case_identifier",
+          "tokens": 4,
+          "roundtrip": true
+        },
+        {
+          "text": "e\u0301 \u2260 \u00e9",
+          "tokens": 8,
+          "roundtrip": true
+        }
+      ],
+      "fragmentation": {
+        "electromagnetism": 6,
+        "deoxyribonucleic": 8,
+        "Schr\u00f6dinger": 5,
+        "HTTPRequestHandler": 1,
+        "get_user_configuration": 3,
+        "np.linalg.eigh": 4,
+        "123456789": 3,
+        "0.000001": 4,
+        "\\frac{a}{b}": 7
+      }
+    },
+    "general_evaluation_sha256": "e4e66512ca51226f56844d61a4d0c065064900b6a6d0bd0243e9e64b8698024f"
+  },
+  {
+    "vocab": 50000,
+    "training_seconds": 9.773740467,
+    "tokenizer_sha256": "ce4bf31da579b5be65d25bea6670d071cdbbf2ef95a22ec26807cd8fc997660c",
+    "corpus_manifest_sha256": "a4328e99a51bee64daaee66efe25e80ce01a263d568a69a130b94f687edef407",
+    "execution": "modal-cpu",
+    "general_metrics": {
+      "sources": {
+        "fineweb_edu": {
+          "bytes": 3355444,
+          "tokens": 887035,
+          "chunks": 927,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.782763927015281,
+          "chunk_tokens_p95": 2216
+        },
+        "code_python": {
+          "bytes": 10066330,
+          "tokens": 2411019,
+          "chunks": 2092,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 4.175135077740988,
+          "chunk_tokens_p95": 2224
+        },
+        "finemath_4plus": {
+          "bytes": 6710887,
+          "tokens": 2067410,
+          "chunks": 1789,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.2460358612950504,
+          "chunk_tokens_p95": 2725
+        },
+        "pes2o": {
+          "bytes": 3355444,
+          "tokens": 880780,
+          "chunks": 422,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.8096278298780626,
+          "chunk_tokens_p95": 2488
+        },
+        "pubmed": {
+          "bytes": 1006633,
+          "tokens": 298208,
+          "chunks": 128,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.3756069589011695,
+          "chunk_tokens_p95": 3008
+        },
+        "arxiv": {
+          "bytes": 671089,
+          "tokens": 201611,
+          "chunks": 83,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.32863286229422,
+          "chunk_tokens_p95": 3256
+        },
+        "pressbooks": {
+          "bytes": 1006633,
+          "tokens": 258196,
+          "chunks": 359,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.8987164789539728,
+          "chunk_tokens_p95": 1950
+        },
+        "libretexts": {
+          "bytes": 1006632,
+          "tokens": 265207,
+          "chunks": 285,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.7956464195892266,
+          "chunk_tokens_p95": 2232
+        },
+        "doab": {
+          "bytes": 1342177,
+          "tokens": 364241,
+          "chunks": 315,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.6848597494515993,
+          "chunk_tokens_p95": 2448
+        },
+        "gutenberg": {
+          "bytes": 1677721,
+          "tokens": 497991,
+          "chunks": 210,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.3689785558373546,
+          "chunk_tokens_p95": 2852
+        },
+        "pre_1929_books": {
+          "bytes": 2348810,
+          "tokens": 657482,
+          "chunks": 291,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.5724324011912114,
+          "chunk_tokens_p95": 2649
+        },
+        "library_of_congress": {
+          "bytes": 1006632,
+          "tokens": 288605,
+          "chunks": 127,
+          "roundtrip_failures": 0,
+          "unk_tokens": 0,
+          "bytes_per_token": 3.4879229396580103,
+          "chunk_tokens_p95": 2684
+        }
+      },
+      "bytes": 33554432,
+      "tokens": 9077785,
+      "bytes_per_token": 3.6963237177351083,
+      "evaluation_seconds": 21.129434848000002,
+      "evaluation_mib_per_second": 1.5144749601776002,
+      "vocabulary_used": 39147,
+      "vocabulary_use_fraction": 0.78294,
+      "vocabulary_unseen": 10853,
+      "vocabulary_used_at_most_10_times": 13602,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "probes": [
+        {
+          "text": "def square(x):\r\n\treturn x ** 2\r\n",
+          "tokens": 10,
+          "roundtrip": true
+        },
+        {
+          "text": "\u222b\u2080\u00b9 x\u00b2 dx = \u2153",
+          "tokens": 17,
+          "roundtrip": true
+        },
+        {
+          "text": "12345678901234567890",
+          "tokens": 7,
+          "roundtrip": true
+        },
+        {
+          "text": "na\u00efve caf\u00e9 \u4e2d\u6587 \u0627\u0644\u0639\u0631\u0628\u064a\u0629 \ud83d\ude00",
+          "tokens": 30,
+          "roundtrip": true
+        },
+        {
+          "text": "\u0000\t \r\n",
+          "tokens": 3,
+          "roundtrip": true
+        },
+        {
+          "text": "HTTPRequestHandler snake_case_identifier",
+          "tokens": 4,
+          "roundtrip": true
+        },
+        {
+          "text": "e\u0301 \u2260 \u00e9",
+          "tokens": 8,
+          "roundtrip": true
+        }
+      ],
+      "fragmentation": {
+        "electromagnetism": 6,
+        "deoxyribonucleic": 8,
+        "Schr\u00f6dinger": 5,
+        "HTTPRequestHandler": 1,
+        "get_user_configuration": 3,
+        "np.linalg.eigh": 4,
+        "123456789": 3,
+        "0.000001": 4,
+        "\\frac{a}{b}": 7
+      }
+    },
+    "general_evaluation_sha256": "e4e66512ca51226f56844d61a4d0c065064900b6a6d0bd0243e9e64b8698024f"
+  }
+]

research/structured-code-v1/general-regression.json

@@ -0,0 +1,38 @@
+[
+  {
+    "vocab": 16384,
+    "baseline_size_mib": 64,
+    "token_change_percent": 13.8842083685768,
+    "bytes_per_token": 3.2895479620878447
+  },
+  {
+    "vocab": 16384,
+    "baseline_size_mib": 2048,
+    "token_change_percent": 14.48309849875795,
+    "bytes_per_token": 3.2895479620878447
+  },
+  {
+    "vocab": 32768,
+    "baseline_size_mib": 64,
+    "token_change_percent": 13.066679257845637,
+    "bytes_per_token": 3.5474462447867094
+  },
+  {
+    "vocab": 32768,
+    "baseline_size_mib": 2048,
+    "token_change_percent": 13.9913345559888,
+    "bytes_per_token": 3.5474462447867094
+  },
+  {
+    "vocab": 50000,
+    "baseline_size_mib": 64,
+    "token_change_percent": 11.982248703193132,
+    "bytes_per_token": 3.6963237177351083
+  },
+  {
+    "vocab": 50000,
+    "baseline_size_mib": 2048,
+    "token_change_percent": 13.048704871757423,
+    "bytes_per_token": 3.6963237177351083
+  }
+]

research/structured-code-v1/measurements.json

@@ -0,0 +1,1241 @@
+[
+  {
+    "name": "focused-64mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "e63d7d483dde1aa536251a2342d6809ff6d327bf958304ead50fd817e72081d5",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 651392,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.8579595696600513,
+        "mean_tokens_per_example": 1272.25,
+        "p95_tokens_per_example": 2339
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 760588,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6580934750482523,
+        "mean_tokens_per_example": 1485.5234375,
+        "p95_tokens_per_example": 2608
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 55115,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3956636124467026,
+        "mean_tokens_per_example": 107.646484375,
+        "p95_tokens_per_example": 255
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 263776,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6609926604391605,
+        "mean_tokens_per_example": 515.1875,
+        "p95_tokens_per_example": 2443
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 14,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 26,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 4,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "focused",
+    "size_mib": 64
+  },
+  {
+    "name": "focused-64mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 615688,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.08168422967477,
+        "mean_tokens_per_example": 1202.515625,
+        "p95_tokens_per_example": 2184
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 722785,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.849418568453966,
+        "mean_tokens_per_example": 1411.689453125,
+        "p95_tokens_per_example": 2514
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 52862,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.5403881805455715,
+        "mean_tokens_per_example": 103.24609375,
+        "p95_tokens_per_example": 249
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 251703,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.8365931276146887,
+        "mean_tokens_per_example": 491.607421875,
+        "p95_tokens_per_example": 2304
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 14,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 25,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 4,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "focused",
+    "size_mib": 64
+  },
+  {
+    "name": "focused-64mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "ce4bf31da579b5be65d25bea6670d071cdbbf2ef95a22ec26807cd8fc997660c",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 601208,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.179990951550877,
+        "mean_tokens_per_example": 1174.234375,
+        "p95_tokens_per_example": 2144
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 707451,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.932854713612674,
+        "mean_tokens_per_example": 1381.740234375,
+        "p95_tokens_per_example": 2419
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 51789,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.613740369576551,
+        "mean_tokens_per_example": 101.150390625,
+        "p95_tokens_per_example": 237
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 246543,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.916890765505409,
+        "mean_tokens_per_example": 481.529296875,
+        "p95_tokens_per_example": 2267
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 14,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 25,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 4,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "focused",
+    "size_mib": 64
+  },
+  {
+    "name": "original-64mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "97fa4177ad5a038cc18df87a2acb76c3ed3da89a9b7848517442bcbd7029d83d",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 686845,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6588225873377547,
+        "mean_tokens_per_example": 1341.494140625,
+        "p95_tokens_per_example": 2464
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 859400,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2374936001861765,
+        "mean_tokens_per_example": 1678.515625,
+        "p95_tokens_per_example": 3008
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 64373,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 2.907305858046075,
+        "mean_tokens_per_example": 125.728515625,
+        "p95_tokens_per_example": 287
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 317957,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.037146532392745,
+        "mean_tokens_per_example": 621.009765625,
+        "p95_tokens_per_example": 2919
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 18,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 30,
+      "get_user_configuration": 4,
+      "\"additionalProperties\":false": 7,
+      "\"tool_call_id\":\"call_0123456789\"": 14
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 64
+  },
+  {
+    "name": "original-64mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "cd9109d658834401ef05b4de231cdfd789385bb0d9f294e343e19b447d94b52a",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 639444,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.930045477008151,
+        "mean_tokens_per_example": 1248.9140625,
+        "p95_tokens_per_example": 2281
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 792841,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.5092811799591597,
+        "mean_tokens_per_example": 1548.517578125,
+        "p95_tokens_per_example": 2779
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 60430,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.0970047989409233,
+        "mean_tokens_per_example": 118.02734375,
+        "p95_tokens_per_example": 267
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 295090,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.272499915280084,
+        "mean_tokens_per_example": 576.34765625,
+        "p95_tokens_per_example": 2703
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 18,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 6,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 64
+  },
+  {
+    "name": "original-64mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "86de4d58e6eb5417264afe227857e98c0b0431eb7adbfaf6a9ebacac37d77726",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 618301,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.064434636204696,
+        "mean_tokens_per_example": 1207.619140625,
+        "p95_tokens_per_example": 2199
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 762463,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.649097726709362,
+        "mean_tokens_per_example": 1489.185546875,
+        "p95_tokens_per_example": 2637
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 56136,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3339033775117572,
+        "mean_tokens_per_example": 109.640625,
+        "p95_tokens_per_example": 254
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 283856,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.4020136970858466,
+        "mean_tokens_per_example": 554.40625,
+        "p95_tokens_per_example": 2588
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 16,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 26,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 64
+  },
+  {
+    "name": "original-128mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "64356327337221b75b445cc7c761d655a40c1af2bccd8f8ab5453fd1fa1035c0",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 684734,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6701025507715404,
+        "mean_tokens_per_example": 1337.37109375,
+        "p95_tokens_per_example": 2404
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 848790,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.277962746969215,
+        "mean_tokens_per_example": 1657.79296875,
+        "p95_tokens_per_example": 2970
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 63820,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 2.9324976496396116,
+        "mean_tokens_per_example": 124.6484375,
+        "p95_tokens_per_example": 287
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 317026,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.0460656223779754,
+        "mean_tokens_per_example": 619.19140625,
+        "p95_tokens_per_example": 2882
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 18,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 4,
+      "\"additionalProperties\":false": 7,
+      "\"tool_call_id\":\"call_0123456789\"": 14
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 128
+  },
+  {
+    "name": "original-128mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "ade4e945319da6e9b010ad7f48f19d9489f7ac583448318a91cf302c21aaf321",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 637085,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.9445976596529504,
+        "mean_tokens_per_example": 1244.306640625,
+        "p95_tokens_per_example": 2256
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 791494,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.515253432116984,
+        "mean_tokens_per_example": 1545.88671875,
+        "p95_tokens_per_example": 2750
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 60206,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.1085273892967478,
+        "mean_tokens_per_example": 117.58984375,
+        "p95_tokens_per_example": 264
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 293856,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2862422410976806,
+        "mean_tokens_per_example": 573.9375,
+        "p95_tokens_per_example": 2688
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 6,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 128
+  },
+  {
+    "name": "original-128mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "db13f817c797466237e07a17ca3d62b65dafe334745d6c72714073fa47d36380",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 615615,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.082168238265799,
+        "mean_tokens_per_example": 1202.373046875,
+        "p95_tokens_per_example": 2185
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 761584,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.65330941826509,
+        "mean_tokens_per_example": 1487.46875,
+        "p95_tokens_per_example": 2625
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 56060,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3384231180877633,
+        "mean_tokens_per_example": 109.4921875,
+        "p95_tokens_per_example": 255
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 284063,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.399534610280114,
+        "mean_tokens_per_example": 554.810546875,
+        "p95_tokens_per_example": 2613
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 28,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 128
+  },
+  {
+    "name": "original-256mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "e76d8e8a71db29a3e1ed036d8e6813ce180ac568b0310d0cdc01a2c8974c92e6",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 683686,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.675728331427,
+        "mean_tokens_per_example": 1335.32421875,
+        "p95_tokens_per_example": 2431
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 846841,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2855069605746534,
+        "mean_tokens_per_example": 1653.986328125,
+        "p95_tokens_per_example": 2972
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 64278,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 2.9116027256604124,
+        "mean_tokens_per_example": 125.54296875,
+        "p95_tokens_per_example": 289
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 316910,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.047180587548515,
+        "mean_tokens_per_example": 618.96484375,
+        "p95_tokens_per_example": 2874
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 18,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 30,
+      "get_user_configuration": 4,
+      "\"additionalProperties\":false": 7,
+      "\"tool_call_id\":\"call_0123456789\"": 14
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 256
+  },
+  {
+    "name": "original-256mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "3839415fcdd736bb9322b958a45ba82ba01310e2007a1b139f77e3486568c1b6",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 635593,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.9538572640038514,
+        "mean_tokens_per_example": 1241.392578125,
+        "p95_tokens_per_example": 2248
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 788761,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.527433531830301,
+        "mean_tokens_per_example": 1540.548828125,
+        "p95_tokens_per_example": 2744
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 59948,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.1219056515646892,
+        "mean_tokens_per_example": 117.0859375,
+        "p95_tokens_per_example": 264
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 293252,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2930107893552303,
+        "mean_tokens_per_example": 572.7578125,
+        "p95_tokens_per_example": 2703
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 6,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 256
+  },
+  {
+    "name": "original-256mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "eb4b69846169b0aedd86c72eb2bc7e85b96b074519f7bba8563423cd200baf81",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 613844,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.093945693042532,
+        "mean_tokens_per_example": 1198.9140625,
+        "p95_tokens_per_example": 2162
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 760204,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.659941278919869,
+        "mean_tokens_per_example": 1484.7734375,
+        "p95_tokens_per_example": 2628
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 55978,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.343313444567509,
+        "mean_tokens_per_example": 109.33203125,
+        "p95_tokens_per_example": 254
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 283137,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.410652793523983,
+        "mean_tokens_per_example": 553.001953125,
+        "p95_tokens_per_example": 2581
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 28,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 256
+  },
+  {
+    "name": "original-512mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "c3bae921b15a294391e45fe3bf2ce0a2f3ac1b6678c0f32541613106f322078e",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 682997,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6794363664847722,
+        "mean_tokens_per_example": 1333.978515625,
+        "p95_tokens_per_example": 2411
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 845018,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2925949506401047,
+        "mean_tokens_per_example": 1650.42578125,
+        "p95_tokens_per_example": 2971
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 64290,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 2.911059262715819,
+        "mean_tokens_per_example": 125.56640625,
+        "p95_tokens_per_example": 290
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 319399,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.0234346381798316,
+        "mean_tokens_per_example": 623.826171875,
+        "p95_tokens_per_example": 2883
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 20,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 30,
+      "get_user_configuration": 4,
+      "\"additionalProperties\":false": 7,
+      "\"tool_call_id\":\"call_0123456789\"": 15
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 512
+  },
+  {
+    "name": "original-512mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "ee276c0fba28e5d4f555f85971e9d99b5cee6b53aa5181bc1c4dd33417572829",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 634379,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.9614236915156398,
+        "mean_tokens_per_example": 1239.021484375,
+        "p95_tokens_per_example": 2267
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 789081,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.5260030339090664,
+        "mean_tokens_per_example": 1541.173828125,
+        "p95_tokens_per_example": 2729
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 57989,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2273707082377694,
+        "mean_tokens_per_example": 113.259765625,
+        "p95_tokens_per_example": 264
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 292371,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3029336014857833,
+        "mean_tokens_per_example": 571.037109375,
+        "p95_tokens_per_example": 2685
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 6,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 512
+  },
+  {
+    "name": "original-512mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "1f4b70d34b608d97a3ad7a9bcd7b0094fa82c652281d39bcc12325c1f16865e2",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 612400,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.103598954931417,
+        "mean_tokens_per_example": 1196.09375,
+        "p95_tokens_per_example": 2147
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 759199,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6647861759565017,
+        "mean_tokens_per_example": 1482.810546875,
+        "p95_tokens_per_example": 2621
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 55806,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3536178905494034,
+        "mean_tokens_per_example": 108.99609375,
+        "p95_tokens_per_example": 254
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 282554,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.417690069862752,
+        "mean_tokens_per_example": 551.86328125,
+        "p95_tokens_per_example": 2588
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 28,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 512
+  },
+  {
+    "name": "original-1024mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "020928562c47ea392514da50b9c4886b4e3e24fd01cfdc30501599042ffb1359",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 682180,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.6838429739951333,
+        "mean_tokens_per_example": 1332.3828125,
+        "p95_tokens_per_example": 2410
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 844701,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2938305980459357,
+        "mean_tokens_per_example": 1649.806640625,
+        "p95_tokens_per_example": 2974
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 64244,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 2.913143639872984,
+        "mean_tokens_per_example": 125.4765625,
+        "p95_tokens_per_example": 289
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 317323,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.0432146424936106,
+        "mean_tokens_per_example": 619.771484375,
+        "p95_tokens_per_example": 2882
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 18,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 30,
+      "get_user_configuration": 4,
+      "\"additionalProperties\":false": 7,
+      "\"tool_call_id\":\"call_0123456789\"": 14
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 1024
+  },
+  {
+    "name": "original-1024mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "17ce952fb54679634b5a723756a217bb536ad1d2f6616917ac0172b7a6ef255c",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 633318,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.9680602793541317,
+        "mean_tokens_per_example": 1236.94921875,
+        "p95_tokens_per_example": 2219
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 787329,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.5338492548858227,
+        "mean_tokens_per_example": 1537.751953125,
+        "p95_tokens_per_example": 2739
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 57750,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.240727272727273,
+        "mean_tokens_per_example": 112.79296875,
+        "p95_tokens_per_example": 260
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 292030,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3067903982467555,
+        "mean_tokens_per_example": 570.37109375,
+        "p95_tokens_per_example": 2677
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 6,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 1024
+  },
+  {
+    "name": "original-1024mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "ce5e83a52cab742ea5edc2440c8defd26c946d05ca2f1d3f64ab3dedc26b744d",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 611455,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.109941042268033,
+        "mean_tokens_per_example": 1194.248046875,
+        "p95_tokens_per_example": 2151
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 757438,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.673306594071066,
+        "mean_tokens_per_example": 1479.37109375,
+        "p95_tokens_per_example": 2618
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 55758,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3565048961583988,
+        "mean_tokens_per_example": 108.90234375,
+        "p95_tokens_per_example": 248
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 281364,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.4321448372926175,
+        "mean_tokens_per_example": 549.5390625,
+        "p95_tokens_per_example": 2566
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 28,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 1024
+  },
+  {
+    "name": "original-2048mib-16384",
+    "vocab": 16384,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "f95ec3899180fd6aef876f92e8fa1e6ec9cd96c180e22b23ead4e88b3b002561",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 682473,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.682261422796213,
+        "mean_tokens_per_example": 1332.955078125,
+        "p95_tokens_per_example": 2401
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 845529,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.2906050531678983,
+        "mean_tokens_per_example": 1651.423828125,
+        "p95_tokens_per_example": 2970
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 64223,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 2.9140961960668297,
+        "mean_tokens_per_example": 125.435546875,
+        "p95_tokens_per_example": 289
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 316917,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.0471132820265243,
+        "mean_tokens_per_example": 618.978515625,
+        "p95_tokens_per_example": 2881
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 18,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 30,
+      "get_user_configuration": 4,
+      "\"additionalProperties\":false": 7,
+      "\"tool_call_id\":\"call_0123456789\"": 14
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 2048
+  },
+  {
+    "name": "original-2048mib-32768",
+    "vocab": 32768,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "6f8742023692929139e450a3a64802e355649742e7051f9d8a6e6db47fbc8e96",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 633463,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.967151988356068,
+        "mean_tokens_per_example": 1237.232421875,
+        "p95_tokens_per_example": 2278
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 787465,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.533238937603576,
+        "mean_tokens_per_example": 1538.017578125,
+        "p95_tokens_per_example": 2730
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 59957,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.121437029871408,
+        "mean_tokens_per_example": 117.103515625,
+        "p95_tokens_per_example": 268
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 292566,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3007321424909253,
+        "mean_tokens_per_example": 571.41796875,
+        "p95_tokens_per_example": 2676
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 29,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 2048
+  },
+  {
+    "name": "original-2048mib-50000",
+    "vocab": 50000,
+    "execution": "modal-cpu",
+    "tokenizer_sha256": "5e3de8f77cfa5a5820a96bf0205e7cceee7f27c43e16afdc16d452df7a37f724",
+    "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+    "metrics": {
+      "python": {
+        "bytes": 2513044,
+        "tokens": 611191,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 4.111716304723074,
+        "mean_tokens_per_example": 1193.732421875,
+        "p95_tokens_per_example": 2152
+      },
+      "json": {
+        "bytes": 2782302,
+        "tokens": 762018,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.651228711132808,
+        "mean_tokens_per_example": 1488.31640625,
+        "p95_tokens_per_example": 2631
+      },
+      "git_diff": {
+        "bytes": 187152,
+        "tokens": 55765,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.3560835649601004,
+        "mean_tokens_per_example": 108.916015625,
+        "p95_tokens_per_example": 248
+      },
+      "tool_call": {
+        "bytes": 965682,
+        "tokens": 282399,
+        "examples": 512,
+        "roundtrip_failures": 0,
+        "unk_tokens": 0,
+        "bytes_per_token": 3.4195659333071293,
+        "mean_tokens_per_example": 551.560546875,
+        "p95_tokens_per_example": 2587
+      }
+    },
+    "fragmentation": {
+      "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 17,
+      "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 28,
+      "get_user_configuration": 3,
+      "\"additionalProperties\":false": 5,
+      "\"tool_call_id\":\"call_0123456789\"": 12
+    },
+    "correctness_passed": true,
+    "family": "original",
+    "size_mib": 2048
+  }
+]

research/structured-code-v1/paired-comparisons.json

@@ -0,0 +1,242 @@
+[
+  {
+    "vocab": 16384,
+    "category": "python",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 5.161717709235713,
+    "paired_ci95": [
+      4.805887801278443,
+      5.518307765192585
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "python",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 4.554172839072024,
+    "paired_ci95": [
+      4.2022398257968305,
+      4.8870282779627345
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "json",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 11.49778915522458,
+    "paired_ci95": [
+      11.051183144800918,
+      11.937857395041755
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "json",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 10.045900258891184,
+    "paired_ci95": [
+      9.62401271480091,
+      10.448147880543663
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "git_diff",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 14.381806036692401,
+    "paired_ci95": [
+      13.52058267964924,
+      15.107384868191106
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "git_diff",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 14.181835168086199,
+    "paired_ci95": [
+      13.483675527600566,
+      14.828775308338756
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "tool_call",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 17.040354513346145,
+    "paired_ci95": [
+      16.288536671705394,
+      17.857604436611496
+    ]
+  },
+  {
+    "vocab": 16384,
+    "category": "tool_call",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 16.768112786628674,
+    "paired_ci95": [
+      16.00072559221649,
+      17.65307903734406
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "python",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 3.715102495292788,
+    "paired_ci95": [
+      3.416089462295272,
+      3.98995603061774
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "python",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 2.8060044548773977,
+    "paired_ci95": [
+      2.5062225893620456,
+      3.0750785140490904
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "json",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 8.836071797497858,
+    "paired_ci95": [
+      8.489760736047739,
+      9.168205525309618
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "json",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 8.21369838659496,
+    "paired_ci95": [
+      7.84936586072181,
+      8.534293914918637
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "git_diff",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 12.523581002813177,
+    "paired_ci95": [
+      11.74804909818136,
+      13.237170838119336
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "git_diff",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 11.833480661140483,
+    "paired_ci95": [
+      11.044209250897488,
+      12.573471647810612
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "tool_call",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 14.702971974651803,
+    "paired_ci95": [
+      13.87174606871349,
+      15.585460177596222
+    ]
+  },
+  {
+    "vocab": 32768,
+    "category": "tool_call",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 13.967104858390922,
+    "paired_ci95": [
+      13.094244188134251,
+      14.86115232472458
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "python",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 2.76451113616184,
+    "paired_ci95": [
+      2.507650374711118,
+      3.0204865576016524
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "python",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 1.6333682923995951,
+    "paired_ci95": [
+      1.3690041312459533,
+      1.8792020981977022
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "json",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 7.215038631382765,
+    "paired_ci95": [
+      6.885572584118245,
+      7.538642446610321
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "json",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 7.160854468004696,
+    "paired_ci95": [
+      6.8230077267262095,
+      7.490274125295937
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "git_diff",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 7.743693886276182,
+    "paired_ci95": [
+      7.125926910643197,
+      8.312421798378304
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "git_diff",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 7.129920200842821,
+    "paired_ci95": [
+      6.607120345561133,
+      7.644764143960624
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "tool_call",
+    "baseline_size_mib": 64,
+    "token_saving_percent": 13.145045375119778,
+    "paired_ci95": [
+      12.422289981936078,
+      13.92135869878892
+    ]
+  },
+  {
+    "vocab": 50000,
+    "category": "tool_call",
+    "baseline_size_mib": 2048,
+    "token_saving_percent": 12.696928813487297,
+    "paired_ci95": [
+      11.968021889121987,
+      13.519645028868915
+    ]
+  }
+]

research/structured-code-v1/tool-subtypes.json

@@ -0,0 +1,674 @@
+[
+  {
+    "name": "original-64mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8595,
+    "mean_tokens_per_example": 62.28260869565217
+  },
+  {
+    "name": "original-64mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 37223,
+    "mean_tokens_per_example": 310.19166666666666
+  },
+  {
+    "name": "original-64mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 235899,
+    "mean_tokens_per_example": 1760.4402985074628
+  },
+  {
+    "name": "original-64mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 13373,
+    "mean_tokens_per_example": 111.44166666666666
+  },
+  {
+    "name": "original-64mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 9395,
+    "mean_tokens_per_example": 68.07971014492753
+  },
+  {
+    "name": "original-64mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 39947,
+    "mean_tokens_per_example": 332.89166666666665
+  },
+  {
+    "name": "original-64mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 254590,
+    "mean_tokens_per_example": 1899.9253731343283
+  },
+  {
+    "name": "original-64mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 14025,
+    "mean_tokens_per_example": 116.875
+  },
+  {
+    "name": "original-64mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8369,
+    "mean_tokens_per_example": 60.64492753623188
+  },
+  {
+    "name": "original-64mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 35332,
+    "mean_tokens_per_example": 294.43333333333334
+  },
+  {
+    "name": "original-64mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 227370,
+    "mean_tokens_per_example": 1696.7910447761194
+  },
+  {
+    "name": "original-64mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12785,
+    "mean_tokens_per_example": 106.54166666666667
+  },
+  {
+    "name": "original-128mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 9384,
+    "mean_tokens_per_example": 68.0
+  },
+  {
+    "name": "original-128mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 40035,
+    "mean_tokens_per_example": 333.625
+  },
+  {
+    "name": "original-128mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 253705,
+    "mean_tokens_per_example": 1893.3208955223881
+  },
+  {
+    "name": "original-128mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 13902,
+    "mean_tokens_per_example": 115.85
+  },
+  {
+    "name": "original-128mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8461,
+    "mean_tokens_per_example": 61.31159420289855
+  },
+  {
+    "name": "original-128mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 37017,
+    "mean_tokens_per_example": 308.475
+  },
+  {
+    "name": "original-128mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 235371,
+    "mean_tokens_per_example": 1756.5
+  },
+  {
+    "name": "original-128mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 13007,
+    "mean_tokens_per_example": 108.39166666666667
+  },
+  {
+    "name": "original-128mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8365,
+    "mean_tokens_per_example": 60.61594202898551
+  },
+  {
+    "name": "original-128mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 35323,
+    "mean_tokens_per_example": 294.35833333333335
+  },
+  {
+    "name": "original-128mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 227570,
+    "mean_tokens_per_example": 1698.2835820895523
+  },
+  {
+    "name": "original-128mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12805,
+    "mean_tokens_per_example": 106.70833333333333
+  },
+  {
+    "name": "original-256mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 9380,
+    "mean_tokens_per_example": 67.97101449275362
+  },
+  {
+    "name": "original-256mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 40021,
+    "mean_tokens_per_example": 333.5083333333333
+  },
+  {
+    "name": "original-256mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 253622,
+    "mean_tokens_per_example": 1892.7014925373135
+  },
+  {
+    "name": "original-256mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 13887,
+    "mean_tokens_per_example": 115.725
+  },
+  {
+    "name": "original-256mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8455,
+    "mean_tokens_per_example": 61.268115942028984
+  },
+  {
+    "name": "original-256mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 36954,
+    "mean_tokens_per_example": 307.95
+  },
+  {
+    "name": "original-256mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 234845,
+    "mean_tokens_per_example": 1752.5746268656717
+  },
+  {
+    "name": "original-256mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12998,
+    "mean_tokens_per_example": 108.31666666666666
+  },
+  {
+    "name": "original-512mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 9937,
+    "mean_tokens_per_example": 72.0072463768116
+  },
+  {
+    "name": "original-512mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 40599,
+    "mean_tokens_per_example": 338.325
+  },
+  {
+    "name": "original-512mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 253895,
+    "mean_tokens_per_example": 1894.7388059701493
+  },
+  {
+    "name": "original-512mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 14968,
+    "mean_tokens_per_example": 124.73333333333333
+  },
+  {
+    "name": "original-256mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8366,
+    "mean_tokens_per_example": 60.6231884057971
+  },
+  {
+    "name": "original-256mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 35267,
+    "mean_tokens_per_example": 293.89166666666665
+  },
+  {
+    "name": "original-256mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 226705,
+    "mean_tokens_per_example": 1691.8283582089553
+  },
+  {
+    "name": "original-256mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12799,
+    "mean_tokens_per_example": 106.65833333333333
+  },
+  {
+    "name": "original-512mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8446,
+    "mean_tokens_per_example": 61.20289855072464
+  },
+  {
+    "name": "original-512mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 36492,
+    "mean_tokens_per_example": 304.1
+  },
+  {
+    "name": "original-512mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 234429,
+    "mean_tokens_per_example": 1749.4701492537313
+  },
+  {
+    "name": "original-512mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 13004,
+    "mean_tokens_per_example": 108.36666666666666
+  },
+  {
+    "name": "original-512mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8360,
+    "mean_tokens_per_example": 60.57971014492754
+  },
+  {
+    "name": "original-512mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 35096,
+    "mean_tokens_per_example": 292.46666666666664
+  },
+  {
+    "name": "original-512mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 226557,
+    "mean_tokens_per_example": 1690.7238805970148
+  },
+  {
+    "name": "original-512mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12541,
+    "mean_tokens_per_example": 104.50833333333334
+  },
+  {
+    "name": "original-1024mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 9385,
+    "mean_tokens_per_example": 68.0072463768116
+  },
+  {
+    "name": "original-1024mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 40140,
+    "mean_tokens_per_example": 334.5
+  },
+  {
+    "name": "original-1024mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 253793,
+    "mean_tokens_per_example": 1893.9776119402984
+  },
+  {
+    "name": "original-1024mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 14005,
+    "mean_tokens_per_example": 116.70833333333333
+  },
+  {
+    "name": "original-1024mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8431,
+    "mean_tokens_per_example": 61.094202898550726
+  },
+  {
+    "name": "original-1024mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 36385,
+    "mean_tokens_per_example": 303.2083333333333
+  },
+  {
+    "name": "original-1024mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 234232,
+    "mean_tokens_per_example": 1748.0
+  },
+  {
+    "name": "original-1024mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12982,
+    "mean_tokens_per_example": 108.18333333333334
+  },
+  {
+    "name": "original-1024mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8358,
+    "mean_tokens_per_example": 60.56521739130435
+  },
+  {
+    "name": "original-1024mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 35108,
+    "mean_tokens_per_example": 292.56666666666666
+  },
+  {
+    "name": "original-1024mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 225118,
+    "mean_tokens_per_example": 1679.9850746268658
+  },
+  {
+    "name": "original-1024mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12780,
+    "mean_tokens_per_example": 106.5
+  },
+  {
+    "name": "original-2048mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 9380,
+    "mean_tokens_per_example": 67.97101449275362
+  },
+  {
+    "name": "original-2048mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 40089,
+    "mean_tokens_per_example": 334.075
+  },
+  {
+    "name": "original-2048mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 253445,
+    "mean_tokens_per_example": 1891.3805970149253
+  },
+  {
+    "name": "original-2048mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 14003,
+    "mean_tokens_per_example": 116.69166666666666
+  },
+  {
+    "name": "original-2048mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8361,
+    "mean_tokens_per_example": 60.58695652173913
+  },
+  {
+    "name": "original-2048mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 35220,
+    "mean_tokens_per_example": 293.5
+  },
+  {
+    "name": "original-2048mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 226049,
+    "mean_tokens_per_example": 1686.9328358208954
+  },
+  {
+    "name": "original-2048mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12769,
+    "mean_tokens_per_example": 106.40833333333333
+  },
+  {
+    "name": "original-2048mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 8443,
+    "mean_tokens_per_example": 61.18115942028985
+  },
+  {
+    "name": "original-2048mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 36917,
+    "mean_tokens_per_example": 307.64166666666665
+  },
+  {
+    "name": "original-2048mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 234319,
+    "mean_tokens_per_example": 1748.6492537313434
+  },
+  {
+    "name": "original-2048mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 12887,
+    "mean_tokens_per_example": 107.39166666666667
+  },
+  {
+    "name": "focused-64mib-16384",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 7389,
+    "mean_tokens_per_example": 53.54347826086956
+  },
+  {
+    "name": "focused-64mib-16384",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 32032,
+    "mean_tokens_per_example": 266.93333333333334
+  },
+  {
+    "name": "focused-64mib-16384",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 214219,
+    "mean_tokens_per_example": 1598.6492537313434
+  },
+  {
+    "name": "focused-64mib-16384",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 10136,
+    "mean_tokens_per_example": 84.46666666666667
+  },
+  {
+    "name": "focused-64mib-32768",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 7157,
+    "mean_tokens_per_example": 51.86231884057971
+  },
+  {
+    "name": "focused-64mib-32768",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 30871,
+    "mean_tokens_per_example": 257.2583333333333
+  },
+  {
+    "name": "focused-64mib-32768",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 203681,
+    "mean_tokens_per_example": 1520.0074626865671
+  },
+  {
+    "name": "focused-64mib-32768",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 9994,
+    "mean_tokens_per_example": 83.28333333333333
+  },
+  {
+    "name": "focused-64mib-50000",
+    "kind": "read_file_call",
+    "examples": 138,
+    "bytes": 23364,
+    "tokens": 7111,
+    "mean_tokens_per_example": 51.528985507246375
+  },
+  {
+    "name": "focused-64mib-50000",
+    "kind": "apply_patch_call",
+    "examples": 120,
+    "bytes": 113181,
+    "tokens": 30310,
+    "mean_tokens_per_example": 252.58333333333334
+  },
+  {
+    "name": "focused-64mib-50000",
+    "kind": "tool_result",
+    "examples": 134,
+    "bytes": 780904,
+    "tokens": 199176,
+    "mean_tokens_per_example": 1486.3880597014925
+  },
+  {
+    "name": "focused-64mib-50000",
+    "kind": "schema_and_search_call",
+    "examples": 120,
+    "bytes": 48233,
+    "tokens": 9946,
+    "mean_tokens_per_example": 82.88333333333334
+  }
+]

variants/structured-code-v1-32768/config.json

@@ -0,0 +1,18 @@
+{
+  "format": "huggingface-tokenizers-json",
+  "vocab_size": 32768,
+  "special_tokens": {
+    "<|pad|>": 0,
+    "<|bos|>": 1,
+    "<|eos|>": 2,
+    "<|unk|>": 3
+  },
+  "normalization": null,
+  "pretokenization_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "add_prefix_space": false,
+  "automatic_bos_eos": false,
+  "padding": null,
+  "truncation": null,
+  "decode_skip_special_tokens_default_for_roundtrips": false,
+  "all_valid_UTF8_supported": true
+}

variants/structured-code-v1-32768/evaluation.json

@@ -0,0 +1,59 @@
+{
+  "name": "focused-64mib-32768",
+  "vocab": 32768,
+  "execution": "modal-cpu",
+  "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+  "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+  "metrics": {
+    "python": {
+      "bytes": 2513044,
+      "tokens": 615688,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 4.08168422967477,
+      "mean_tokens_per_example": 1202.515625,
+      "p95_tokens_per_example": 2184
+    },
+    "json": {
+      "bytes": 2782302,
+      "tokens": 722785,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 3.849418568453966,
+      "mean_tokens_per_example": 1411.689453125,
+      "p95_tokens_per_example": 2514
+    },
+    "git_diff": {
+      "bytes": 187152,
+      "tokens": 52862,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 3.5403881805455715,
+      "mean_tokens_per_example": 103.24609375,
+      "p95_tokens_per_example": 249
+    },
+    "tool_call": {
+      "bytes": 965682,
+      "tokens": 251703,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 3.8365931276146887,
+      "mean_tokens_per_example": 491.607421875,
+      "p95_tokens_per_example": 2304
+    }
+  },
+  "fragmentation": {
+    "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 14,
+    "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 25,
+    "get_user_configuration": 3,
+    "\"additionalProperties\":false": 4,
+    "\"tool_call_id\":\"call_0123456789\"": 12
+  },
+  "correctness_passed": true,
+  "family": "focused",
+  "size_mib": 64
+}

variants/structured-code-v1-32768/freeze-manifest.json

@@ -0,0 +1,12 @@
+{
+  "id": "structured-code-bpe-v1",
+  "status": "frozen",
+  "vocab_size": 32768,
+  "files": {
+    "recipe.json": "a4328e99a51bee64daaee66efe25e80ce01a263d568a69a130b94f687edef407",
+    "config.json": "2a4fa59b0a93ebd0995f3ea308c01484647e9ede935e128b4dba163b384224aa",
+    "tokenizer.json": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+    "evaluation.json": "922b0fc33a0c38f06eaedbf3497a50723f4c66ac4dd8f96267038ad64764d80f"
+  },
+  "contract": "Immutable. Record tokenizer SHA256 in shards and checkpoints. Prior tokenizer data must be retokenized in a new version."
+}

variants/structured-code-v1-32768/recipe.json

@@ -0,0 +1,37 @@
+{
+  "seed": 20261016,
+  "target_bytes": 67108864,
+  "weights_percent": {
+    "python": 40,
+    "json": 20,
+    "git_diff": 20,
+    "tool_call": 20
+  },
+  "actual_bytes": {
+    "python": 26847131,
+    "json": 13424847,
+    "git_diff": 13421882,
+    "tool_call": 13422435
+  },
+  "training_examples": {
+    "python": 5059,
+    "json": 2290,
+    "git_diff": 33758,
+    "tool_call": 6555
+  },
+  "fitting_parents": 14911,
+  "evaluation_parents_per_category": 512,
+  "train_sha256": "cdc34355495f1c21bba4742ac11e9bcc0a0d0df836ab49eed9a1e01f97ab9a09",
+  "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+  "source_train_sha256": "5e1c04e310375d9922e008142e7d9813f56e541918b729dd7764eaec3ffbadea",
+  "source_validation_sha256": "611010954c9c48ccad9b9a28f7377d56ab180373b5e7eb76fe87ab723b2b8040",
+  "implementation_sha256": "cd1f6e5351aad5acce1a1021af649c4dd8a099c2b0d506e7cc67cd5aee230163",
+  "limitations": [
+    "Python excerpts from the existing pinned codeparrot corpus, not necessarily complete parseable files",
+    "JSON, tool calls and patches are generated controlled fixtures, not independent production traces",
+    "Fitting/evaluation parents are disjoint; generator templates are shared",
+    "Each category quota includes its final whole example, so 64 MiB is approximate",
+    "Unified diffs cover insertions, deletions and identifier replacements, but not binary patches or renames",
+    "One fitting seed; no model-level tool-call or code-generation correctness measured"
+  ]
+}

variants/structured-code-v1-32768/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

variants/structured-code-v1-32768/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}