Update tokenizer: 73K training objects, 125 keys, DOI 10.5281/zenodo.18879110

README.md

@@ -16,7 +16,7 @@ pipeline_tag: text-generation
 
 A structure-aware tokenizer that assigns dedicated single tokens to JSON grammar elements, learns a compact key vocabulary from training data, and applies byte-pair encoding (BPE) only to value content.
 
-**Paper:** [Structure-Aware Tokenization for JSON: Exploiting Schema Repetition for Compact Token Sequences with a 90x Smaller Vocabulary](https://doi.org/10.5281/zenodo.XXXXXXX)
+**Paper:** [Structure-Aware Tokenization for JSON: Exploiting Schema Repetition for Compact Token Sequences with a 90x Smaller Vocabulary](https://doi.org/10.5281/zenodo.18879110)
 
 **Code:** [github.com/anthony-maio/json-tokenizer](https://github.com/anthony-maio/json-tokenizer)
 
@@ -25,7 +25,7 @@ A structure-aware tokenizer that assigns dedicated single tokens to JSON grammar
 | Metric | Value |
 |--------|-------|
 | Token savings vs cl100k_base | **5-15%** on schema-repetitive JSON |
-| Vocabulary size | **4,190 tokens** (vs 100,256 for cl100k_base) |
+| Vocabulary size | **4,251 tokens** (vs 100,256 for cl100k_base) |
 | Vocab reduction | **~90x smaller** |
 | Roundtrip fidelity | **100% lossless** across 4,200+ test objects |
 | Crossover point | Beats cl100k_base at just **558 tokens** |
@@ -34,19 +34,20 @@ A structure-aware tokenizer that assigns dedicated single tokens to JSON grammar
 
 Three-tier vocabulary:
 1. **Structural tokens** (IDs 0-15): `{`, `}`, `[`, `]`, `:`, `,`, `true`, `false`, `null`, type markers
-2. **Key vocabulary** (IDs 32-N): Learned single-token keys from training data (65 keys)
+2. **Key vocabulary** (IDs 32-N): Learned single-token keys from training data (125 keys)
 3. **BPE subwords** (IDs N+1 to N+B): Byte-pair encoding trained on JSON value strings (4,096 tokens)
 
 ## This Model
 
-This pretrained tokenizer was trained on four structured JSON datasets:
+This pretrained tokenizer was trained on structured JSON datasets:
 - GeoJSON city features (geographic data)
 - Observability telemetry logs (monitoring data)
 - Kubernetes manifests (infrastructure config)
 - Structured API outputs
+- Synthetic training corpus (700 objects)
 
-**Total training objects:** 10,530
-**Vocabulary:** 4,190 tokens (16 structural + 16 reserved + 65 keys + 4,096 BPE)
+**Total training objects:** 72,991
+**Vocabulary:** 4,251 tokens (16 structural + 16 reserved + 125 keys + 4,096 BPE)
 
 ## Usage
 
@@ -114,11 +115,14 @@ hf_tok.save_pretrained("./my_hf_tokenizer")
 ## Citation
 
 ```bibtex
-@article{maio2026json,
-  title={Structure-Aware Tokenization for JSON: Exploiting Schema Repetition for Compact Token Sequences with a 90x Smaller Vocabulary},
-  author={Maio, Anthony},
-  year={2026},
-  url={https://doi.org/10.5281/zenodo.XXXXXXX}
+@software{maio2026jsontokenizer,
+  author    = {Maio, Anthony},
+  title     = {Structure-Aware Tokenization for {JSON}: Exploiting Schema Repetition for Compact Token Sequences with a 90x Smaller Vocabulary},
+  year      = {2026},
+  url       = {https://github.com/anthony-maio/json-tokenizer},
+  doi       = {10.5281/zenodo.18879110},
+  version   = {0.2.0},
+  license   = {MIT}
 }
 ```