| --- |
| library_name: transformers |
| language: |
| - en |
| tags: |
| - tokenizer |
| - byte-level-bpe |
| - code |
| - tool-calling |
| --- |
| |
| # Code and tool-calling tokenizer |
|
|
| The default is **structured-code-v1: 32,768 entries, fitted on approximately 64 MiB** |
| on Modal. It was explicitly selected for code and tool-calling workloads. |
| This repository contains tokenizer assets, not language-model weights. |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("jfrick/scholarly-code-tokenizer") |
| text = '{"name":"read_file","arguments":{"path":"src/main.py"}}' |
| ids = tokenizer.encode(text, add_special_tokens=False) |
| assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text |
| |
| # The previous balanced default remains available: |
| previous = AutoTokenizer.from_pretrained( |
| "jfrick/scholarly-code-tokenizer", subfolder="variants/2048mib-32768" |
| ) |
| ``` |
|
|
| The default is also available in `variants/structured-code-v1-32768`. All 18 original |
| learning-curve variants remain available under their existing names. Pin a commit |
| revision when training models; changing a tokenizer requires retokenizing data. |
| Old tokenized shards and model checkpoints must not be relabeled with the new IDs. |
|
|
| ## Fitting and measurements |
|
|
| The focused fitting mixture is 40% real Python excerpts, 20% JSON, 20% Git diffs and |
| 20% tool-call formats by UTF-8 bytes. JSON, diffs and tool formats are generated |
| controlled fixtures derived from Python excerpts. Fitting and evaluation source |
| parents are disjoint, but generator templates are shared. They are not production |
| traces. Source Python comes from the pinned, license-filtered CodeParrot inventory |
| in the original [source recipe](recipe.json); raw source text is not distributed. |
| Source datasets retain their respective licenses and provenance. |
|
|
| Compared with the previous 2 GiB / 32K balanced default, this tokenizer uses 2.81% |
| fewer tokens on Python, 8.21% fewer on JSON, 11.83% fewer on Git diffs, and 13.97% |
| fewer on tool-format fixtures. It uses **13.99% more tokens on the broader scholarly |
| mixture**. This is an explicit specialization trade-off, not a claim of better |
| language-model quality or valid tool execution. |
|
|
| [Structured-output report](research/structured-code-v1/REPORT.md) · |
| [PDF](research/structured-code-v1/research-report.pdf) · |
| [Original learning-curve report](research/REPORT.md) |
|
|
| The reports preserve their original experimental conclusions; this model card |
| records the subsequent user decision to promote the focused tokenizer. |
|
|
| ## Contract and reproducibility |
|
|
| Byte-level BPE; full 256-byte alphabet; no normalization, prefix insertion or |
| automatic BOS/EOS. PAD=0, BOS=1, EOS=2, UNK=3. Add document boundaries explicitly. |
| No model context length is implied. All held-out examples pass exact round-trip |
| and zero-UNK checks. JSON validity and tool-call success require model evaluation. |
|
|
| Standard loading is verified with `transformers==4.57.6`, `tokenizers==0.22.2`; |
| no custom remote code is required. Decode with `skip_special_tokens=False` to |
| preserve special literals. The Rust `tokenizers.Tokenizer.from_file` API also works. |
|
|
| [Publication manifest](publication-manifest.json) pins all tokenizer hashes. |
| The default variant includes its immutable freeze manifest, fitting corpus metadata |
| and evaluation metrics. [Implementation PR](https://github.com/koogle/lm-exploration/pull/8). |
|
|
