---
library_name: transformers
language:
- en
tags:
- tokenizer
- byte-level-bpe
- code
- tool-calling
---

# Code and tool-calling tokenizer

The default is **structured-code-v1: 32,768 entries, fitted on approximately 64 MiB**
on Modal. It was explicitly selected for code and tool-calling workloads.
This repository contains tokenizer assets, not language-model weights.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("jfrick/scholarly-code-tokenizer")
text = '{"name":"read_file","arguments":{"path":"src/main.py"}}'
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text

# The previous balanced default remains available:
previous = AutoTokenizer.from_pretrained(
    "jfrick/scholarly-code-tokenizer", subfolder="variants/2048mib-32768"
)
```

The default is also available in `variants/structured-code-v1-32768`. All 18 original
learning-curve variants remain available under their existing names. Pin a commit
revision when training models; changing a tokenizer requires retokenizing data.
Old tokenized shards and model checkpoints must not be relabeled with the new IDs.

## Fitting and measurements

The focused fitting mixture is 40% real Python excerpts, 20% JSON, 20% Git diffs and
20% tool-call formats by UTF-8 bytes. JSON, diffs and tool formats are generated
controlled fixtures derived from Python excerpts. Fitting and evaluation source
parents are disjoint, but generator templates are shared. They are not production
traces. Source Python comes from the pinned, license-filtered CodeParrot inventory
in the original [source recipe](recipe.json); raw source text is not distributed.
Source datasets retain their respective licenses and provenance.

Compared with the previous 2 GiB / 32K balanced default, this tokenizer uses 2.81%
fewer tokens on Python, 8.21% fewer on JSON, 11.83% fewer on Git diffs, and 13.97%
fewer on tool-format fixtures. It uses **13.99% more tokens on the broader scholarly
mixture**. This is an explicit specialization trade-off, not a claim of better
language-model quality or valid tool execution.

[Structured-output report](research/structured-code-v1/REPORT.md) ·
[PDF](research/structured-code-v1/research-report.pdf) ·
[Original learning-curve report](research/REPORT.md)

The reports preserve their original experimental conclusions; this model card
records the subsequent user decision to promote the focused tokenizer.

## Contract and reproducibility

Byte-level BPE; full 256-byte alphabet; no normalization, prefix insertion or
automatic BOS/EOS. PAD=0, BOS=1, EOS=2, UNK=3. Add document boundaries explicitly.
No model context length is implied. All held-out examples pass exact round-trip
and zero-UNK checks. JSON validity and tool-call success require model evaluation.

Standard loading is verified with `transformers==4.57.6`, `tokenizers==0.22.2`;
no custom remote code is required. Decode with `skip_special_tokens=False` to
preserve special literals. The Rust `tokenizers.Tokenizer.from_file` API also works.

[Publication manifest](publication-manifest.json) pins all tokenizer hashes.
The default variant includes its immutable freeze manifest, fitting corpus metadata
and evaluation metrics. [Implementation PR](https://github.com/koogle/lm-exploration/pull/8).