README.md

---
language:
- en
license: apache-2.0
tags:
- tokenizer
- bpe
- byte-level
- english
- code
- capcode
- length-max
- text-tokenization
---

# StenToken

**StenToken** is a fork-trained BPE tokenizer family with separate English and English+code bundles across vocabulary targets from 1,024 to 32,768 tokens. These deliberately small vocabularies explore compact embedding tables and text segmentation; current comparison results do not show StenToken as competitive with the alternatives listed below.

> This is a **tokenizer repository**, not a language-model repository. It contains tokenizer artifacts, the separately scoped `StenTokenRuntime-rs` inference runtime, and this model card. It does not include model weights, a training script, or the original custom Python source code.

---

## Basic Information

StenToken is a compact tokenizer family that uses Capcode to represent capitalization.

### Current status and intended use

This first version of StenToken is a stepping stone in StentorLabs’ tokenizer work, not a recommended tokenizer for training a new language model or for production deployment. Use it for tokenizer research, inspection, and learning while better StentorLabs tokenizers are developed.

### Important Unicode round-trip limitation

StenToken does **not** fully guarantee exact Unicode-code-point reversibility. Python case conversion used by its Capcode path can change a character's code-point composition, so decoding may not reproduce the original code-point sequence in every case. If your application requires strict code-point-for-code-point round trips, treat this as a material limitation and validate your own text before adopting StenToken.

### Use the tokenizer

Install the Hugging Face Hub client:

~~~bash
pip install huggingface_hub
~~~

Download the English or EnglishCode bundle your compatible model requires:

~~~python
from huggingface_hub import hf_hub_download

repo_id = "StentorLabs/StenToken"
subfolder = "Stentoken/English/8k"

files = {
    name: hf_hub_download(repo_id=repo_id, filename=f"{subfolder}/{name}")
    for name in ("vocab.json", "mergeable_ranks.json", "sten_tokenizer.pt")
}
~~~

Load those three files with the matching **StenToken Capcode runtime**. The repository includes the separately scoped Rust inference runtime in `StenTokenRuntime-rs`; its compatibility requirements are summarized in the Token-ID Contract.

### Choosing a size

These estimates use a model’s **core parameter count**, excluding embedding parameters, and assume tied input/output embeddings. They were calculated from Chinchilla scaling laws and the filtered-C4 StenToken bytes-per-token results reported below.

| Vocabulary | Estimated best core range at 20× | Estimated best core range at 600× |
|---:|---:|---:|
| **1,024** | Below **1.5M** | Below **8.03M** |
| **2,048** | **1.5M–5M** | **8M–25M** |
| **4,096** | **5M–17M** | **25M–80M** |
| **8,192** | **17M–70M** | **80M–335M** |
| **16,384** | **70M–290M** | **335M–≈1.4B** |
| **24,576** | **290M–≈485M** | **≈1.4B–≈2.4B** |
| **32,768** | Above **≈485M** | Above **≈2.4B** |

These are exploratory StentorLabs estimates, not established tokenizer-selection rules.

### Pick the right bundle

Choose **English** for English text and **EnglishCode** for English plus code. For example, the 8K English bundle is in **Stentoken/English/8k/**; the 8K English-and-code bundle is in **Stentoken/EnglishCode/8k/**.

### Overall assessment

Many StenToken tokens are ordinary subwords, complete words, Capcode forms, technical vocabulary, and code patterns. A small portion is less useful, including memorized numeric identifiers, project-specific code fragments, repeated boilerplate, and awkward incomplete word fragments.

**Examples of relatively ordinary tokens:**

~~~text
the
ing
tion
information
classification
conversation
instructions
confidence
dataframe
import
return
array([
strip()
readline()
from django.
db import models
<|tool_call|>
<|fim_prefix|>
~~~

**Examples of the worst outlier tokens:**

~~~text
8077683550237, 417.
8077683550243, 417.
",CcrackCcontrolCvars(id
pos= CcrackCcontrolCbase
Csection= "Cm2hastCizqWr
0,steelCstress= -4.
slha', 'condition': 0.
of the Clicense at\n#\n#
setChorizontalCstret
http://ww
iloc[i,2]+
CqCsize(45, 15))\n
~~~

These are rare, worst-case outliers and are not representative of the vocabulary overall.

### Reported bytes-per-token (BPT) comparison

The following table gives the comparison. **BPT means bytes per token**: the average number of input bytes represented by one token. The table reports the measured values for each corpus.

| Tokenizer | Vocab size | C4 filtered bytes/token | C4 unfiltered bytes/token | Python bytes/token |
|---|---:|---:|---:|---:|
| TokenMonster english-1024-balanced-v1 | 1,024 | 2.488 | 1.868 | 1.932 |
| TokenMonster english-2048-balanced-v1 | 2,048 | 3.006 | 2.163 | 2.270 |
| TokenMonster english-4096-balanced-v1 | 4,096 | 3.582 | 2.526 | 2.579 |
| TokenMonster english-8000-balanced-v1 | 8,000 | 4.145 | 2.800 | 2.885 |
| TokenMonster english-16000-balanced-v1 | 16,000 | 4.754 | 3.164 | 3.220 |
| TokenMonster english-24000-balanced-v1 | 24,000 | 5.107 | 3.370 | 3.412 |
| TokenMonster english-32000-balanced-v1 | 32,000 | 5.354 | 3.510 | 3.531 |
| StenToken English | 1,024 | 1.629 | 1.418 | 1.300 |
| StenToken English | 2,048 | 1.805 | 1.537 | 1.382 |
| StenToken English | 4,096 | 1.999 | 1.689 | 1.455 |
| StenToken English | 8,192 | 2.205 | 1.851 | 1.545 |
| StenToken English | 16,384 | 2.391 | 2.026 | 1.668 |
| StenToken English | 24,576 | 2.473 | 2.126 | 1.741 |
| StenToken English | 32,768 | 2.534 | 2.202 | 1.772 |
| StenToken EnglishCode | 1,024 | 1.614 | 1.417 | 1.428 |
| StenToken EnglishCode | 2,048 | 1.790 | 1.542 | 1.536 |
| StenToken EnglishCode | 4,096 | 1.982 | 1.692 | 1.658 |
| StenToken EnglishCode | 8,192 | 2.184 | 1.858 | 1.756 |
| StenToken EnglishCode | 16,384 | 2.374 | 2.044 | 1.843 |
| StenToken EnglishCode | 24,576 | 2.457 | 2.132 | 1.880 |
| StenToken EnglishCode | 32,768 | 2.520 | 2.206 | 1.917 |
| Mistral-7B-v0.1 | 32,000 | 4.159 | 3.000 | 3.093 |
| Llama-2-7b | 32,000 | 4.002 | 2.918 | 3.057 |
| Phi-3-mini-4k-instruct | 32,000 | 4.002 | 2.918 | 3.057 |
| GPT-2 | 50,257 | 4.542 | 3.504 | 2.180 |

The reported BPT results are tokenizer-level measurements. They do not establish downstream model quality, accuracy, or usefulness for training.

### Why it is called StenToken

StenToken is named after **Stentor**, the tiny but smart single-celled organism.

---

# Detailed Overview

Everything below is reference material for people comparing tokenizer designs or investigating exact behavior.

## Tokenizer Architecture

| Component | Value | Notes |
|---|---:|---|
| Tokenizer family | BPE | Rank-based byte-pair encoding at inference |
| Published fork targets | 1,024–32,768 | Seven artifact sizes; trained separately by family |
| Merge objective | Length-MAX | Frequency weighted by combined merge length |
| Length exponent | Scheduled, mix-aware | Starts at 1.2; rises to the active family’s endpoint |
| Semantic gate | Scheduled NPMI | Rejects low-association candidate merges |
| Subword phase | First | Learns primarily within pre-token units |
| Superword phase | Second | Adds cross-boundary structure after subword learning |
| Case representation | Capcode-only | Required throughout the published 1,024–32,768 family |
| Inference | Rank-based BPE | Applies learned merges in rank order |
| Stochastic mode | BPE-dropout | Optional training-time segmentation variation |

## Token-ID Contract

The ID layout is deliberately explicit. A compatible model must preserve these assignments exactly.

| ID range | Meaning | Stability requirement |
|---:|---|---|
| 0 | Padding | Fixed |
| 1–256 | Byte fallback tokens for byte values 0x00–0xFF | Fixed |
| 257+ | Control tokens and learned BPE token surfaces | Bundle-specific |
| Learned token range | BPE vocabulary learned from the training corpus | Must match the exported vocab and merge ranks |
| vocab.json token IDs | Embedding rows and output logits are indexed by these IDs | Must remain unchanged |
| mergeable_ranks.json ordering | Changes segmentation and therefore input IDs | Must remain unchanged |
| sten_tokenizer.pt state | Must match the vocabulary and merge ranks | Bundle-specific |
| Capcode preprocessing | Mandatory text representation before BPE | Must remain enabled |
| Special/control token IDs | Changes sequence structure and padding semantics | Must remain unchanged |
| Padding ID (0) | Affects attention masks and loss masking | Fixed |
| Artifact revision | Small tokenizer changes are model-breaking changes | Record the exact revision |

### Reserved control tokens

The tokenizer reserves control-token IDs separately from learned BPE surfaces. Raw user text should be encoded with the ordinary text encoder. Applications that need to insert controls should do so structurally, using their known IDs, rather than relying on a textual occurrence being interpreted as a control token.

This separation prevents a string in user text from silently changing sequence structure.

---

## Core Design Choices

### 1. Length-MAX merge selection

Vanilla BPE selects candidate merges mainly by frequency. StenToken uses a length-weighted objective:

~~~text
score(a, b) = frequency(a, b) × (length(a) + length(b))^1.8
~~~

The exponent is not fixed at 1.8 during the fork pipeline. It starts at 1.2 for the first 65% of merge progress, then rises linearly toward the active family’s mix-aware endpoint. With the default sources, the endpoints are approximately 1.769 for English and 1.725 for English+code; 1.8 is the global ceiling.

The intent is to give a recurring longer unit more credit than a very short pair with the same frequency. That may affect segmentation at compact vocabulary sizes, where spending a learned slot on a longer unit matters more.

### 2. NPMI semantic merge gate

Candidate pairs must satisfy a normalized pointwise mutual information gate before they can be considered for merging. NPMI is bounded to the interval from -1 to 1, which makes it less dominated by very rare coincidences than raw PMI.

| Gate | Default threshold | Role |
|---|---:|---|
| Subword NPMI | 0.4 → 0.2 | Relaxes linearly as the vocabulary fills |
| Superword NPMI | 0.5 → 0.2 | Starts stricter, then relaxes over merge progress |
| Fork minimum pair frequency | 50 | Effective floor for every default target below 65,536 |
| Dynamic frequency floor | 2e-7 of corpus scale | Raises the effective floor when corpus scale requires it |

The gate is intended to screen out one class of undesirable token: a frequent-looking sequence that has little stable semantic or structural relationship. It cannot prove that a token is linguistically meaningful.

### 3. Two-phase BPE

StenToken does not open all possible cross-boundary merges immediately.

| Target range | Subword allocation | Superword allocation |
|---|---:|---|
| 1,024–16,384 | 93.75% | 6.25% |
| 24,576–32,768 | 87.5% | 12.5% |

The subword allocation learns word-internal morphology, common spellings, and code fragments. The remaining slots are reserved for cross-boundary superword units. Delaying cross-boundary merges is a guardrail: if they are available from the first merge, broad fragments such as a leading space plus a common word can absorb capacity too early and crowd out internal structure.

### 4. Picky-BPE and LiteToken cleanup

The training pipeline includes two vocabulary-management mechanisms:

- **Picky-BPE pruning:** a scaffold token can be removed during training when a merge explains nearly all of its usage.
- **LiteToken residue handling:** tokens frequently created as merge targets but rarely used in final segmentation can be identified as residues.

These mechanisms are intended to manage vocabulary allocation. They do not guarantee exact reversibility.

### 5. Rank-based BPE inference

At encoding time, StenToken applies the lowest-ranked available merge repeatedly within each pre-tokenized chunk. This is ordinary rank-based BPE behavior rather than a greedy search over arbitrary learned strings.

---

## Vocabulary Training

### Streaming-first corpus collection

The training system streams source datasets rather than downloading the full corpus to local disk. It uses bounded retention and checkpointing so a large run can continue after a runtime interruption.

| Training control | Default | Why it exists |
|---|---:|---|
| Shared corpus target | 150 GB of text | Normal fork-pipeline default |
| English family word types | 1,200,000 → 900,000 | Effective cap and prune target |
| English+code family word types | 1,500,000 → 1,200,000 | Effective cap and prune target |
| Checkpoint cadence | 5 GB or 20 min | Makes long runs resumable |
| Sentence-cache cap | 150,000 types | Bounds cross-boundary statistics |
| BPE dropout protected ranks | First 500 | Keeps foundational merges stable |

The published training process derives both family variants, then branches each family to its target sizes. Checkpoints are training artifacts, not runtime dependencies for deployment.

### Training objective details

Candidate merge scoring combines frequency, length, and association quality. The system tracks document/source sketches, pair counts, and pre-token statistics in an attempt to reduce the chance that a small number of repeated documents dominates candidate selection.

The implementation uses indexed mutable BPE state and periodically rescored pair heaps to avoid rebuilding the full candidate table after every merge. This is intended to reduce work at larger corpus sizes, but it does not establish practical performance or eliminate resource requirements.

### Reproducibility boundaries

The default seed is 42. Exact reproduction can still differ when dataset snapshots, streaming order, library versions, source availability, and interruption/resume points differ. Tokenizer training should be considered reproducible only when the full artifact set, source revisions, configuration, and environment are controlled.

---

## Data Pipeline

The shared input mixture is English-heavy and includes Python code:

| Source | Configuration | Weight | Intended contribution |
|---|---|---:|---|
| HuggingFaceFW/fineweb | sample-10BT, train | 50% | General English web text |
| wikimedia/wikipedia | 20231101.en, train | 20% | Encyclopedic English |
| fddemarco/pushshift-reddit-comments | train | 10% | Informal conversational text |
| Cyrile/dataset-the-stack-v2-dedup-sub | Python, train | 20% | Python code |

The percentages are configured shared-stream weights, not a guarantee of exact final byte proportions. The normal fork pipeline derives two output families from this stream:

| Output family | Sources used | Effective family weights |
|---|---|---|
| English | FineWeb, Wikipedia, Reddit | 62.5% / 25% / 12.5% after normalization |
| English+code | All four shared sources | 50% / 20% / 10% / 20% |

Streaming sources can contain unavailable shards, changed revisions, formatting variation, duplicate content, and text that fails quality gates.

### Quality and normalization controls

Before vocabulary statistics are accumulated, the pipeline can apply lightweight checks and normalization:

- Unicode normalization with punctuation normalization.
- Source-specific code handling; general web text preserves Unicode by default.
- Bounded control-character ratio.
- Repeated-character-run handling.
- Text-length and allowed-character checks.
- Rolling document deduplication.
- Code-aware splitting for identifiers such as snake_case and CamelCase.
- Preservation of whitespace in pre-tokenization.

These filters are pragmatic data engineering. They are not a safety filter and they do not guarantee data quality, legality, representativeness, or absence of personal information in upstream corpora.

---

## Text Processing

### Unicode and byte fallback

Input text is represented through UTF-8 bytes internally. The learned vocabulary contains byte-space surfaces, while byte fallback preserves coverage for anything not represented by a learned merge.

This means StenToken can encode non-English text, emoji, unusual punctuation, and arbitrary byte sequences. It does **not** mean it is optimized for them.

### Capcode

StenToken is a Capcode tokenizer. Capcode is the capitalization representation used throughout the published 1,024–32,768 family. It is applied before StenToken’s pre-tokenization and ranked BPE stages.

- It reduces duplicated vocabulary capacity across lowercase, title-case, and uppercase forms.
- It preserves casing rather than simply lowercasing input.
- It gives the tokenizer a consistent representation for all-caps spans, identifiers, and ordinary words.

#### Inspiration and attribution

Thank you to **Alasdair Forsythe** for the Capcode tokenizer inspiration. StenToken’s custom Capcode system is a tweaked version of the TokenMonster approach: it keeps the central idea of sharing lowercase, title-case, and uppercase vocabulary capacity while adding behavior for mixed prose, source code, identifiers, acronyms, and technical text.

#### Code-aware identifier decomposition

StenToken recognizes common identifier structure, including CamelCase, PascalCase, snake_case, acronym transitions, and letter-and-number boundaries. This lets an identifier such as the following expose component pieces instead of becoming one opaque unit:

~~~text
getHTTPResponse
~~~

That can create overlap across related identifiers:

~~~text
getHTTPResponse
parseHTTPResponse
HTTPResponseCode
http_response
~~~

This identifier decomposition is an additional StenToken capability for code-heavy data; it goes beyond capitalization encoding alone.

#### Consecutive uppercase spans

StenToken adds a block-capitalization representation using B and E markers. A whitespace-separated span such as the following can use one uppercase block rather than a separate capitalization instruction before every word:

~~~text
UNITED STATES NAVY
→ Bunited states navyE
~~~

This is intended to reduce capitalization-marker overhead for headings, acronyms, constants, legal text, and technical labels before BPE begins. It does not by itself guarantee a lower final token count, because learned BPE merges can segment either representation differently.

#### Honest limits

StenToken’s Capcode is not inherently better merely because it is newer or more elaborate. It has concrete limits:

- Its custom C, W, B, and E marker language requires the StenToken decoder; it is not interchangeable with TokenMonster’s native decoder.
- Uppercase-block detection is optimized for whitespace-separated words. Punctuation-separated spans such as NASA-USA-API do not receive the same block treatment.

Future improvements include formally lossless Unicode handling and explicit validation or constraints for valid C, W, B, and E sequences.

### Pretokenization

The pre-tokenizer preserves whitespace and handles several text/code patterns deliberately:

| Pattern | Intended handling |
|---|---|
| Whitespace | Preserved rather than discarded |
| Contractions | Kept as structured text units |
| Python identifiers | Can be split around meaningful case/underscore components |
| Operators | Common multi-character operators treated atomically |
| Keywords with trailing space | Recognized to support code structure |
| Repeated-character runs | Isolated or bounded to avoid pathological learned units |

No pre-tokenization scheme is neutral. These choices prioritize English prose and Python-like code, and may be less appropriate for languages or data formats with different boundaries.

---
---

## Ethical Considerations & Societal Impact

### Inherited data biases

The tokenizer’s vocabulary is shaped by its source mixture. Its web and code sources can overrepresent some dialects, topics, technical conventions, and online communities while underrepresenting others.

Tokenizer bias can appear as uneven compression: a form of text that uses fewer tokens may receive more effective context capacity in a fixed-window model than a form that fragments into many byte-level pieces.

### Content and privacy

Upstream web-scale sources can include harmful, incorrect, sensitive, or personal content. A tokenizer does not retain documents as a retrieval database, but its vocabulary-learning process is still derived from corpus statistics. Use appropriate data governance, licensing review, and privacy practices when retraining or extending it.

---

## Performance and Resource Considerations

Runtime tokenization cost depends on text length, merge-table size, backend, batching, and hardware. The artifact itself is much smaller than a language model, but tokenizer throughput can still matter in high-volume preprocessing.

---

## Related Work

| Work | Relationship |
|---|---|
| Byte Pair Encoding (BPE) | Base merge-based vocabulary method |
| BPE-dropout | Stochastic merge dropping for segmentation augmentation |
| PMI/NPMI association measures | Used to assess merge association strength |
| Picky BPE | Influences vocabulary-pruning strategy |
| Subword regularization | Broader family of stochastic segmentation approaches |

---

## Environmental Impact

Tokenizer training consumes compute through streaming, statistics collection, pair scoring, checkpointing, and merge selection. This repository does not publish a verified energy or carbon measurement for a particular completed tokenizer run.

---

## Citation

If you use StenToken in research or a project, please cite the tokenizer repository and record the exact artifact revision:

~~~bibtex
@software{stentoken,
  title        = {StenToken: Streaming Length-MAX Byte-Fallback BPE Tokenizer},
  author       = {StentorLabs},
  year         = {2026},
  url          = {https://huggingface.co/StentorLabs/StenToken},
  note         = {Tokenizer artifacts; record the exact revision used}
}
~~~

---

## StenToken Card Contact

Questions, benchmarks, or feedback: [StentorLabs@gmail.com](mailto:StentorLabs@gmail.com) or open a [discussion](https://huggingface.co/StentorLabs/StenToken/discussions).

<div align="center">

Made with ❤️ by [StentorLabs](https://huggingface.co/StentorLabs)

_Democratizing AI through accessible, efficient models — trained on free compute, shared with everyone._

</div>