---
library_name: transformers
license: cc-by-nc-4.0
language:
- en
tags:
- tokenizer
- bpe
- byte-level
- tr-hash
---

# TR-HASH Tokenizer 32K

The 32,000-token ByteLevel BPE tokenizer used by the TR-HASH language-model
research line and the 200B-token pretraining mixture.

## Properties

- Vocabulary size: 32,000
- Model: ByteLevel BPE
- Normalization: Unicode NFC
- EOS: `</s>` (ID 0)
- PAD: `<pad>` (ID 1)
- BOS: `<s>` (ID 2)
- UNK: `<unk>` (ID 3)
- Declared model context: 2,048 tokens

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "AETHORIA-AI/TR-HASH-Tokenizer-32K"
)
encoded = tokenizer("Token routing starts with tokenization.")
decoded = tokenizer.decode(encoded["input_ids"])
```

This repository contains the tokenizer only. It does not define a chat
template or a model architecture.

## Comparison with GPT `r50k_base`

`r50k_base` is used here as the approximately 50K-token GPT reference. The
comparison uses `transformers` without added special tokens for TR-HASH and
`tiktoken` for `r50k_base`.

| Encoding | Vocabulary | Tokens on fixed suite | Characters/token |
|---|---:|---:|---:|
| TR-HASH Tokenizer 32K | 32,000 | 375 | 3.712 |
| GPT `r50k_base` | 50,257 | 354 | 3.932 |

TR-HASH uses a **36.3% smaller vocabulary**. On the fixed 1,392-character
suite included in [`benchmark_r50k.py`](benchmark_r50k.py), it produces 5.9%
more tokens than `r50k_base`. At hidden size 1,024, the smaller vocabulary
removes 18,695,168 embedding parameters, or about 37.4 MB in BF16/FP16, when
input and output embeddings are tied.

This small suite covers English prose, technical text, code/JSON, mathematics,
and French. It is an illustrative, reproducible check rather than a claim
about every corpus. Compression should be measured again on the intended
training or deployment distribution.

```bash
pip install transformers tiktoken
python benchmark_r50k.py
```

## Associated dataset

[AETHORIA-AI/data-32k-200b-tokens](https://huggingface.co/datasets/AETHORIA-AI/data-32k-200b-tokens)