| --- |
| library_name: transformers |
| license: cc-by-nc-4.0 |
| language: |
| - en |
| tags: |
| - tokenizer |
| - bpe |
| - byte-level |
| - tr-hash |
| --- |
| |
| # TR-HASH Tokenizer 32K |
|
|
| The 32,000-token ByteLevel BPE tokenizer used by the TR-HASH language-model |
| research line and the 200B-token pretraining mixture. |
|
|
| ## Properties |
|
|
| - Vocabulary size: 32,000 |
| - Model: ByteLevel BPE |
| - Normalization: Unicode NFC |
| - EOS: `</s>` (ID 0) |
| - PAD: `<pad>` (ID 1) |
| - BOS: `<s>` (ID 2) |
| - UNK: `<unk>` (ID 3) |
| - Declared model context: 2,048 tokens |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| "AETHORIA-AI/TR-HASH-Tokenizer-32K" |
| ) |
| encoded = tokenizer("Token routing starts with tokenization.") |
| decoded = tokenizer.decode(encoded["input_ids"]) |
| ``` |
|
|
| This repository contains the tokenizer only. It does not define a chat |
| template or a model architecture. |
|
|
| ## Comparison with GPT `r50k_base` |
| |
| `r50k_base` is used here as the approximately 50K-token GPT reference. The |
| comparison uses `transformers` without added special tokens for TR-HASH and |
| `tiktoken` for `r50k_base`. |
|
|
| | Encoding | Vocabulary | Tokens on fixed suite | Characters/token | |
| |---|---:|---:|---:| |
| | TR-HASH Tokenizer 32K | 32,000 | 375 | 3.712 | |
| | GPT `r50k_base` | 50,257 | 354 | 3.932 | |
|
|
| TR-HASH uses a **36.3% smaller vocabulary**. On the fixed 1,392-character |
| suite included in [`benchmark_r50k.py`](benchmark_r50k.py), it produces 5.9% |
| more tokens than `r50k_base`. At hidden size 1,024, the smaller vocabulary |
| removes 18,695,168 embedding parameters, or about 37.4 MB in BF16/FP16, when |
| input and output embeddings are tied. |
|
|
| This small suite covers English prose, technical text, code/JSON, mathematics, |
| and French. It is an illustrative, reproducible check rather than a claim |
| about every corpus. Compression should be measured again on the intended |
| training or deployment distribution. |
|
|
| ```bash |
| pip install transformers tiktoken |
| python benchmark_r50k.py |
| ``` |
|
|
| ## Associated dataset |
|
|
| [AETHORIA-AI/data-32k-200b-tokens](https://huggingface.co/datasets/AETHORIA-AI/data-32k-200b-tokens) |
|
|
