| --- |
| license: mit |
| library_name: tokenizers |
| language: |
| - en |
| tags: |
| - tokenizer |
| - bpe |
| - byte-level-bpe |
| - text |
| pipeline_tag: text-generation |
| --- |
| |
| # my-tokenizer |
|
|
| **byte-level Byte-Pair Encoding (BPE)** tokenizer trained from scratch with |
| the `tokenizers` library. It was built as a learning exercise: text is scraped |
| from a Wikipedia article, a BPE tokenizer is trained on it, and the result is |
| wrapped as a `PreTrainedTokenizerFast` so it loads through `AutoTokenizer`. |
|
|
| ## Details |
|
|
| | Property | Value | |
| |---|---| |
| | Algorithm | Byte-level BPE | |
| | Vocabulary size | 2048 (2¹¹) | |
| | Special tokens | `<unk>`, `<pad>`, `<bos>`, `<eos>` | |
| | Pre-tokenizer | `ByteLevel(add_prefix_space=False)` | |
| | Decoder | `ByteLevel` | |
| | Training data | Plain text of the English Wikipedia article *Large language model* | |
|
|
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("gorkemergune/my-tokenizer") |
| |
| example = "Hello! This is a very small tokenizer example." |
| token_ids = tokenizer.encode(example) |
| |
| print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids)) |
| print("Token IDs:", token_ids) |
| print("Decoded:", tokenizer.decode(token_ids)) |
| ``` |
|
|
| ## Limitations |
|
|
| - Trained on a **single, short** Wikipedia article, so its merges are biased toward |
| that domain (LLM / machine-learning vocabulary) and English text. |
| - The vocabulary is intentionally tiny (2048), so longer or rarer words are split |
| into many sub-tokens. This is a demo tokenizer, not a production one. |
|
|
| ## How it was built |
|
|
| The training pipeline is available on GitHub (gorkemergune/wiki2bpe). |
| In short: |
|
|
| 1. `scraper.py` downloads the Wikipedia article as plain text into `text.txt`. |
| 2. `script.py` trains the byte-level BPE tokenizer on `text.txt` and pushes it here. |
|
|
