---
license: mit
library_name: tokenizers
language:
- en
tags:
- tokenizer
- bpe
- byte-level-bpe
- text
pipeline_tag: text-generation
---

# my-tokenizer

**byte-level Byte-Pair Encoding (BPE)** tokenizer trained from scratch with
the `tokenizers` library. It was built as a learning exercise: text is scraped
from a Wikipedia article, a BPE tokenizer is trained on it, and the result is
wrapped as a `PreTrainedTokenizerFast` so it loads through `AutoTokenizer`.

## Details

| Property | Value |
|---|---|
| Algorithm | Byte-level BPE |
| Vocabulary size | 2048 (2¹¹) |
| Special tokens | `<unk>`, `<pad>`, `<bos>`, `<eos>` |
| Pre-tokenizer | `ByteLevel(add_prefix_space=False)` |
| Decoder | `ByteLevel` |
| Training data | Plain text of the English Wikipedia article *Large language model* |


## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gorkemergune/my-tokenizer")

example = "Hello! This is a very small tokenizer example."
token_ids = tokenizer.encode(example)

print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids))
print("Token IDs:", token_ids)
print("Decoded:", tokenizer.decode(token_ids))
```

## Limitations

- Trained on a **single, short** Wikipedia article, so its merges are biased toward
  that domain (LLM / machine-learning vocabulary) and English text.
- The vocabulary is intentionally tiny (2048), so longer or rarer words are split
  into many sub-tokens. This is a demo tokenizer, not a production one.

## How it was built

The training pipeline is available on GitHub (gorkemergune/wiki2bpe). 
In short:

1. `scraper.py` downloads the Wikipedia article as plain text into `text.txt`.
2. `script.py` trains the byte-level BPE tokenizer on `text.txt` and pushes it here.