README.md

metadata

license: mit
library_name: tokenizers
language:
  - en
tags:
  - tokenizer
  - bpe
  - byte-level-bpe
  - text
pipeline_tag: text-generation

my-tokenizer

byte-level Byte-Pair Encoding (BPE) tokenizer trained from scratch with the tokenizers library. It was built as a learning exercise: text is scraped from a Wikipedia article, a BPE tokenizer is trained on it, and the result is wrapped as a PreTrainedTokenizerFast so it loads through AutoTokenizer.

Details

Property	Value
Algorithm	Byte-level BPE
Vocabulary size	2048 (2¹¹)
Special tokens	<unk>, <pad>, <bos>, <eos>
Pre-tokenizer	ByteLevel(add_prefix_space=False)
Decoder	ByteLevel
Training data	Plain text of the English Wikipedia article Large language model

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gorkemergune/my-tokenizer")

example = "Hello! This is a very small tokenizer example."
token_ids = tokenizer.encode(example)

print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids))
print("Token IDs:", token_ids)
print("Decoded:", tokenizer.decode(token_ids))

Limitations

• Trained on a single, short Wikipedia article, so its merges are biased toward that domain (LLM / machine-learning vocabulary) and English text.
• The vocabulary is intentionally tiny (2048), so longer or rarer words are split into many sub-tokens. This is a demo tokenizer, not a production one.

How it was built

The training pipeline is available on GitHub (gorkemergune/wiki2bpe). In short:

1. scraper.py downloads the Wikipedia article as plain text into text.txt.
2. script.py trains the byte-level BPE tokenizer on text.txt and pushes it here.