---
language: en
tags:
- tokenizers
- wordpiece
- bytepairencoding
- xlnet
- nlp
license: mit
---

# Basic Tokenizers Collection

This repository contains **three different tokenizers** trained and wrapped for experimentation and educational purposes:

## 📦 Contents
- **WordPiece Tokenizer**  
  Path: `ByteMeHarder-404/tokenizers/wordpiece`  
  Classic subword tokenizer (used in BERT). Splits words into subword units based on frequency, ensuring full coverage with a compact vocab.

- **Byte-Pair Encoding (BPE) Tokenizer**  
  Path: `ByteMeHarder-404/tokenizers/bpe`  
  Uses byte-level BPE, similar to GPT-2 and RoBERTa. Handles any UTF-8 character without unknown tokens by working directly on bytes.

- **XLNet-Style Tokenizer**  
  Path: `ByteMeHarder-404/tokenizers/xlnet`  
  Follows the XLNet tokenization approach, leveraging sentencepiece-like segmentation.

## 🚀 Usage
You can load each tokenizer with `transformers`:

```python
from transformers import PreTrainedTokenizerFast

# WordPiece
tok_wordpiece = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/wordpiece")

# BPE
tok_bpe = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/bpe")

# XLNet-style
tok_xlnet = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/xlnet")
```

## 📚 Notes

- These tokenizers are minimal examples and **not pretrained with embeddings or models**.  
- Intended for experimentation, educational purposes, and as a foundation for building custom models.  
- You can extend them by training a new vocabulary on your dataset.