File size: 1,589 Bytes
e68e4d0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
---
language: en
tags:
- tokenizers
- wordpiece
- bytepairencoding
- xlnet
- nlp
license: mit
---
# Basic Tokenizers Collection
This repository contains **three different tokenizers** trained and wrapped for experimentation and educational purposes:
## 📦 Contents
- **WordPiece Tokenizer**
Path: `ByteMeHarder-404/tokenizers/wordpiece`
Classic subword tokenizer (used in BERT). Splits words into subword units based on frequency, ensuring full coverage with a compact vocab.
- **Byte-Pair Encoding (BPE) Tokenizer**
Path: `ByteMeHarder-404/tokenizers/bpe`
Uses byte-level BPE, similar to GPT-2 and RoBERTa. Handles any UTF-8 character without unknown tokens by working directly on bytes.
- **XLNet-Style Tokenizer**
Path: `ByteMeHarder-404/tokenizers/xlnet`
Follows the XLNet tokenization approach, leveraging sentencepiece-like segmentation.
## 🚀 Usage
You can load each tokenizer with `transformers`:
```python
from transformers import PreTrainedTokenizerFast
# WordPiece
tok_wordpiece = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/wordpiece")
# BPE
tok_bpe = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/bpe")
# XLNet-style
tok_xlnet = PreTrainedTokenizerFast.from_pretrained("ByteMeHarder-404/tokenizers/xlnet")
```
## 📚 Notes
- These tokenizers are minimal examples and **not pretrained with embeddings or models**.
- Intended for experimentation, educational purposes, and as a foundation for building custom models.
- You can extend them by training a new vocabulary on your dataset.
|