tokenizers
Collection
trained and adapted tokenizers - various • 18 items • Updated
BEE-spoke-data/MiniTokenizer-20480
This is a ByteLevelBPETokenizer trained on the JeanKaddour/minipile dataset with the aim to create a compact English-only tokenizer.
Usage
load with AutoTokenizer, i.e.:
from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support