File size: 680 Bytes
c77c1e3 b2da60a c77c1e3 b2da60a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # Vietnamese GPT-2 Tokenizer
A ByteLevel BPE tokenizer trained on 400,000 samples from the
nmthien/vietnamese-curated-2m Vietnamese corpus.
## Configuration
- Model: BPE
- Vocabulary size: 32,000
- Pre-tokenizer: ByteLevel
- Decoder: ByteLevel
- Minimum frequency: 2
- Language: Vietnamese
## Purpose
This tokenizer is designed to improve Vietnamese text
tokenization for GPT-2 fine-tuning.
The original GPT-2 tokenizer produces inefficient tokenization
for Vietnamese text because GPT-2 was originally trained primarily
on non-Vietnamese text.
This tokenizer is trained specifically on Vietnamese text.
## Training data
Dataset:
nmthien/vietnamese-curated-2m (2m subset) |