# Vietnamese GPT-2 Tokenizer

A ByteLevel BPE tokenizer trained on 400,000 samples from the
nmthien/vietnamese-curated-2m Vietnamese corpus.

## Configuration

- Model: BPE
- Vocabulary size: 32,000
- Pre-tokenizer: ByteLevel
- Decoder: ByteLevel
- Minimum frequency: 2
- Language: Vietnamese

## Purpose

This tokenizer is designed to improve Vietnamese text
tokenization for GPT-2 fine-tuning.

The original GPT-2 tokenizer produces inefficient tokenization
for Vietnamese text because GPT-2 was originally trained primarily
on non-Vietnamese text.

This tokenizer is trained specifically on Vietnamese text.

## Training data

Dataset:
nmthien/vietnamese-curated-2m (2m subset)