README.md

Vietnamese GPT-2 Tokenizer

A ByteLevel BPE tokenizer trained on 400,000 samples from the nmthien/vietnamese-curated-2m Vietnamese corpus.

Configuration

• Model: BPE
• Vocabulary size: 32,000
• Pre-tokenizer: ByteLevel
• Decoder: ByteLevel
• Minimum frequency: 2
• Language: Vietnamese

Purpose

This tokenizer is designed to improve Vietnamese text tokenization for GPT-2 fine-tuning.

The original GPT-2 tokenizer produces inefficient tokenization for Vietnamese text because GPT-2 was originally trained primarily on non-Vietnamese text.

This tokenizer is trained specifically on Vietnamese text.

Training data

Dataset: nmthien/vietnamese-curated-2m (2m subset)