| # Vietnamese GPT-2 Tokenizer |
|
|
| A ByteLevel BPE tokenizer trained on 400,000 samples from the |
| nmthien/vietnamese-curated-2m Vietnamese corpus. |
|
|
| ## Configuration |
|
|
| - Model: BPE |
| - Vocabulary size: 32,000 |
| - Pre-tokenizer: ByteLevel |
| - Decoder: ByteLevel |
| - Minimum frequency: 2 |
| - Language: Vietnamese |
|
|
| ## Purpose |
|
|
| This tokenizer is designed to improve Vietnamese text |
| tokenization for GPT-2 fine-tuning. |
|
|
| The original GPT-2 tokenizer produces inefficient tokenization |
| for Vietnamese text because GPT-2 was originally trained primarily |
| on non-Vietnamese text. |
|
|
| This tokenizer is trained specifically on Vietnamese text. |
|
|
| ## Training data |
|
|
| Dataset: |
| nmthien/vietnamese-curated-2m (2m subset) |
