README.md

---
license: apache-2.0
datasets:
- VTSNLP/vietnamese_curated_dataset
language:
- vi
tags:
- tokenizer
- vietnamese
- byte-bpe
- causal-lm
- nlp
---
# Vietnamese Tokenizer

This repository contains a **ByteLevel BPE tokenizer** trained **from scratch** specifically for the **Vietnamese language**, designed for **decoder-only language model pretraining**.

---

## 🚀 Usage

### Load tokenizer

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "tranhuyHoang/mini_VN_decoder_tokenizer",
    use_fast=True
)