metadata
language: vi
license: apache-2.0
tags:
- tokenizer
- bpe
- vietnamese
- xnli
- nlp-research
datasets:
- facebook/xnli
NIRVLab — BPE Tokenizer for Vietnamese XNLI
A Byte-Pair Encoding (BPE) tokenizer trained from scratch on the Vietnamese (vi) subset
of the facebook/xnli dataset.
Training Details
| Parameter | Value |
|---|---|
| Algorithm | BPE (ByteLevel) |
| Vocabulary size | 8,000 |
| Min frequency | 2 |
| Special tokens | <s>, <pad>, </s>, <unk>, <mask> |
| Corpus | facebook/xnli / vi — all splits |
| Corpus size | 800,404 sentences |
| Normalizer | NFC Unicode |
| Pre-tokenizer | ByteLevel |
Evaluation Metrics
| Metric | Value |
|---|---|
| Tokens / char | 0.2680 |
| Fertility (tokens / word) | 1.1868 |
| Avg sequence length | 22.40 tokens |
| Vocabulary coverage | 1.0000 |
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-bpe-vi")
tokens = tokenizer("Xin chào thế giới!", return_tensors="pt")