---
language:
- ko
library_name: transformers
---

based on phi-3 tokenizer, expanded 17291 tokens

Following [The Optimal Vocabulary Size Predictor](https://huggingface.co/spaces/sail/scaling-with-vocab-demo), I recommend using this tokenizer with 3-4B model such as phi-3-mini