| language: | |
| - ko | |
| library_name: transformers | |
| based on phi-3 tokenizer, expanded 17291 tokens | |
| Following [The Optimal Vocabulary Size Predictor](https://huggingface.co/spaces/sail/scaling-with-vocab-demo), I recommend using this tokenizer with 3-4B model such as phi-3-mini |