File size: 278 Bytes
19d30ed 242a90c 01abd33 19d30ed d81eb24 | 1 2 3 4 5 6 7 8 9 | ---
language:
- ko
library_name: transformers
---
based on phi-3 tokenizer, expanded 17291 tokens
Following [The Optimal Vocabulary Size Predictor](https://huggingface.co/spaces/sail/scaling-with-vocab-demo), I recommend using this tokenizer with 3-4B model such as phi-3-mini |