Update README.md
Browse files
README.md
CHANGED
|
@@ -4,4 +4,6 @@ language:
|
|
| 4 |
library_name: transformers
|
| 5 |
---
|
| 6 |
|
| 7 |
-
based on phi-3 tokenizer, expanded 17291 tokens
|
|
|
|
|
|
|
|
|
| 4 |
library_name: transformers
|
| 5 |
---
|
| 6 |
|
| 7 |
+
based on phi-3 tokenizer, expanded 17291 tokens
|
| 8 |
+
|
| 9 |
+
Following [The Optimal Vocabulary Size Predictor](https://huggingface.co/spaces/sail/scaling-with-vocab-demo), I recommend using this tokenizer with 3-4B model such as phi-3-mini
|