Update README.md
Browse files
README.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
| 9 |
|
| 10 |
year = {2025}}
|
| 11 |
|
| 12 |
-
## Qwen tokenizer trained on Irish language data
|
| 13 |
- Provides a ~50% reduction in number of tokens. (399 → 200 in test set).
|
| 14 |
- Significantly improves identifying words as tokens.
|
| 15 |
|
|
|
|
| 9 |
|
| 10 |
year = {2025}}
|
| 11 |
|
| 12 |
+
## Monolingual Qwen tokenizer trained on Irish language data
|
| 13 |
- Provides a ~50% reduction in number of tokens. (399 → 200 in test set).
|
| 14 |
- Significantly improves identifying words as tokens.
|
| 15 |
|