Add arXiv:2608.26449 (paper link, HF paper tag, BibTeX)
Browse files
README.md
CHANGED
|
@@ -10,6 +10,7 @@ tags:
|
|
| 10 |
- nepali
|
| 11 |
- devanagari
|
| 12 |
- low-resource
|
|
|
|
| 13 |
---
|
| 14 |
|
| 15 |
# Arkios Tokenizer — Nepali + English, 65,536 tokens
|
|
@@ -54,7 +55,7 @@ this is not only a compression result.
|
|
| 54 |
|
| 55 |
Code, harness and full results: **https://github.com/sajalregmi/arkios-tokenizer**
|
| 56 |
Paper: *Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual
|
| 57 |
-
Tokenizer Fertility*
|
| 58 |
|
| 59 |
## Measured performance
|
| 60 |
|
|
@@ -168,12 +169,16 @@ revision of every baseline and the sha256 of every corpus.
|
|
| 168 |
## Citation
|
| 169 |
|
| 170 |
```bibtex
|
| 171 |
-
@
|
| 172 |
-
title
|
| 173 |
-
|
| 174 |
-
author
|
| 175 |
-
year
|
| 176 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 177 |
}
|
| 178 |
```
|
| 179 |
|
|
|
|
| 10 |
- nepali
|
| 11 |
- devanagari
|
| 12 |
- low-resource
|
| 13 |
+
- arxiv:2608.26449
|
| 14 |
---
|
| 15 |
|
| 16 |
# Arkios Tokenizer — Nepali + English, 65,536 tokens
|
|
|
|
| 55 |
|
| 56 |
Code, harness and full results: **https://github.com/sajalregmi/arkios-tokenizer**
|
| 57 |
Paper: *Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual
|
| 58 |
+
Tokenizer Fertility* — **[arXiv:2608.26449](https://arxiv.org/abs/2608.26449)**
|
| 59 |
|
| 60 |
## Measured performance
|
| 61 |
|
|
|
|
| 169 |
## Citation
|
| 170 |
|
| 171 |
```bibtex
|
| 172 |
+
@misc{regmi2026vowelsigns,
|
| 173 |
+
title = {Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on
|
| 174 |
+
Multilingual Tokenizer Fertility},
|
| 175 |
+
author = {Regmi, Sajal and Pudasaini, Siddhartha and Pun, Chetan Phakami},
|
| 176 |
+
year = {2026},
|
| 177 |
+
eprint = {2608.26449},
|
| 178 |
+
archivePrefix = {arXiv},
|
| 179 |
+
primaryClass = {cs.CL},
|
| 180 |
+
doi = {10.48550/arXiv.2608.26449},
|
| 181 |
+
url = {https://arxiv.org/abs/2608.26449}
|
| 182 |
}
|
| 183 |
```
|
| 184 |
|