sajalregmi4 commited on
Commit
aef562f
·
verified ·
1 Parent(s): eb795f4

Add arXiv:2608.26449 (paper link, HF paper tag, BibTeX)

Browse files
Files changed (1) hide show
  1. README.md +12 -7
README.md CHANGED
@@ -10,6 +10,7 @@ tags:
10
  - nepali
11
  - devanagari
12
  - low-resource
 
13
  ---
14
 
15
  # Arkios Tokenizer — Nepali + English, 65,536 tokens
@@ -54,7 +55,7 @@ this is not only a compression result.
54
 
55
  Code, harness and full results: **https://github.com/sajalregmi/arkios-tokenizer**
56
  Paper: *Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual
57
- Tokenizer Fertility* (arXiv link added on posting).
58
 
59
  ## Measured performance
60
 
@@ -168,12 +169,16 @@ revision of every baseline and the sha256 of every corpus.
168
  ## Citation
169
 
170
  ```bibtex
171
- @article{regmi2026vowelsigns,
172
- title = {Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on
173
- Multilingual Tokenizer Fertility},
174
- author = {Regmi, Sajal and Pudasaini, Siddhartha and Pun, Chetan Phakami},
175
- year = {2026},
176
- note = {arXiv preprint; identifier added on posting}
 
 
 
 
177
  }
178
  ```
179
 
 
10
  - nepali
11
  - devanagari
12
  - low-resource
13
+ - arxiv:2608.26449
14
  ---
15
 
16
  # Arkios Tokenizer — Nepali + English, 65,536 tokens
 
55
 
56
  Code, harness and full results: **https://github.com/sajalregmi/arkios-tokenizer**
57
  Paper: *Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual
58
+ Tokenizer Fertility* **[arXiv:2608.26449](https://arxiv.org/abs/2608.26449)**
59
 
60
  ## Measured performance
61
 
 
169
  ## Citation
170
 
171
  ```bibtex
172
+ @misc{regmi2026vowelsigns,
173
+ title = {Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on
174
+ Multilingual Tokenizer Fertility},
175
+ author = {Regmi, Sajal and Pudasaini, Siddhartha and Pun, Chetan Phakami},
176
+ year = {2026},
177
+ eprint = {2608.26449},
178
+ archivePrefix = {arXiv},
179
+ primaryClass = {cs.CL},
180
+ doi = {10.48550/arXiv.2608.26449},
181
+ url = {https://arxiv.org/abs/2608.26449}
182
  }
183
  ```
184