kalixlouiis commited on
Commit
c4320d9
·
verified ·
1 Parent(s): afcaac7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -2
README.md CHANGED
@@ -62,7 +62,7 @@ The primary goal of **myX-Tokenizer** is to optimize the "Token-to-Character" ra
62
  ### Quantitative Evaluation
63
  To ensure an unbiased evaluation, we benchmarked the tokenizers using the **jojo-ai-mst/Myanmar-Agricutlure-1K** dataset (specifically the "Output" column with 1,053 rows). **Note:** This dataset was held out during the training phase of myX-Tokenizer to serve as a clean test set.
64
 
65
- ![Efficiency Benchmark](https://huggingface.co/DatarrX/myX-Tokenizer/blob/main/benchmark.png)
66
 
67
  | Tokenizer | Avg Tokens per Sentence | Efficiency Score (Fertility) |
68
  | :--- | :---: | :---: |
@@ -127,7 +127,17 @@ The tokenizer includes pre-defined IDs for modern AI workflows:
127
  ## Development & Distribution
128
  * **Developed by:** [Khant Sint Heinn (Kalix Louis)](https://huggingface.co/kalixlouiis)
129
  * **Published by:** [DatarrX (Myanmar Open Source NGO)](https://huggingface.co/DatarrX)
130
- - **License**: Apache License 2.0
 
 
 
 
 
 
 
 
 
 
131
 
132
  ## Citation
133
  If you utilize this tokenizer in your research or applications, please use the following citation:
 
62
  ### Quantitative Evaluation
63
  To ensure an unbiased evaluation, we benchmarked the tokenizers using the **jojo-ai-mst/Myanmar-Agricutlure-1K** dataset (specifically the "Output" column with 1,053 rows). **Note:** This dataset was held out during the training phase of myX-Tokenizer to serve as a clean test set.
64
 
65
+ ![Efficiency Benchmark](benchmark.png)
66
 
67
  | Tokenizer | Avg Tokens per Sentence | Efficiency Score (Fertility) |
68
  | :--- | :---: | :---: |
 
127
  ## Development & Distribution
128
  * **Developed by:** [Khant Sint Heinn (Kalix Louis)](https://huggingface.co/kalixlouiis)
129
  * **Published by:** [DatarrX (Myanmar Open Source NGO)](https://huggingface.co/DatarrX)
130
+ * **License**: Apache License 2.0
131
+ * **Training Datasets**:
132
+ - kalixlouiis/HFcourse-english-burmese-parallel-corpus
133
+ - kalixlouiis/Myanmar-English-general-text-translation
134
+ - kalixlouiis/myanmar-written-spoken-text-pairs
135
+ - kalixlouiis/myanmar-linguistic-ambiguitie-001
136
+ - kalixlouiis/general-burmese-sentences
137
+ - DatarrX/Burmese-English-Code-Mixed-Corpus
138
+ - DatarrX/tipitaka-dataset
139
+ - agentlans/high-quality-english-sentences
140
+ - DatarrX/myX-Burmese-Morpho-Synthetic
141
 
142
  ## Citation
143
  If you utilize this tokenizer in your research or applications, please use the following citation: