Update README.md
Browse files
README.md
CHANGED
|
@@ -62,7 +62,7 @@ The primary goal of **myX-Tokenizer** is to optimize the "Token-to-Character" ra
|
|
| 62 |
### Quantitative Evaluation
|
| 63 |
To ensure an unbiased evaluation, we benchmarked the tokenizers using the **jojo-ai-mst/Myanmar-Agricutlure-1K** dataset (specifically the "Output" column with 1,053 rows). **Note:** This dataset was held out during the training phase of myX-Tokenizer to serve as a clean test set.
|
| 64 |
|
| 65 |
-
 |
|
| 68 |
| :--- | :---: | :---: |
|
|
@@ -127,7 +127,17 @@ The tokenizer includes pre-defined IDs for modern AI workflows:
|
|
| 127 |
## Development & Distribution
|
| 128 |
* **Developed by:** [Khant Sint Heinn (Kalix Louis)](https://huggingface.co/kalixlouiis)
|
| 129 |
* **Published by:** [DatarrX (Myanmar Open Source NGO)](https://huggingface.co/DatarrX)
|
| 130 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
## Citation
|
| 133 |
If you utilize this tokenizer in your research or applications, please use the following citation:
|
|
|
|
| 62 |
### Quantitative Evaluation
|
| 63 |
To ensure an unbiased evaluation, we benchmarked the tokenizers using the **jojo-ai-mst/Myanmar-Agricutlure-1K** dataset (specifically the "Output" column with 1,053 rows). **Note:** This dataset was held out during the training phase of myX-Tokenizer to serve as a clean test set.
|
| 64 |
|
| 65 |
+

|
| 66 |
|
| 67 |
| Tokenizer | Avg Tokens per Sentence | Efficiency Score (Fertility) |
|
| 68 |
| :--- | :---: | :---: |
|
|
|
|
| 127 |
## Development & Distribution
|
| 128 |
* **Developed by:** [Khant Sint Heinn (Kalix Louis)](https://huggingface.co/kalixlouiis)
|
| 129 |
* **Published by:** [DatarrX (Myanmar Open Source NGO)](https://huggingface.co/DatarrX)
|
| 130 |
+
* **License**: Apache License 2.0
|
| 131 |
+
* **Training Datasets**:
|
| 132 |
+
- kalixlouiis/HFcourse-english-burmese-parallel-corpus
|
| 133 |
+
- kalixlouiis/Myanmar-English-general-text-translation
|
| 134 |
+
- kalixlouiis/myanmar-written-spoken-text-pairs
|
| 135 |
+
- kalixlouiis/myanmar-linguistic-ambiguitie-001
|
| 136 |
+
- kalixlouiis/general-burmese-sentences
|
| 137 |
+
- DatarrX/Burmese-English-Code-Mixed-Corpus
|
| 138 |
+
- DatarrX/tipitaka-dataset
|
| 139 |
+
- agentlans/high-quality-english-sentences
|
| 140 |
+
- DatarrX/myX-Burmese-Morpho-Synthetic
|
| 141 |
|
| 142 |
## Citation
|
| 143 |
If you utilize this tokenizer in your research or applications, please use the following citation:
|