Upload folder using huggingface_hub

README.md

@@ -21,9 +21,9 @@ A **Byte-Level BPE** tokenizer trained on **['ell_Grek']** data from Fineweb-2-H
 | Algorithm | Byte-Level BPE |
 | Language | `['ell_Grek']` |
 | Target Vocab Size | 2,000 |
-| Final Vocab Size | 2,000 |
+| Final Vocab Size | 383 |
 | Pre-tokenizer | custom:addition |
-| Number handling | learned |
+| Number handling | ltr_3digit |
 | Contraction handling | False |
 | Normalizer | NFC |
 | Special Tokens | `<s>`, `</s>`, `<pad>`, `<unk>` |

tokenizer_config.json

@@ -53,5 +53,5 @@
   "pad_token": "<pad>",
   "tokenizer_class": "PreTrainedTokenizerFast",
   "unk_token": null,
-  "number_handling": "learned"
+  "number_handling": "ltr_3digit"
 }