Upload folder using huggingface_hub

README.md

@@ -20,7 +20,7 @@ A **Byte-Level BPE** tokenizer trained on **numeric** data from Fineweb-2-HQ.
 | Algorithm | Byte-Level BPE |
 | Language | `numeric` |
 | Target Vocab Size | 10,007 |
-| Final Vocab Size | 1,027 |
+| Final Vocab Size | 9,430 |
 | Pre-tokenizer | byte_level |
 | Number handling | ltr_4digit |
 | Contraction handling | False |
@@ -46,4 +46,4 @@ tokens = tokenizer.encode("Hello, world!")
 ## Sample Encoding
 | Text | Tokens | Token IDs |
 |------|--------|-----------|
-| `12345009 mod 67` | `12, 34, 50, 09,  , mod,  , 67` | `18, 35, 47, 116, 6, 4, 6, 53` |
+| `12345009 mod 67` | `1234, 5009,  , mod,  , 67` | `633, 3614, 6, 4, 6, 53` |

tokenizer_config.json

@@ -69,5 +69,6 @@
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "<pad>",
   "tokenizer_class": "PreTrainedTokenizerFast",
-  "unk_token": "<unk>"
-}
+  "unk_token": "<unk>",
+  "number_handling": "ltr_4digit"
+}