Upload folder using huggingface_hub

README.md

@@ -42,3 +42,8 @@ tokens = tokenizer.encode("Hello, world!")
 - `tokenizer.json` — Full HuggingFace tokenizer
 - `vocab.json` — Vocabulary mapping
 - `merges.txt` — BPE merge rules
+
+## Sample Encoding
+| Text | Tokens | Token IDs |
+|------|--------|-----------|
+| `12345009 mod 67` | `1, 2, 3, 4, 5, 0, 0, 9,  , mod,  , 6, 7` | `8, 9, 10, 11, 12, 7, 7, 16, 6, 4, 6, 13, 14` |

vocab.json

@@ -1,19 +1,19 @@
 {
-  "6": 13,
-  "4": 11,
-  "1": 8,
-  "3": 10,
   "<unk>": 0,
+  "8": 15,
   "2": 9,
+  "1": 8,
   "0": 7,
-  "9": 16,
-  "5": 12,
-  "<pad>": 3,
+  "6": 13,
   "</s>": 2,
-  "7": 14,
-  "=": 5,
   "<s>": 1,
-  " ": 6,
+  "9": 16,
+  "7": 14,
   "mod": 4,
-  "8": 15
+  "<pad>": 3,
+  " ": 6,
+  "=": 5,
+  "5": 12,
+  "3": 10,
+  "4": 11
 }