Hugging Face's logo

flexitok
/
mod-tokenizers-individual

Safetensors
Undetermined
llama
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community
gsaltintas commited on
Commit
1d40baf
·
verified ·
1 Parent(s): 6634830

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. README.md +1 -1
  2. vocab.json +12 -12
README.md CHANGED
@@ -46,4 +46,4 @@ tokens = tokenizer.encode("Hello, world!")
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
- | `123450019 mod 67` | `1, 2, 3, 4, 5, 0, 0, 1, 9, , mod, , 6, 7` | `8, 9, 10, 11, 12, 7, 7, 8, 16, 6, 4, 6, 13, 14` |
 
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
+ | `1234500119 mod 67` | `1, 2, 3, 4, 5, 0, 0, 1, 1, 9, , mod, , 6, 7` | `8, 9, 10, 11, 12, 7, 7, 8, 8, 16, 6, 4, 6, 13, 14` |
vocab.json CHANGED
@@ -1,19 +1,19 @@
1
  {
2
- "3": 10,
 
 
3
  "=": 5,
4
- "</s>": 2,
5
- "8": 15,
6
- "5": 12,
7
- "7": 14,
8
- " ": 6,
9
- "<s>": 1,
10
  "<pad>": 3,
11
- "4": 11,
12
- "0": 7,
13
- "mod": 4,
14
  "9": 16,
15
- "<unk>": 0,
16
  "2": 9,
 
 
17
  "6": 13,
18
- "1": 8
 
19
  }
 
1
  {
2
+ "<unk>": 0,
3
+ "mod": 4,
4
+ "4": 11,
5
  "=": 5,
6
+ "1": 8,
 
 
 
 
 
7
  "<pad>": 3,
8
+ "7": 14,
9
+ "5": 12,
10
+ "3": 10,
11
  "9": 16,
12
+ "<s>": 1,
13
  "2": 9,
14
+ "0": 7,
15
+ "8": 15,
16
  "6": 13,
17
+ " ": 6,
18
+ "</s>": 2
19
  }