Hugging Face's logo

imgremlin
/
deepsmiles-tokenizer-bart-512

Model card Files
xet
Community
imgremlin commited on
Commit
48a019b
·
1 Parent(s): 504c9f4

Upload tokenizer

Browse files
Files changed (5) hide show
  1. merges.txt +252 -0
  2. special_tokens_map.json +15 -0
  3. tokenizer.json +854 -0
  4. tokenizer_config.json +22 -0
  5. vocab.json +1 -0
merges.txt ADDED
@@ -0,0 +1,252 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #version: 0.2
2
+ ) )
3
+ c c
4
+ C C
5
+ )) ))
6
+ cc cc
7
+ CC CC
8
+ C O
9
+ )))) ))))
10
+ CC C
11
+ cccc cc
12
+ cc c
13
+ )) )
14
+ N C
15
+ O C
16
+ 1 0
17
+ n c
18
+ CC O
19
+ C l
20
+ ) =
21
+ N CC
22
+ )))) ))
23
+ CCCC C
24
+ )))) )))
25
+ n H
26
+ cccc c
27
+ CCCC CCCC
28
+ O CC
29
+ C N
30
+ CO C
31
+ c n
32
+ B r
33
+ )))))))) )
34
+ C cccc
35
+ ) [
36
+ )))) )
37
+ CO cccc
38
+ )))))))) ))
39
+ C NC
40
+ CO CCO
41
+ CCCC CC
42
+ + ]
43
+ - ]
44
+ c C
45
+ c nc
46
+ cc O
47
+ o c
48
+ 1 4
49
+ CC OC
50
+ cc C
51
+ )))))))) )))
52
+ CO ccc
53
+ cccc O
54
+ 10 6
55
+ C ccc
56
+ c O
57
+ CC N
58
+ C cccccc
59
+ +] =
60
+ N cccc
61
+ CO CC
62
+ 6 9
63
+ c OC
64
+ 9 6
65
+ N CCCC
66
+ N CCC
67
+ CC NC
68
+ n cc
69
+ cccc C
70
+ -] ))
71
+ C S
72
+ CCCC CCC
73
+ C F
74
+ C NCC
75
+ C cc
76
+ N cccccc
77
+ CC CO
78
+ ccc O
79
+ cccc Cl
80
+ 1 3
81
+ CO cccccc
82
+ N CCO
83
+ )))))))) ))))
84
+ CCCC CCCCC
85
+ s c
86
+ 1 7
87
+ 9 5
88
+ n n
89
+ CC NCC
90
+ CO ccccc
91
+ O CCO
92
+ c Cl
93
+ CC l
94
+ C ccccc
95
+ cc OC
96
+ C cn
97
+ ccc C
98
+ n ccC
99
+ COcccc C
100
+ CO P
101
+ O COCCO
102
+ CCC N
103
+ O CO
104
+ )))))))) )))))
105
+ cccc F
106
+ 1 1
107
+ S C
108
+ CO COCCO
109
+ cccc OC
110
+ ccc Cl
111
+ -] )
112
+ cc n
113
+ c N
114
+ )))))))) ))))))))
115
+ nc nc
116
+ cccc Br
117
+ cccc nc
118
+ O CCC
119
+ N S
120
+ CC cccccc
121
+ Ccccc O
122
+ 1 5
123
+ CCC NC
124
+ cccc N
125
+ )))))))) )))))))
126
+ CCCCCCCC CCCCCCCC
127
+ n C
128
+ c o
129
+ ccc OC
130
+ COccc C
131
+ cc Cl
132
+ CCCC N
133
+ nc C
134
+ Ccccc C
135
+ nc N
136
+ Cccc C
137
+ ccccc O
138
+ COccc O
139
+ c s
140
+ CCCC O
141
+ C cnc
142
+ ccc Br
143
+ N CCCCC
144
+ CC S
145
+ NCC cc
146
+ cccc cccccc
147
+ cc nc
148
+ NCC NCC
149
+ 1 2
150
+ n cccccc
151
+ NCC cccccc
152
+ C nc
153
+ cccc NC
154
+ ccccc n
155
+ N cn
156
+ NC cccccc
157
+ c CC
158
+ c F
159
+ o cccO
160
+ CCO COCCO
161
+ ccccc Cl
162
+ Ncccc Cl
163
+ F CF
164
+ N N
165
+ ccc F
166
+ cn cccccc
167
+ CCO P
168
+ Ccccc NC
169
+ O CCCC
170
+ n nc
171
+ CCO CC
172
+ ccccc C
173
+ OCC OC
174
+ NCCCC N
175
+ N cnc
176
+ cn n
177
+ )) =
178
+ CCCCC nccC
179
+ CC COC
180
+ 1 6
181
+ C c
182
+ )))))))) ))))))
183
+ ccc ncc
184
+ Ccccc S
185
+ cc Br
186
+ CCC NCC
187
+ NCC S
188
+ cccc cccc
189
+ 1 8
190
+ Ncccc C
191
+ O COCC
192
+ CO cc
193
+ N CCCCCC
194
+ )))))))) )))))))))
195
+ NCCO CC
196
+ O cccccc
197
+ c Br
198
+ cc N
199
+ CO ccO
200
+ CCC cccccc
201
+ CCCCCCCC CCCCCC
202
+ +] [
203
+ N O
204
+ CN CCCC
205
+ CN CCC
206
+ CO COCC
207
+ COccc OC
208
+ OCC COP
209
+ CCCC NC
210
+ Cl cccc
211
+ O cccc
212
+ cc F
213
+ nc Cl
214
+ cc CC
215
+ CC ccccO
216
+ CCO CCO
217
+ OC cccccc
218
+ CCCCCCCC CCCCCCC
219
+ S CC
220
+ NCC ccccO
221
+ Ccccc Cl
222
+ C ccO
223
+ c sc
224
+ nc NC
225
+ cnc Cl
226
+ n ccc
227
+ CCCC CCO
228
+ 2 0
229
+ S N
230
+ c oc
231
+ )))))))) ))))))))))
232
+ NCC NC
233
+ CCCCCCCC CC
234
+ N NC
235
+ cccc s
236
+ Cccc O
237
+ C ccC
238
+ F cccc
239
+ ccccc Br
240
+ 13 6
241
+ cnc nc
242
+ N ccc
243
+ CC cc
244
+ cccc ccccc
245
+ CCC l
246
+ Cccccc C
247
+ 2 2
248
+ CCCC CO
249
+ ccc oc
250
+ nc n
251
+ NCC N
252
+ ccC F
special_tokens_map.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "cls_token": "<s>",
4
+ "eos_token": "</s>",
5
+ "mask_token": {
6
+ "content": "<mask>",
7
+ "lstrip": true,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "pad_token": "<pad>",
13
+ "sep_token": "</s>",
14
+ "unk_token": "<unk>"
15
+ }
tokenizer.json ADDED
@@ -0,0 +1,854 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "1.0",
3
+ "truncation": null,
4
+ "padding": null,
5
+ "added_tokens": [
6
+ {
7
+ "id": 0,
8
+ "content": "<s>",
9
+ "single_word": false,
10
+ "lstrip": false,
11
+ "rstrip": false,
12
+ "normalized": false,
13
+ "special": true
14
+ },
15
+ {
16
+ "id": 1,
17
+ "content": "<pad>",
18
+ "single_word": false,
19
+ "lstrip": false,
20
+ "rstrip": false,
21
+ "normalized": false,
22
+ "special": true
23
+ },
24
+ {
25
+ "id": 2,
26
+ "content": "</s>",
27
+ "single_word": false,
28
+ "lstrip": false,
29
+ "rstrip": false,
30
+ "normalized": false,
31
+ "special": true
32
+ },
33
+ {
34
+ "id": 3,
35
+ "content": "<unk>",
36
+ "single_word": false,
37
+ "lstrip": false,
38
+ "rstrip": false,
39
+ "normalized": false,
40
+ "special": true
41
+ },
42
+ {
43
+ "id": 4,
44
+ "content": "<mask>",
45
+ "single_word": false,
46
+ "lstrip": true,
47
+ "rstrip": false,
48
+ "normalized": false,
49
+ "special": true
50
+ }
51
+ ],
52
+ "normalizer": null,
53
+ "pre_tokenizer": {
54
+ "type": "ByteLevel",
55
+ "add_prefix_space": false,
56
+ "trim_offsets": true,
57
+ "use_regex": true
58
+ },
59
+ "post_processor": {
60
+ "type": "RobertaProcessing",
61
+ "sep": [
62
+ "</s>",
63
+ 2
64
+ ],
65
+ "cls": [
66
+ "<s>",
67
+ 0
68
+ ],
69
+ "trim_offsets": true,
70
+ "add_prefix_space": false
71
+ },
72
+ "decoder": {
73
+ "type": "ByteLevel",
74
+ "add_prefix_space": true,
75
+ "trim_offsets": true,
76
+ "use_regex": true
77
+ },
78
+ "model": {
79
+ "type": "BPE",
80
+ "dropout": null,
81
+ "unk_token": null,
82
+ "continuing_subword_prefix": "",
83
+ "end_of_word_suffix": "",
84
+ "fuse_unk": false,
85
+ "byte_fallback": false,
86
+ "vocab": {
87
+ "<s>": 0,
88
+ "<pad>": 1,
89
+ "</s>": 2,
90
+ "<unk>": 3,
91
+ "<mask>": 4,
92
+ "!": 5,
93
+ "\"": 6,
94
+ "#": 7,
95
+ "$": 8,
96
+ "%": 9,
97
+ "&": 10,
98
+ "'": 11,
99
+ "(": 12,
100
+ ")": 13,
101
+ "*": 14,
102
+ "+": 15,
103
+ ",": 16,
104
+ "-": 17,
105
+ ".": 18,
106
+ "/": 19,
107
+ "0": 20,
108
+ "1": 21,
109
+ "2": 22,
110
+ "3": 23,
111
+ "4": 24,
112
+ "5": 25,
113
+ "6": 26,
114
+ "7": 27,
115
+ "8": 28,
116
+ "9": 29,
117
+ ":": 30,
118
+ ";": 31,
119
+ "<": 32,
120
+ "=": 33,
121
+ ">": 34,
122
+ "?": 35,
123
+ "@": 36,
124
+ "A": 37,
125
+ "B": 38,
126
+ "C": 39,
127
+ "D": 40,
128
+ "E": 41,
129
+ "F": 42,
130
+ "G": 43,
131
+ "H": 44,
132
+ "I": 45,
133
+ "J": 46,
134
+ "K": 47,
135
+ "L": 48,
136
+ "M": 49,
137
+ "N": 50,
138
+ "O": 51,
139
+ "P": 52,
140
+ "Q": 53,
141
+ "R": 54,
142
+ "S": 55,
143
+ "T": 56,
144
+ "U": 57,
145
+ "V": 58,
146
+ "W": 59,
147
+ "X": 60,
148
+ "Y": 61,
149
+ "Z": 62,
150
+ "[": 63,
151
+ "\\": 64,
152
+ "]": 65,
153
+ "^": 66,
154
+ "_": 67,
155
+ "`": 68,
156
+ "a": 69,
157
+ "b": 70,
158
+ "c": 71,
159
+ "d": 72,
160
+ "e": 73,
161
+ "f": 74,
162
+ "g": 75,
163
+ "h": 76,
164
+ "i": 77,
165
+ "j": 78,
166
+ "k": 79,
167
+ "l": 80,
168
+ "m": 81,
169
+ "n": 82,
170
+ "o": 83,
171
+ "p": 84,
172
+ "q": 85,
173
+ "r": 86,
174
+ "s": 87,
175
+ "t": 88,
176
+ "u": 89,
177
+ "v": 90,
178
+ "w": 91,
179
+ "x": 92,
180
+ "y": 93,
181
+ "z": 94,
182
+ "{": 95,
183
+ "|": 96,
184
+ "}": 97,
185
+ "~": 98,
186
+ "¡": 99,
187
+ "¢": 100,
188
+ "£": 101,
189
+ "¤": 102,
190
+ "¥": 103,
191
+ "¦": 104,
192
+ "§": 105,
193
+ "¨": 106,
194
+ "©": 107,
195
+ "ª": 108,
196
+ "«": 109,
197
+ "¬": 110,
198
+ "®": 111,
199
+ "¯": 112,
200
+ "°": 113,
201
+ "±": 114,
202
+ "²": 115,
203
+ "³": 116,
204
+ "´": 117,
205
+ "µ": 118,
206
+ "¶": 119,
207
+ "·": 120,
208
+ "¸": 121,
209
+ "¹": 122,
210
+ "º": 123,
211
+ "»": 124,
212
+ "¼": 125,
213
+ "½": 126,
214
+ "¾": 127,
215
+ "¿": 128,
216
+ "À": 129,
217
+ "Á": 130,
218
+ "Â": 131,
219
+ "Ã": 132,
220
+ "Ä": 133,
221
+ "Å": 134,
222
+ "Æ": 135,
223
+ "Ç": 136,
224
+ "È": 137,
225
+ "É": 138,
226
+ "Ê": 139,
227
+ "Ë": 140,
228
+ "Ì": 141,
229
+ "Í": 142,
230
+ "Î": 143,
231
+ "Ï": 144,
232
+ "Ð": 145,
233
+ "Ñ": 146,
234
+ "Ò": 147,
235
+ "Ó": 148,
236
+ "Ô": 149,
237
+ "Õ": 150,
238
+ "Ö": 151,
239
+ "×": 152,
240
+ "Ø": 153,
241
+ "Ù": 154,
242
+ "Ú": 155,
243
+ "Û": 156,
244
+ "Ü": 157,
245
+ "Ý": 158,
246
+ "Þ": 159,
247
+ "ß": 160,
248
+ "à": 161,
249
+ "á": 162,
250
+ "â": 163,
251
+ "ã": 164,
252
+ "ä": 165,
253
+ "å": 166,
254
+ "æ": 167,
255
+ "ç": 168,
256
+ "è": 169,
257
+ "é": 170,
258
+ "ê": 171,
259
+ "ë": 172,
260
+ "ì": 173,
261
+ "í": 174,
262
+ "î": 175,
263
+ "ï": 176,
264
+ "ð": 177,
265
+ "ñ": 178,
266
+ "ò": 179,
267
+ "ó": 180,
268
+ "ô": 181,
269
+ "õ": 182,
270
+ "ö": 183,
271
+ "÷": 184,
272
+ "ø": 185,
273
+ "ù": 186,
274
+ "ú": 187,
275
+ "û": 188,
276
+ "ü": 189,
277
+ "ý": 190,
278
+ "þ": 191,
279
+ "ÿ": 192,
280
+ "Ā": 193,
281
+ "ā": 194,
282
+ "Ă": 195,
283
+ "ă": 196,
284
+ "Ą": 197,
285
+ "ą": 198,
286
+ "Ć": 199,
287
+ "ć": 200,
288
+ "Ĉ": 201,
289
+ "ĉ": 202,
290
+ "Ċ": 203,
291
+ "ċ": 204,
292
+ "Č": 205,
293
+ "č": 206,
294
+ "Ď": 207,
295
+ "ď": 208,
296
+ "Đ": 209,
297
+ "đ": 210,
298
+ "Ē": 211,
299
+ "ē": 212,
300
+ "Ĕ": 213,
301
+ "ĕ": 214,
302
+ "Ė": 215,
303
+ "ė": 216,
304
+ "Ę": 217,
305
+ "ę": 218,
306
+ "Ě": 219,
307
+ "ě": 220,
308
+ "Ĝ": 221,
309
+ "ĝ": 222,
310
+ "Ğ": 223,
311
+ "ğ": 224,
312
+ "Ġ": 225,
313
+ "ġ": 226,
314
+ "Ģ": 227,
315
+ "ģ": 228,
316
+ "Ĥ": 229,
317
+ "ĥ": 230,
318
+ "Ħ": 231,
319
+ "ħ": 232,
320
+ "Ĩ": 233,
321
+ "ĩ": 234,
322
+ "Ī": 235,
323
+ "ī": 236,
324
+ "Ĭ": 237,
325
+ "ĭ": 238,
326
+ "Į": 239,
327
+ "į": 240,
328
+ "İ": 241,
329
+ "ı": 242,
330
+ "IJ": 243,
331
+ "ij": 244,
332
+ "Ĵ": 245,
333
+ "ĵ": 246,
334
+ "Ķ": 247,
335
+ "ķ": 248,
336
+ "ĸ": 249,
337
+ "Ĺ": 250,
338
+ "ĺ": 251,
339
+ "Ļ": 252,
340
+ "ļ": 253,
341
+ "Ľ": 254,
342
+ "ľ": 255,
343
+ "Ŀ": 256,
344
+ "ŀ": 257,
345
+ "Ł": 258,
346
+ "ł": 259,
347
+ "Ń": 260,
348
+ "))": 261,
349
+ "cc": 262,
350
+ "CC": 263,
351
+ "))))": 264,
352
+ "cccc": 265,
353
+ "CCCC": 266,
354
+ "CO": 267,
355
+ "))))))))": 268,
356
+ "CCC": 269,
357
+ "cccccc": 270,
358
+ "ccc": 271,
359
+ ")))": 272,
360
+ "NC": 273,
361
+ "OC": 274,
362
+ "10": 275,
363
+ "nc": 276,
364
+ "CCO": 277,
365
+ "Cl": 278,
366
+ ")=": 279,
367
+ "NCC": 280,
368
+ "))))))": 281,
369
+ "CCCCC": 282,
370
+ ")))))))": 283,
371
+ "nH": 284,
372
+ "ccccc": 285,
373
+ "CCCCCCCC": 286,
374
+ "OCC": 287,
375
+ "CN": 288,
376
+ "COC": 289,
377
+ "cn": 290,
378
+ "Br": 291,
379
+ ")))))))))": 292,
380
+ "Ccccc": 293,
381
+ ")[": 294,
382
+ ")))))": 295,
383
+ "COcccc": 296,
384
+ "))))))))))": 297,
385
+ "CNC": 298,
386
+ "COCCO": 299,
387
+ "CCCCCC": 300,
388
+ "+]": 301,
389
+ "-]": 302,
390
+ "cC": 303,
391
+ "cnc": 304,
392
+ "ccO": 305,
393
+ "oc": 306,
394
+ "14": 307,
395
+ "CCOC": 308,
396
+ "ccC": 309,
397
+ ")))))))))))": 310,
398
+ "COccc": 311,
399
+ "ccccO": 312,
400
+ "106": 313,
401
+ "Cccc": 314,
402
+ "cO": 315,
403
+ "CCN": 316,
404
+ "Ccccccc": 317,
405
+ "+]=": 318,
406
+ "Ncccc": 319,
407
+ "COCC": 320,
408
+ "69": 321,
409
+ "cOC": 322,
410
+ "96": 323,
411
+ "NCCCC": 324,
412
+ "NCCC": 325,
413
+ "CCNC": 326,
414
+ "ncc": 327,
415
+ "ccccC": 328,
416
+ "-]))": 329,
417
+ "CS": 330,
418
+ "CCCCCCC": 331,
419
+ "CF": 332,
420
+ "CNCC": 333,
421
+ "Ccc": 334,
422
+ "Ncccccc": 335,
423
+ "CCCO": 336,
424
+ "cccO": 337,
425
+ "ccccCl": 338,
426
+ "13": 339,
427
+ "COcccccc": 340,
428
+ "NCCO": 341,
429
+ "))))))))))))": 342,
430
+ "CCCCCCCCC": 343,
431
+ "sc": 344,
432
+ "17": 345,
433
+ "95": 346,
434
+ "nn": 347,
435
+ "CCNCC": 348,
436
+ "COccccc": 349,
437
+ "OCCO": 350,
438
+ "cCl": 351,
439
+ "CCl": 352,
440
+ "Cccccc": 353,
441
+ "ccOC": 354,
442
+ "Ccn": 355,
443
+ "cccC": 356,
444
+ "nccC": 357,
445
+ "COccccC": 358,
446
+ "COP": 359,
447
+ "OCOCCO": 360,
448
+ "CCCN": 361,
449
+ "OCO": 362,
450
+ ")))))))))))))": 363,
451
+ "ccccF": 364,
452
+ "11": 365,
453
+ "SC": 366,
454
+ "COCOCCO": 367,
455
+ "ccccOC": 368,
456
+ "cccCl": 369,
457
+ "-])": 370,
458
+ "ccn": 371,
459
+ "cN": 372,
460
+ "))))))))))))))))": 373,
461
+ "ncnc": 374,
462
+ "ccccBr": 375,
463
+ "ccccnc": 376,
464
+ "OCCC": 377,
465
+ "NS": 378,
466
+ "CCcccccc": 379,
467
+ "CccccO": 380,
468
+ "15": 381,
469
+ "CCCNC": 382,
470
+ "ccccN": 383,
471
+ ")))))))))))))))": 384,
472
+ "CCCCCCCCCCCCCCCC": 385,
473
+ "nC": 386,
474
+ "co": 387,
475
+ "cccOC": 388,
476
+ "COcccC": 389,
477
+ "ccCl": 390,
478
+ "CCCCN": 391,
479
+ "ncC": 392,
480
+ "CccccC": 393,
481
+ "ncN": 394,
482
+ "CcccC": 395,
483
+ "cccccO": 396,
484
+ "COcccO": 397,
485
+ "cs": 398,
486
+ "CCCCO": 399,
487
+ "Ccnc": 400,
488
+ "cccBr": 401,
489
+ "NCCCCC": 402,
490
+ "CCS": 403,
491
+ "NCCcc": 404,
492
+ "cccccccccc": 405,
493
+ "ccnc": 406,
494
+ "NCCNCC": 407,
495
+ "12": 408,
496
+ "ncccccc": 409,
497
+ "NCCcccccc": 410,
498
+ "Cnc": 411,
499
+ "ccccNC": 412,
500
+ "cccccn": 413,
501
+ "Ncn": 414,
502
+ "NCcccccc": 415,
503
+ "cCC": 416,
504
+ "cF": 417,
505
+ "occcO": 418,
506
+ "CCOCOCCO": 419,
507
+ "cccccCl": 420,
508
+ "NccccCl": 421,
509
+ "FCF": 422,
510
+ "NN": 423,
511
+ "cccF": 424,
512
+ "cncccccc": 425,
513
+ "CCOP": 426,
514
+ "CccccNC": 427,
515
+ "OCCCC": 428,
516
+ "nnc": 429,
517
+ "CCOCC": 430,
518
+ "cccccC": 431,
519
+ "OCCOC": 432,
520
+ "NCCCCN": 433,
521
+ "Ncnc": 434,
522
+ "cnn": 435,
523
+ "))=": 436,
524
+ "CCCCCnccC": 437,
525
+ "CCCOC": 438,
526
+ "16": 439,
527
+ "Cc": 440,
528
+ "))))))))))))))": 441,
529
+ "cccncc": 442,
530
+ "CccccS": 443,
531
+ "ccBr": 444,
532
+ "CCCNCC": 445,
533
+ "NCCS": 446,
534
+ "cccccccc": 447,
535
+ "18": 448,
536
+ "NccccC": 449,
537
+ "OCOCC": 450,
538
+ "COcc": 451,
539
+ "NCCCCCC": 452,
540
+ ")))))))))))))))))": 453,
541
+ "NCCOCC": 454,
542
+ "Occcccc": 455,
543
+ "cBr": 456,
544
+ "ccN": 457,
545
+ "COccO": 458,
546
+ "CCCcccccc": 459,
547
+ "CCCCCCCCCCCCCC": 460,
548
+ "+][": 461,
549
+ "NO": 462,
550
+ "CNCCCC": 463,
551
+ "CNCCC": 464,
552
+ "COCOCC": 465,
553
+ "COcccOC": 466,
554
+ "OCCCOP": 467,
555
+ "CCCCNC": 468,
556
+ "Clcccc": 469,
557
+ "Occcc": 470,
558
+ "ccF": 471,
559
+ "ncCl": 472,
560
+ "ccCC": 473,
561
+ "CCccccO": 474,
562
+ "CCOCCO": 475,
563
+ "OCcccccc": 476,
564
+ "CCCCCCCCCCCCCCC": 477,
565
+ "SCC": 478,
566
+ "NCCccccO": 479,
567
+ "CccccCl": 480,
568
+ "CccO": 481,
569
+ "csc": 482,
570
+ "ncNC": 483,
571
+ "cncCl": 484,
572
+ "nccc": 485,
573
+ "CCCCCCO": 486,
574
+ "20": 487,
575
+ "SN": 488,
576
+ "coc": 489,
577
+ "))))))))))))))))))": 490,
578
+ "NCCNC": 491,
579
+ "CCCCCCCCCC": 492,
580
+ "NNC": 493,
581
+ "ccccs": 494,
582
+ "CcccO": 495,
583
+ "CccC": 496,
584
+ "Fcccc": 497,
585
+ "cccccBr": 498,
586
+ "136": 499,
587
+ "cncnc": 500,
588
+ "Nccc": 501,
589
+ "CCcc": 502,
590
+ "ccccccccc": 503,
591
+ "CCCl": 504,
592
+ "CcccccC": 505,
593
+ "22": 506,
594
+ "CCCCCO": 507,
595
+ "cccoc": 508,
596
+ "ncn": 509,
597
+ "NCCN": 510,
598
+ "ccCF": 511
599
+ },
600
+ "merges": [
601
+ ") )",
602
+ "c c",
603
+ "C C",
604
+ ")) ))",
605
+ "cc cc",
606
+ "CC CC",
607
+ "C O",
608
+ ")))) ))))",
609
+ "CC C",
610
+ "cccc cc",
611
+ "cc c",
612
+ ")) )",
613
+ "N C",
614
+ "O C",
615
+ "1 0",
616
+ "n c",
617
+ "CC O",
618
+ "C l",
619
+ ") =",
620
+ "N CC",
621
+ ")))) ))",
622
+ "CCCC C",
623
+ ")))) )))",
624
+ "n H",
625
+ "cccc c",
626
+ "CCCC CCCC",
627
+ "O CC",
628
+ "C N",
629
+ "CO C",
630
+ "c n",
631
+ "B r",
632
+ ")))))))) )",
633
+ "C cccc",
634
+ ") [",
635
+ ")))) )",
636
+ "CO cccc",
637
+ ")))))))) ))",
638
+ "C NC",
639
+ "CO CCO",
640
+ "CCCC CC",
641
+ "+ ]",
642
+ "- ]",
643
+ "c C",
644
+ "c nc",
645
+ "cc O",
646
+ "o c",
647
+ "1 4",
648
+ "CC OC",
649
+ "cc C",
650
+ ")))))))) )))",
651
+ "CO ccc",
652
+ "cccc O",
653
+ "10 6",
654
+ "C ccc",
655
+ "c O",
656
+ "CC N",
657
+ "C cccccc",
658
+ "+] =",
659
+ "N cccc",
660
+ "CO CC",
661
+ "6 9",
662
+ "c OC",
663
+ "9 6",
664
+ "N CCCC",
665
+ "N CCC",
666
+ "CC NC",
667
+ "n cc",
668
+ "cccc C",
669
+ "-] ))",
670
+ "C S",
671
+ "CCCC CCC",
672
+ "C F",
673
+ "C NCC",
674
+ "C cc",
675
+ "N cccccc",
676
+ "CC CO",
677
+ "ccc O",
678
+ "cccc Cl",
679
+ "1 3",
680
+ "CO cccccc",
681
+ "N CCO",
682
+ ")))))))) ))))",
683
+ "CCCC CCCCC",
684
+ "s c",
685
+ "1 7",
686
+ "9 5",
687
+ "n n",
688
+ "CC NCC",
689
+ "CO ccccc",
690
+ "O CCO",
691
+ "c Cl",
692
+ "CC l",
693
+ "C ccccc",
694
+ "cc OC",
695
+ "C cn",
696
+ "ccc C",
697
+ "n ccC",
698
+ "COcccc C",
699
+ "CO P",
700
+ "O COCCO",
701
+ "CCC N",
702
+ "O CO",
703
+ ")))))))) )))))",
704
+ "cccc F",
705
+ "1 1",
706
+ "S C",
707
+ "CO COCCO",
708
+ "cccc OC",
709
+ "ccc Cl",
710
+ "-] )",
711
+ "cc n",
712
+ "c N",
713
+ ")))))))) ))))))))",
714
+ "nc nc",
715
+ "cccc Br",
716
+ "cccc nc",
717
+ "O CCC",
718
+ "N S",
719
+ "CC cccccc",
720
+ "Ccccc O",
721
+ "1 5",
722
+ "CCC NC",
723
+ "cccc N",
724
+ ")))))))) )))))))",
725
+ "CCCCCCCC CCCCCCCC",
726
+ "n C",
727
+ "c o",
728
+ "ccc OC",
729
+ "COccc C",
730
+ "cc Cl",
731
+ "CCCC N",
732
+ "nc C",
733
+ "Ccccc C",
734
+ "nc N",
735
+ "Cccc C",
736
+ "ccccc O",
737
+ "COccc O",
738
+ "c s",
739
+ "CCCC O",
740
+ "C cnc",
741
+ "ccc Br",
742
+ "N CCCCC",
743
+ "CC S",
744
+ "NCC cc",
745
+ "cccc cccccc",
746
+ "cc nc",
747
+ "NCC NCC",
748
+ "1 2",
749
+ "n cccccc",
750
+ "NCC cccccc",
751
+ "C nc",
752
+ "cccc NC",
753
+ "ccccc n",
754
+ "N cn",
755
+ "NC cccccc",
756
+ "c CC",
757
+ "c F",
758
+ "o cccO",
759
+ "CCO COCCO",
760
+ "ccccc Cl",
761
+ "Ncccc Cl",
762
+ "F CF",
763
+ "N N",
764
+ "ccc F",
765
+ "cn cccccc",
766
+ "CCO P",
767
+ "Ccccc NC",
768
+ "O CCCC",
769
+ "n nc",
770
+ "CCO CC",
771
+ "ccccc C",
772
+ "OCC OC",
773
+ "NCCCC N",
774
+ "N cnc",
775
+ "cn n",
776
+ ")) =",
777
+ "CCCCC nccC",
778
+ "CC COC",
779
+ "1 6",
780
+ "C c",
781
+ ")))))))) ))))))",
782
+ "ccc ncc",
783
+ "Ccccc S",
784
+ "cc Br",
785
+ "CCC NCC",
786
+ "NCC S",
787
+ "cccc cccc",
788
+ "1 8",
789
+ "Ncccc C",
790
+ "O COCC",
791
+ "CO cc",
792
+ "N CCCCCC",
793
+ ")))))))) )))))))))",
794
+ "NCCO CC",
795
+ "O cccccc",
796
+ "c Br",
797
+ "cc N",
798
+ "CO ccO",
799
+ "CCC cccccc",
800
+ "CCCCCCCC CCCCCC",
801
+ "+] [",
802
+ "N O",
803
+ "CN CCCC",
804
+ "CN CCC",
805
+ "CO COCC",
806
+ "COccc OC",
807
+ "OCC COP",
808
+ "CCCC NC",
809
+ "Cl cccc",
810
+ "O cccc",
811
+ "cc F",
812
+ "nc Cl",
813
+ "cc CC",
814
+ "CC ccccO",
815
+ "CCO CCO",
816
+ "OC cccccc",
817
+ "CCCCCCCC CCCCCCC",
818
+ "S CC",
819
+ "NCC ccccO",
820
+ "Ccccc Cl",
821
+ "C ccO",
822
+ "c sc",
823
+ "nc NC",
824
+ "cnc Cl",
825
+ "n ccc",
826
+ "CCCC CCO",
827
+ "2 0",
828
+ "S N",
829
+ "c oc",
830
+ ")))))))) ))))))))))",
831
+ "NCC NC",
832
+ "CCCCCCCC CC",
833
+ "N NC",
834
+ "cccc s",
835
+ "Cccc O",
836
+ "C ccC",
837
+ "F cccc",
838
+ "ccccc Br",
839
+ "13 6",
840
+ "cnc nc",
841
+ "N ccc",
842
+ "CC cc",
843
+ "cccc ccccc",
844
+ "CCC l",
845
+ "Cccccc C",
846
+ "2 2",
847
+ "CCCC CO",
848
+ "ccc oc",
849
+ "nc n",
850
+ "NCC N",
851
+ "ccC F"
852
+ ]
853
+ }
854
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<s>",
4
+ "clean_up_tokenization_spaces": true,
5
+ "cls_token": "<s>",
6
+ "eos_token": "</s>",
7
+ "errors": "replace",
8
+ "mask_token": {
9
+ "__type": "AddedToken",
10
+ "content": "<mask>",
11
+ "lstrip": true,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "model_max_length": 1024,
17
+ "pad_token": "<pad>",
18
+ "sep_token": "</s>",
19
+ "tokenizer_class": "BartTokenizer",
20
+ "trim_offsets": true,
21
+ "unk_token": "<unk>"
22
+ }
vocab.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"<s>":0,"<pad>":1,"</s>":2,"<unk>":3,"<mask>":4,"!":5,"\"":6,"#":7,"$":8,"%":9,"&":10,"'":11,"(":12,")":13,"*":14,"+":15,",":16,"-":17,".":18,"/":19,"0":20,"1":21,"2":22,"3":23,"4":24,"5":25,"6":26,"7":27,"8":28,"9":29,":":30,";":31,"<":32,"=":33,">":34,"?":35,"@":36,"A":37,"B":38,"C":39,"D":40,"E":41,"F":42,"G":43,"H":44,"I":45,"J":46,"K":47,"L":48,"M":49,"N":50,"O":51,"P":52,"Q":53,"R":54,"S":55,"T":56,"U":57,"V":58,"W":59,"X":60,"Y":61,"Z":62,"[":63,"\\":64,"]":65,"^":66,"_":67,"`":68,"a":69,"b":70,"c":71,"d":72,"e":73,"f":74,"g":75,"h":76,"i":77,"j":78,"k":79,"l":80,"m":81,"n":82,"o":83,"p":84,"q":85,"r":86,"s":87,"t":88,"u":89,"v":90,"w":91,"x":92,"y":93,"z":94,"{":95,"|":96,"}":97,"~":98,"¡":99,"¢":100,"£":101,"¤":102,"¥":103,"¦":104,"§":105,"¨":106,"©":107,"ª":108,"«":109,"¬":110,"®":111,"¯":112,"°":113,"±":114,"²":115,"³":116,"´":117,"µ":118,"¶":119,"·":120,"¸":121,"¹":122,"º":123,"»":124,"¼":125,"½":126,"¾":127,"¿":128,"À":129,"Á":130,"Â":131,"Ã":132,"Ä":133,"Å":134,"Æ":135,"Ç":136,"È":137,"É":138,"Ê":139,"Ë":140,"Ì":141,"Í":142,"Î":143,"Ï":144,"Ð":145,"Ñ":146,"Ò":147,"Ó":148,"Ô":149,"Õ":150,"Ö":151,"×":152,"Ø":153,"Ù":154,"Ú":155,"Û":156,"Ü":157,"Ý":158,"Þ":159,"ß":160,"à":161,"á":162,"â":163,"ã":164,"ä":165,"å":166,"æ":167,"ç":168,"è":169,"é":170,"ê":171,"ë":172,"ì":173,"í":174,"î":175,"ï":176,"ð":177,"ñ":178,"ò":179,"ó":180,"ô":181,"õ":182,"ö":183,"÷":184,"ø":185,"ù":186,"ú":187,"û":188,"ü":189,"ý":190,"þ":191,"ÿ":192,"Ā":193,"ā":194,"Ă":195,"ă":196,"Ą":197,"ą":198,"Ć":199,"ć":200,"Ĉ":201,"ĉ":202,"Ċ":203,"ċ":204,"Č":205,"č":206,"Ď":207,"ď":208,"Đ":209,"đ":210,"Ē":211,"ē":212,"Ĕ":213,"ĕ":214,"Ė":215,"ė":216,"Ę":217,"ę":218,"Ě":219,"ě":220,"Ĝ":221,"ĝ":222,"Ğ":223,"ğ":224,"Ġ":225,"ġ":226,"Ģ":227,"ģ":228,"Ĥ":229,"ĥ":230,"Ħ":231,"ħ":232,"Ĩ":233,"ĩ":234,"Ī":235,"ī":236,"Ĭ":237,"ĭ":238,"Į":239,"į":240,"İ":241,"ı":242,"IJ":243,"ij":244,"Ĵ":245,"ĵ":246,"Ķ":247,"ķ":248,"ĸ":249,"Ĺ":250,"ĺ":251,"Ļ":252,"ļ":253,"Ľ":254,"ľ":255,"Ŀ":256,"ŀ":257,"Ł":258,"ł":259,"Ń":260,"))":261,"cc":262,"CC":263,"))))":264,"cccc":265,"CCCC":266,"CO":267,"))))))))":268,"CCC":269,"cccccc":270,"ccc":271,")))":272,"NC":273,"OC":274,"10":275,"nc":276,"CCO":277,"Cl":278,")=":279,"NCC":280,"))))))":281,"CCCCC":282,")))))))":283,"nH":284,"ccccc":285,"CCCCCCCC":286,"OCC":287,"CN":288,"COC":289,"cn":290,"Br":291,")))))))))":292,"Ccccc":293,")[":294,")))))":295,"COcccc":296,"))))))))))":297,"CNC":298,"COCCO":299,"CCCCCC":300,"+]":301,"-]":302,"cC":303,"cnc":304,"ccO":305,"oc":306,"14":307,"CCOC":308,"ccC":309,")))))))))))":310,"COccc":311,"ccccO":312,"106":313,"Cccc":314,"cO":315,"CCN":316,"Ccccccc":317,"+]=":318,"Ncccc":319,"COCC":320,"69":321,"cOC":322,"96":323,"NCCCC":324,"NCCC":325,"CCNC":326,"ncc":327,"ccccC":328,"-]))":329,"CS":330,"CCCCCCC":331,"CF":332,"CNCC":333,"Ccc":334,"Ncccccc":335,"CCCO":336,"cccO":337,"ccccCl":338,"13":339,"COcccccc":340,"NCCO":341,"))))))))))))":342,"CCCCCCCCC":343,"sc":344,"17":345,"95":346,"nn":347,"CCNCC":348,"COccccc":349,"OCCO":350,"cCl":351,"CCl":352,"Cccccc":353,"ccOC":354,"Ccn":355,"cccC":356,"nccC":357,"COccccC":358,"COP":359,"OCOCCO":360,"CCCN":361,"OCO":362,")))))))))))))":363,"ccccF":364,"11":365,"SC":366,"COCOCCO":367,"ccccOC":368,"cccCl":369,"-])":370,"ccn":371,"cN":372,"))))))))))))))))":373,"ncnc":374,"ccccBr":375,"ccccnc":376,"OCCC":377,"NS":378,"CCcccccc":379,"CccccO":380,"15":381,"CCCNC":382,"ccccN":383,")))))))))))))))":384,"CCCCCCCCCCCCCCCC":385,"nC":386,"co":387,"cccOC":388,"COcccC":389,"ccCl":390,"CCCCN":391,"ncC":392,"CccccC":393,"ncN":394,"CcccC":395,"cccccO":396,"COcccO":397,"cs":398,"CCCCO":399,"Ccnc":400,"cccBr":401,"NCCCCC":402,"CCS":403,"NCCcc":404,"cccccccccc":405,"ccnc":406,"NCCNCC":407,"12":408,"ncccccc":409,"NCCcccccc":410,"Cnc":411,"ccccNC":412,"cccccn":413,"Ncn":414,"NCcccccc":415,"cCC":416,"cF":417,"occcO":418,"CCOCOCCO":419,"cccccCl":420,"NccccCl":421,"FCF":422,"NN":423,"cccF":424,"cncccccc":425,"CCOP":426,"CccccNC":427,"OCCCC":428,"nnc":429,"CCOCC":430,"cccccC":431,"OCCOC":432,"NCCCCN":433,"Ncnc":434,"cnn":435,"))=":436,"CCCCCnccC":437,"CCCOC":438,"16":439,"Cc":440,"))))))))))))))":441,"cccncc":442,"CccccS":443,"ccBr":444,"CCCNCC":445,"NCCS":446,"cccccccc":447,"18":448,"NccccC":449,"OCOCC":450,"COcc":451,"NCCCCCC":452,")))))))))))))))))":453,"NCCOCC":454,"Occcccc":455,"cBr":456,"ccN":457,"COccO":458,"CCCcccccc":459,"CCCCCCCCCCCCCC":460,"+][":461,"NO":462,"CNCCCC":463,"CNCCC":464,"COCOCC":465,"COcccOC":466,"OCCCOP":467,"CCCCNC":468,"Clcccc":469,"Occcc":470,"ccF":471,"ncCl":472,"ccCC":473,"CCccccO":474,"CCOCCO":475,"OCcccccc":476,"CCCCCCCCCCCCCCC":477,"SCC":478,"NCCccccO":479,"CccccCl":480,"CccO":481,"csc":482,"ncNC":483,"cncCl":484,"nccc":485,"CCCCCCO":486,"20":487,"SN":488,"coc":489,"))))))))))))))))))":490,"NCCNC":491,"CCCCCCCCCC":492,"NNC":493,"ccccs":494,"CcccO":495,"CccC":496,"Fcccc":497,"cccccBr":498,"136":499,"cncnc":500,"Nccc":501,"CCcc":502,"ccccccccc":503,"CCCl":504,"CcccccC":505,"22":506,"CCCCCO":507,"cccoc":508,"ncn":509,"NCCN":510,"ccCF":511}