Hugging Face's logo

Gautam6
/
GAUTAMAI

Text Generation
Safetensors
PyTorch
Hindi
English
llama
gautam-ai
devanagari-hindi
english
causal-lm
self-improving
lora
devanagari
conversational
Model card Files
xet
 Community
Gautam6 commited on
Commit
85abd7d
·
verified ·
1 Parent(s): 259f420

Upload tokenizer

Browse files
Files changed (4) hide show
  1. README.md +3 -0
  2. special_tokens_map.json +6 -0
  3. tokenizer.json +2271 -0
  4. tokenizer_config.json +68 -0
README.md CHANGED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ ---
2
+ {}
3
+ ---
special_tokens_map.json CHANGED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
tokenizer.json CHANGED
@@ -0,0 +1,2271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "1.0",
3
+ "truncation": null,
4
+ "padding": null,
5
+ "added_tokens": [
6
+ {
7
+ "id": 0,
8
+ "content": "<s>",
9
+ "single_word": false,
10
+ "lstrip": false,
11
+ "rstrip": false,
12
+ "normalized": false,
13
+ "special": true
14
+ },
15
+ {
16
+ "id": 1,
17
+ "content": "<pad>",
18
+ "single_word": false,
19
+ "lstrip": false,
20
+ "rstrip": false,
21
+ "normalized": false,
22
+ "special": true
23
+ },
24
+ {
25
+ "id": 2,
26
+ "content": "</s>",
27
+ "single_word": false,
28
+ "lstrip": false,
29
+ "rstrip": false,
30
+ "normalized": false,
31
+ "special": true
32
+ },
33
+ {
34
+ "id": 3,
35
+ "content": "<unk>",
36
+ "single_word": false,
37
+ "lstrip": false,
38
+ "rstrip": false,
39
+ "normalized": false,
40
+ "special": true
41
+ },
42
+ {
43
+ "id": 4,
44
+ "content": "<|im_start|>",
45
+ "single_word": false,
46
+ "lstrip": false,
47
+ "rstrip": false,
48
+ "normalized": false,
49
+ "special": true
50
+ },
51
+ {
52
+ "id": 5,
53
+ "content": "<|im_end|>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ {
61
+ "id": 6,
62
+ "content": "<mask|>",
63
+ "single_word": false,
64
+ "lstrip": false,
65
+ "rstrip": false,
66
+ "normalized": false,
67
+ "special": true
68
+ }
69
+ ],
70
+ "normalizer": null,
71
+ "pre_tokenizer": {
72
+ "type": "ByteLevel",
73
+ "add_prefix_space": false,
74
+ "trim_offsets": true,
75
+ "use_regex": true
76
+ },
77
+ "post_processor": {
78
+ "type": "ByteLevel",
79
+ "add_prefix_space": true,
80
+ "trim_offsets": false,
81
+ "use_regex": true
82
+ },
83
+ "decoder": {
84
+ "type": "ByteLevel",
85
+ "add_prefix_space": true,
86
+ "trim_offsets": true,
87
+ "use_regex": true
88
+ },
89
+ "model": {
90
+ "type": "BPE",
91
+ "dropout": null,
92
+ "unk_token": "<unk>",
93
+ "continuing_subword_prefix": null,
94
+ "end_of_word_suffix": null,
95
+ "fuse_unk": false,
96
+ "byte_fallback": false,
97
+ "ignore_merges": false,
98
+ "vocab": {
99
+ "<s>": 0,
100
+ "<pad>": 1,
101
+ "</s>": 2,
102
+ "<unk>": 3,
103
+ "<|im_start|>": 4,
104
+ "<|im_end|>": 5,
105
+ "<mask|>": 6,
106
+ "!": 7,
107
+ "\"": 8,
108
+ "#": 9,
109
+ "$": 10,
110
+ "%": 11,
111
+ "&": 12,
112
+ "'": 13,
113
+ "(": 14,
114
+ ")": 15,
115
+ "*": 16,
116
+ "+": 17,
117
+ ",": 18,
118
+ "-": 19,
119
+ ".": 20,
120
+ "/": 21,
121
+ "0": 22,
122
+ "1": 23,
123
+ "2": 24,
124
+ "3": 25,
125
+ "4": 26,
126
+ "5": 27,
127
+ "6": 28,
128
+ "7": 29,
129
+ "8": 30,
130
+ "9": 31,
131
+ ":": 32,
132
+ ";": 33,
133
+ "<": 34,
134
+ "=": 35,
135
+ ">": 36,
136
+ "?": 37,
137
+ "@": 38,
138
+ "A": 39,
139
+ "B": 40,
140
+ "C": 41,
141
+ "D": 42,
142
+ "E": 43,
143
+ "F": 44,
144
+ "G": 45,
145
+ "H": 46,
146
+ "I": 47,
147
+ "J": 48,
148
+ "K": 49,
149
+ "L": 50,
150
+ "M": 51,
151
+ "N": 52,
152
+ "O": 53,
153
+ "P": 54,
154
+ "Q": 55,
155
+ "R": 56,
156
+ "S": 57,
157
+ "T": 58,
158
+ "U": 59,
159
+ "V": 60,
160
+ "W": 61,
161
+ "X": 62,
162
+ "Y": 63,
163
+ "Z": 64,
164
+ "[": 65,
165
+ "\\": 66,
166
+ "]": 67,
167
+ "^": 68,
168
+ "_": 69,
169
+ "`": 70,
170
+ "a": 71,
171
+ "b": 72,
172
+ "c": 73,
173
+ "d": 74,
174
+ "e": 75,
175
+ "f": 76,
176
+ "g": 77,
177
+ "h": 78,
178
+ "i": 79,
179
+ "j": 80,
180
+ "k": 81,
181
+ "l": 82,
182
+ "m": 83,
183
+ "n": 84,
184
+ "o": 85,
185
+ "p": 86,
186
+ "q": 87,
187
+ "r": 88,
188
+ "s": 89,
189
+ "t": 90,
190
+ "u": 91,
191
+ "v": 92,
192
+ "w": 93,
193
+ "x": 94,
194
+ "y": 95,
195
+ "z": 96,
196
+ "{": 97,
197
+ "|": 98,
198
+ "}": 99,
199
+ "~": 100,
200
+ "¡": 101,
201
+ "¢": 102,
202
+ "£": 103,
203
+ "¤": 104,
204
+ "¥": 105,
205
+ "¦": 106,
206
+ "§": 107,
207
+ "¨": 108,
208
+ "©": 109,
209
+ "ª": 110,
210
+ "«": 111,
211
+ "¬": 112,
212
+ "®": 113,
213
+ "¯": 114,
214
+ "°": 115,
215
+ "±": 116,
216
+ "²": 117,
217
+ "³": 118,
218
+ "´": 119,
219
+ "µ": 120,
220
+ "¶": 121,
221
+ "·": 122,
222
+ "¸": 123,
223
+ "¹": 124,
224
+ "º": 125,
225
+ "»": 126,
226
+ "¼": 127,
227
+ "½": 128,
228
+ "¾": 129,
229
+ "¿": 130,
230
+ "À": 131,
231
+ "Á": 132,
232
+ "Â": 133,
233
+ "Ã": 134,
234
+ "Ä": 135,
235
+ "Å": 136,
236
+ "Æ": 137,
237
+ "Ç": 138,
238
+ "È": 139,
239
+ "É": 140,
240
+ "Ê": 141,
241
+ "Ë": 142,
242
+ "Ì": 143,
243
+ "Í": 144,
244
+ "Î": 145,
245
+ "Ï": 146,
246
+ "Ð": 147,
247
+ "Ñ": 148,
248
+ "Ò": 149,
249
+ "Ó": 150,
250
+ "Ô": 151,
251
+ "Õ": 152,
252
+ "Ö": 153,
253
+ "×": 154,
254
+ "Ø": 155,
255
+ "Ù": 156,
256
+ "Ú": 157,
257
+ "Û": 158,
258
+ "Ü": 159,
259
+ "Ý": 160,
260
+ "Þ": 161,
261
+ "ß": 162,
262
+ "à": 163,
263
+ "á": 164,
264
+ "â": 165,
265
+ "ã": 166,
266
+ "ä": 167,
267
+ "å": 168,
268
+ "æ": 169,
269
+ "ç": 170,
270
+ "è": 171,
271
+ "é": 172,
272
+ "ê": 173,
273
+ "ë": 174,
274
+ "ì": 175,
275
+ "í": 176,
276
+ "î": 177,
277
+ "ï": 178,
278
+ "ð": 179,
279
+ "ñ": 180,
280
+ "ò": 181,
281
+ "ó": 182,
282
+ "ô": 183,
283
+ "õ": 184,
284
+ "ö": 185,
285
+ "÷": 186,
286
+ "ø": 187,
287
+ "ù": 188,
288
+ "ú": 189,
289
+ "û": 190,
290
+ "ü": 191,
291
+ "ý": 192,
292
+ "þ": 193,
293
+ "ÿ": 194,
294
+ "Ā": 195,
295
+ "ā": 196,
296
+ "Ă": 197,
297
+ "ă": 198,
298
+ "Ą": 199,
299
+ "ą": 200,
300
+ "Ć": 201,
301
+ "ć": 202,
302
+ "Ĉ": 203,
303
+ "ĉ": 204,
304
+ "Ċ": 205,
305
+ "ċ": 206,
306
+ "Č": 207,
307
+ "č": 208,
308
+ "Ď": 209,
309
+ "ď": 210,
310
+ "Đ": 211,
311
+ "đ": 212,
312
+ "Ē": 213,
313
+ "ē": 214,
314
+ "Ĕ": 215,
315
+ "ĕ": 216,
316
+ "Ė": 217,
317
+ "ė": 218,
318
+ "Ę": 219,
319
+ "ę": 220,
320
+ "Ě": 221,
321
+ "ě": 222,
322
+ "Ĝ": 223,
323
+ "ĝ": 224,
324
+ "Ğ": 225,
325
+ "ğ": 226,
326
+ "Ġ": 227,
327
+ "ġ": 228,
328
+ "Ģ": 229,
329
+ "ģ": 230,
330
+ "Ĥ": 231,
331
+ "ĥ": 232,
332
+ "Ħ": 233,
333
+ "ħ": 234,
334
+ "Ĩ": 235,
335
+ "ĩ": 236,
336
+ "Ī": 237,
337
+ "ī": 238,
338
+ "Ĭ": 239,
339
+ "ĭ": 240,
340
+ "Į": 241,
341
+ "į": 242,
342
+ "İ": 243,
343
+ "ı": 244,
344
+ "IJ": 245,
345
+ "ij": 246,
346
+ "Ĵ": 247,
347
+ "ĵ": 248,
348
+ "Ķ": 249,
349
+ "ķ": 250,
350
+ "ĸ": 251,
351
+ "Ĺ": 252,
352
+ "ĺ": 253,
353
+ "Ļ": 254,
354
+ "ļ": 255,
355
+ "Ľ": 256,
356
+ "ľ": 257,
357
+ "Ŀ": 258,
358
+ "ŀ": 259,
359
+ "Ł": 260,
360
+ "ł": 261,
361
+ "Ń": 262,
362
+ "à¤": 263,
363
+ "Ġà¤": 264,
364
+ "à¥": 265,
365
+ "Ġà¥": 266,
366
+ "à¥į": 267,
367
+ "or": 268,
368
+ "à¤Ĥ": 269,
369
+ "र": 270,
370
+ "त": 271,
371
+ "ा": 272,
372
+ "ĠA": 273,
373
+ "an": 274,
374
+ "ि": 275,
375
+ "Ġà¤ķ": 276,
376
+ "à¥ĩ": 277,
377
+ "AM": 278,
378
+ "AU": 279,
379
+ "GAU": 280,
380
+ "TAM": 281,
381
+ "er": 282,
382
+ "ig": 283,
383
+ "im": 284,
384
+ "on": 285,
385
+ "व": 286,
386
+ "Ġस": 287,
387
+ "Ġह": 288,
388
+ "ĠAI": 289,
389
+ "GAUTAM": 290,
390
+ "imp": 291,
391
+ "ce": 292,
392
+ "el": 293,
393
+ "for": 294,
394
+ "in": 295,
395
+ "ut": 296,
396
+ "Ġs": 297,
397
+ "द": 298,
398
+ "म": 299,
399
+ "Ġà¤Ķ": 300,
400
+ "ort": 301,
401
+ "import": 302,
402
+ "al": 303,
403
+ "ec": 304,
404
+ "ne": 305,
405
+ "ov": 306,
406
+ "Ġa": 307,
407
+ "Ġan": 308,
408
+ "Ġimport": 309,
409
+ "ष": 310,
410
+ "à¤ķ": 311,
411
+ "à¤Ĺ": 312,
412
+ "Ġद": 313,
413
+ "Ġम": 314,
414
+ "Ġà¤ħ": 315,
415
+ "।": 316,
416
+ "à¥Ģ": 317,
417
+ "à¥ģ": 318,
418
+ "à¥Ī": 319,
419
+ "Ġà¤Ķर": 320,
420
+ "as": 321,
421
+ "de": 322,
422
+ "ex": 323,
423
+ "eig": 324,
424
+ "fu": 325,
425
+ "is": 326,
426
+ "ion": 327,
427
+ "ly": 328,
428
+ "om": 329,
429
+ "re": 330,
430
+ "ss": 331,
431
+ "ur": 332,
432
+ "us": 333,
433
+ "âĢ": 334,
434
+ "Ġ'": 335,
435
+ "Ġ(": 336,
436
+ "Ġ1": 337,
437
+ "Ġt": 338,
438
+ "Ġ{": 339,
439
+ "Ġ}": 340,
440
+ "ĠGAUTAM": 341,
441
+ "Ġfor": 342,
442
+ "Ġne": 343,
443
+ "Ġex": 344,
444
+ "Ġre": 345,
445
+ "ģ।": 346,
446
+ "Ĥà¤": 347,
447
+ "ध": 348,
448
+ "न": 349,
449
+ "ल": 350,
450
+ "श": 351,
451
+ "Ġत": 352,
452
+ "Ġब": 353,
453
+ "Ġव": 354,
454
+ "Ġà¤ı": 355,
455
+ "Ġà¤ľ": 356,
456
+ "Ġà¤Ń": 357,
457
+ "à¥ĭ": 358,
458
+ "à¥Ĥà¤": 359,
459
+ "िà¤Ĥ": 360,
460
+ "ereig": 361,
461
+ "cess": 362,
462
+ "ell": 363,
463
+ "form": 364,
464
+ "ing": 365,
465
+ "ecut": 366,
466
+ "overeig": 367,
467
+ "Ġand": 368,
468
+ "à¥Īà¤Ĥ": 369,
469
+ "âĢĶ": 370,
470
+ "Ġexecut": 371,
471
+ "à¥Ĥà¤ģ।": 372,
472
+ "overeign": 373,
473
+ "';": 374,
474
+ "00": 375,
475
+ "Ca": 376,
476
+ "De": 377,
477
+ "En": 378,
478
+ "For": 379,
479
+ "GI": 380,
480
+ "Hin": 381,
481
+ "Hell": 382,
482
+ "LM": 383,
483
+ "Mo": 384,
484
+ "Sovereign": 385,
485
+ "Tas": 386,
486
+ "ag": 387,
487
+ "ar": 388,
488
+ "at": 389,
489
+ "bu": 390,
490
+ "ch": 391,
491
+ "co": 392,
492
+ "ct": 393,
493
+ "cal": 394,
494
+ "ccess": 395,
495
+ "di": 396,
496
+ "del": 397,
497
+ "ed": 398,
498
+ "en": 399,
499
+ "eTas": 400,
500
+ "fi": 401,
501
+ "fr": 402,
502
+ "gl": 403,
503
+ "hig": 404,
504
+ "il": 405,
505
+ "iv": 406,
506
+ "ifi": 407,
507
+ "js": 408,
508
+ "lly": 409,
509
+ "nat": 410,
510
+ "nct": 411,
511
+ "os": 412,
512
+ "ous": 413,
513
+ "ocess": 414,
514
+ "oMo": 415,
515
+ "pr": 416,
516
+ "per": 417,
517
+ "pec": 418,
518
+ "qu": 419,
519
+ "ran": 420,
520
+ "rov": 421,
521
+ "sig": 422,
522
+ "sform": 423,
523
+ "tw": 424,
524
+ "tur": 425,
525
+ "tell": 426,
526
+ "uccess": 427,
527
+ "van": 428,
528
+ "¯à¤¤": 429,
529
+ "¼à¥Ģ": 430,
530
+ "Ġ!": 431,
531
+ "Ġ\"": 432,
532
+ "Ġ#": 433,
533
+ "Ġ$": 434,
534
+ "Ġ%": 435,
535
+ "Ġ&": 436,
536
+ "Ġ)": 437,
537
+ "Ġ*": 438,
538
+ "Ġ+": 439,
539
+ "Ġ,": 440,
540
+ "Ġ-": 441,
541
+ "Ġ.": 442,
542
+ "Ġ/": 443,
543
+ "Ġ0": 444,
544
+ "Ġ2": 445,
545
+ "Ġ3": 446,
546
+ "Ġ4": 447,
547
+ "Ġ5": 448,
548
+ "Ġ6": 449,
549
+ "Ġ7": 450,
550
+ "Ġ8": 451,
551
+ "Ġ9": 452,
552
+ "Ġ:": 453,
553
+ "Ġ;": 454,
554
+ "Ġ<": 455,
555
+ "Ġ=": 456,
556
+ "Ġ>": 457,
557
+ "Ġ?": 458,
558
+ "Ġ@": 459,
559
+ "ĠI": 460,
560
+ "Ġ[": 461,
561
+ "Ġ]": 462,
562
+ "Ġ^": 463,
563
+ "Ġ_": 464,
564
+ "Ġin": 465,
565
+ "Ġde": 466,
566
+ "Ġis": 467,
567
+ "ĠDe": 468,
568
+ "ĠEn": 469,
569
+ "ĠHin": 470,
570
+ "Ġbu": 471,
571
+ "Ġco": 472,
572
+ "Ġfr": 473,
573
+ "Ġhig": 474,
574
+ "Ġjs": 475,
575
+ "Ġnat": 476,
576
+ "Ġos": 477,
577
+ "Ġpr": 478,
578
+ "ड": 479,
579
+ "ण": 480,
580
+ "प": 481,
581
+ "स": 482,
582
+ "à¤ĥ": 483,
583
+ "à¤ħ": 484,
584
+ "à¤ĵ": 485,
585
+ "à¤ľ": 486,
586
+ "à¤ŀ": 487,
587
+ "à¤Ń": 488,
588
+ "यत": 489,
589
+ "़à¥Ģ": 490,
590
+ "Ġड": 491,
591
+ "Ġढ": 492,
592
+ "Ġण": 493,
593
+ "Ġथ": 494,
594
+ "Ġध": 495,
595
+ "Ġन": 496,
596
+ "Ġप": 497,
597
+ "Ġफ": 498,
598
+ "Ġय": 499,
599
+ "Ġर": 500,
600
+ "Ġल": 501,
601
+ "Ġळ": 502,
602
+ "Ġश": 503,
603
+ "Ġष": 504,
604
+ "Ġि": 505,
605
+ "Ġà¤Ĥ": 506,
606
+ "Ġà¤ĥ": 507,
607
+ "Ġà¤Ĩ": 508,
608
+ "Ġà¤ĩ": 509,
609
+ "Ġà¤Ī": 510,
610
+ "Ġà¤ī": 511,
611
+ "Ġà¤Ĭ": 512,
612
+ "Ġà¤ĭ": 513,
613
+ "Ġà¤IJ": 514,
614
+ "Ġà¤ĵ": 515,
615
+ "Ġà¤ĸ": 516,
616
+ "Ġà¤Ĺ": 517,
617
+ "Ġà¤ĺ": 518,
618
+ "Ġà¤Ļ": 519,
619
+ "Ġà¤ļ": 520,
620
+ "Ġà¤Ľ": 521,
621
+ "Ġà¤Ŀ": 522,
622
+ "Ġà¤ŀ": 523,
623
+ "Ġà¤Ł": 524,
624
+ "Ġà¤ł": 525,
625
+ "०": 526,
626
+ "à¥ĥ": 527,
627
+ "Ġ१": 528,
628
+ "Ġ२": 529,
629
+ "Ġ३": 530,
630
+ "Ġ४": 531,
631
+ "Ġ५": 532,
632
+ "Ġ६": 533,
633
+ "Ġ८": 534,
634
+ "Ġ९": 535,
635
+ "Ġà¥Ģ": 536,
636
+ "Ġà¥ģ": 537,
637
+ "Ġà¥Ĥ": 538,
638
+ "Ġà¥ĥ": 539,
639
+ "Ġà¥ħ": 540,
640
+ "Ġà¥ĩ": 541,
641
+ "Ġà¥Ī": 542,
642
+ "Ġà¥ī": 543,
643
+ "Ġà¥ĭ": 544,
644
+ "Ġà¥Į": 545,
645
+ "Ġà¥į": 546,
646
+ "Ġà¥Ń": 547,
647
+ "ork": 548,
648
+ "orch": 549,
649
+ "à¤ĤâĢĶ": 550,
650
+ "रत": 551,
651
+ "रà¤Ń": 552,
652
+ "ĠAut": 553,
653
+ "ĠAGI": 554,
654
+ "ance": 555,
655
+ "Ġà¤ķरत": 556,
656
+ "à¥ĩ!": 557,
657
+ "à¥ĩà¤Ĥ": 558,
658
+ "ers": 559,
659
+ "ery": 560,
660
+ "igen": 561,
661
+ "onom": 562,
662
+ "oning": 563,
663
+ "वत": 564,
664
+ "वन": 565,
665
+ "वल": 566,
666
+ "improv": 567,
667
+ "elf": 568,
668
+ "utonom": 569,
669
+ "Ġsovereign": 570,
670
+ "Ġspec": 571,
671
+ "Ġsuccess": 572,
672
+ "Ġself": 573,
673
+ "मत": 574,
674
+ "मस": 575,
675
+ "alLM": 576,
676
+ "ned": 577,
677
+ "Ġam": 578,
678
+ "Ġautonom": 579,
679
+ "षण": 580,
680
+ "à¤Ĺर": 581,
681
+ "à¥Ī।": 582,
682
+ "asoning": 583,
683
+ "fully": 584,
684
+ "funct": 585,
685
+ "ish": 586,
686
+ "ural": 587,
687
+ "usalLM": 588,
688
+ "Ġ100": 589,
689
+ "Ġtran": 590,
690
+ "Ġtorch": 591,
691
+ "Ġnetw": 592,
692
+ "Ġneural": 593,
693
+ "Ġretur": 594,
694
+ "Ġreasoning": 595,
695
+ "नमस": 596,
696
+ "Ġà¤ıà¤ķ": 597,
697
+ "formance": 598,
698
+ "Ġexecution": 599,
699
+ "ĠexecuteTas": 600,
700
+ "CausalLM": 601,
701
+ "ForCausalLM": 602,
702
+ "Hello": 603,
703
+ "agar": 604,
704
+ "cally": 605,
705
+ "delForCausalLM": 606,
706
+ "glish": 607,
707
+ "ilt": 608,
708
+ "ive": 609,
709
+ "ifically": 610,
710
+ "ocessed": 611,
711
+ "oModelForCausalLM": 612,
712
+ "performance": 613,
713
+ "query": 614,
714
+ "signed": 615,
715
+ "sformers": 616,
716
+ "telligen": 617,
717
+ "vanagar": 618,
718
+ "Ġintelligen": 619,
719
+ "Ġdesigned": 620,
720
+ "ĠDevanagar": 621,
721
+ "ĠEnglish": 622,
722
+ "ĠHindi": 623,
723
+ "Ġbuilt": 624,
724
+ "Ġcode": 625,
725
+ "Ġfrom": 626,
726
+ "Ġhigh": 627,
727
+ "Ġjson": 628,
728
+ "Ġnative": 629,
729
+ "Ġprocessed": 630,
730
+ "़à¥ĢâĢĶ": 631,
731
+ "ĠAutoModelForCausalLM": 632,
732
+ "improving": 633,
733
+ "Ġspecifically": 634,
734
+ "Ġsuccessfully": 635,
735
+ "Ġautonomous": 636,
736
+ "function": 637,
737
+ "Ġtransformers": 638,
738
+ "Ġnetwork": 639,
739
+ "Ġreturn": 640,
740
+ "ĠexecuteTask": 641,
741
+ "Ġintelligence": 642,
742
+ "ĠDevanagari": 643
743
+ },
744
+ "merges": [
745
+ [
746
+ "à",
747
+ "¤"
748
+ ],
749
+ [
750
+ "Ġ",
751
+ "à¤"
752
+ ],
753
+ [
754
+ "à",
755
+ "¥"
756
+ ],
757
+ [
758
+ "Ġ",
759
+ "à¥"
760
+ ],
761
+ [
762
+ "à¥",
763
+ "į"
764
+ ],
765
+ [
766
+ "o",
767
+ "r"
768
+ ],
769
+ [
770
+ "à¤",
771
+ "��"
772
+ ],
773
+ [
774
+ "à¤",
775
+ "°"
776
+ ],
777
+ [
778
+ "à¤",
779
+ "¤"
780
+ ],
781
+ [
782
+ "à¤",
783
+ "¾"
784
+ ],
785
+ [
786
+ "Ġ",
787
+ "A"
788
+ ],
789
+ [
790
+ "a",
791
+ "n"
792
+ ],
793
+ [
794
+ "à¤",
795
+ "¿"
796
+ ],
797
+ [
798
+ "Ġà¤",
799
+ "ķ"
800
+ ],
801
+ [
802
+ "à¥",
803
+ "ĩ"
804
+ ],
805
+ [
806
+ "A",
807
+ "M"
808
+ ],
809
+ [
810
+ "A",
811
+ "U"
812
+ ],
813
+ [
814
+ "G",
815
+ "AU"
816
+ ],
817
+ [
818
+ "T",
819
+ "AM"
820
+ ],
821
+ [
822
+ "e",
823
+ "r"
824
+ ],
825
+ [
826
+ "i",
827
+ "g"
828
+ ],
829
+ [
830
+ "i",
831
+ "m"
832
+ ],
833
+ [
834
+ "o",
835
+ "n"
836
+ ],
837
+ [
838
+ "à¤",
839
+ "µ"
840
+ ],
841
+ [
842
+ "Ġà¤",
843
+ "¸"
844
+ ],
845
+ [
846
+ "Ġà¤",
847
+ "¹"
848
+ ],
849
+ [
850
+ "ĠA",
851
+ "I"
852
+ ],
853
+ [
854
+ "GAU",
855
+ "TAM"
856
+ ],
857
+ [
858
+ "im",
859
+ "p"
860
+ ],
861
+ [
862
+ "c",
863
+ "e"
864
+ ],
865
+ [
866
+ "e",
867
+ "l"
868
+ ],
869
+ [
870
+ "f",
871
+ "or"
872
+ ],
873
+ [
874
+ "i",
875
+ "n"
876
+ ],
877
+ [
878
+ "u",
879
+ "t"
880
+ ],
881
+ [
882
+ "Ġ",
883
+ "s"
884
+ ],
885
+ [
886
+ "à¤",
887
+ "¦"
888
+ ],
889
+ [
890
+ "à¤",
891
+ "®"
892
+ ],
893
+ [
894
+ "Ġà¤",
895
+ "Ķ"
896
+ ],
897
+ [
898
+ "or",
899
+ "t"
900
+ ],
901
+ [
902
+ "imp",
903
+ "ort"
904
+ ],
905
+ [
906
+ "a",
907
+ "l"
908
+ ],
909
+ [
910
+ "e",
911
+ "c"
912
+ ],
913
+ [
914
+ "n",
915
+ "e"
916
+ ],
917
+ [
918
+ "o",
919
+ "v"
920
+ ],
921
+ [
922
+ "Ġ",
923
+ "a"
924
+ ],
925
+ [
926
+ "Ġ",
927
+ "an"
928
+ ],
929
+ [
930
+ "Ġ",
931
+ "import"
932
+ ],
933
+ [
934
+ "à¤",
935
+ "·"
936
+ ],
937
+ [
938
+ "à¤",
939
+ "ķ"
940
+ ],
941
+ [
942
+ "à¤",
943
+ "Ĺ"
944
+ ],
945
+ [
946
+ "Ġà¤",
947
+ "¦"
948
+ ],
949
+ [
950
+ "Ġà¤",
951
+ "®"
952
+ ],
953
+ [
954
+ "Ġà¤",
955
+ "ħ"
956
+ ],
957
+ [
958
+ "à¥",
959
+ "¤"
960
+ ],
961
+ [
962
+ "à¥",
963
+ "Ģ"
964
+ ],
965
+ [
966
+ "à¥",
967
+ "ģ"
968
+ ],
969
+ [
970
+ "à¥",
971
+ "Ī"
972
+ ],
973
+ [
974
+ "Ġà¤Ķ",
975
+ "र"
976
+ ],
977
+ [
978
+ "a",
979
+ "s"
980
+ ],
981
+ [
982
+ "d",
983
+ "e"
984
+ ],
985
+ [
986
+ "e",
987
+ "x"
988
+ ],
989
+ [
990
+ "e",
991
+ "ig"
992
+ ],
993
+ [
994
+ "f",
995
+ "u"
996
+ ],
997
+ [
998
+ "i",
999
+ "s"
1000
+ ],
1001
+ [
1002
+ "i",
1003
+ "on"
1004
+ ],
1005
+ [
1006
+ "l",
1007
+ "y"
1008
+ ],
1009
+ [
1010
+ "o",
1011
+ "m"
1012
+ ],
1013
+ [
1014
+ "r",
1015
+ "e"
1016
+ ],
1017
+ [
1018
+ "s",
1019
+ "s"
1020
+ ],
1021
+ [
1022
+ "u",
1023
+ "r"
1024
+ ],
1025
+ [
1026
+ "u",
1027
+ "s"
1028
+ ],
1029
+ [
1030
+ "â",
1031
+ "Ģ"
1032
+ ],
1033
+ [
1034
+ "Ġ",
1035
+ "'"
1036
+ ],
1037
+ [
1038
+ "Ġ",
1039
+ "("
1040
+ ],
1041
+ [
1042
+ "Ġ",
1043
+ "1"
1044
+ ],
1045
+ [
1046
+ "Ġ",
1047
+ "t"
1048
+ ],
1049
+ [
1050
+ "Ġ",
1051
+ "{"
1052
+ ],
1053
+ [
1054
+ "Ġ",
1055
+ "}"
1056
+ ],
1057
+ [
1058
+ "Ġ",
1059
+ "GAUTAM"
1060
+ ],
1061
+ [
1062
+ "Ġ",
1063
+ "for"
1064
+ ],
1065
+ [
1066
+ "Ġ",
1067
+ "ne"
1068
+ ],
1069
+ [
1070
+ "Ġ",
1071
+ "ex"
1072
+ ],
1073
+ [
1074
+ "Ġ",
1075
+ "re"
1076
+ ],
1077
+ [
1078
+ "ģ",
1079
+ "।"
1080
+ ],
1081
+ [
1082
+ "Ĥ",
1083
+ "à¤"
1084
+ ],
1085
+ [
1086
+ "à¤",
1087
+ "§"
1088
+ ],
1089
+ [
1090
+ "à¤",
1091
+ "¨"
1092
+ ],
1093
+ [
1094
+ "à¤",
1095
+ "²"
1096
+ ],
1097
+ [
1098
+ "à¤",
1099
+ "¶"
1100
+ ],
1101
+ [
1102
+ "Ġà¤",
1103
+ "¤"
1104
+ ],
1105
+ [
1106
+ "Ġà¤",
1107
+ "¬"
1108
+ ],
1109
+ [
1110
+ "Ġà¤",
1111
+ "µ"
1112
+ ],
1113
+ [
1114
+ "Ġà¤",
1115
+ "ı"
1116
+ ],
1117
+ [
1118
+ "Ġà¤",
1119
+ "ľ"
1120
+ ],
1121
+ [
1122
+ "Ġà¤",
1123
+ "Ń"
1124
+ ],
1125
+ [
1126
+ "à¥",
1127
+ "ĭ"
1128
+ ],
1129
+ [
1130
+ "à¥",
1131
+ "Ĥà¤"
1132
+ ],
1133
+ [
1134
+ "ि",
1135
+ "à¤Ĥ"
1136
+ ],
1137
+ [
1138
+ "er",
1139
+ "eig"
1140
+ ],
1141
+ [
1142
+ "ce",
1143
+ "ss"
1144
+ ],
1145
+ [
1146
+ "el",
1147
+ "l"
1148
+ ],
1149
+ [
1150
+ "for",
1151
+ "m"
1152
+ ],
1153
+ [
1154
+ "in",
1155
+ "g"
1156
+ ],
1157
+ [
1158
+ "ec",
1159
+ "ut"
1160
+ ],
1161
+ [
1162
+ "ov",
1163
+ "ereig"
1164
+ ],
1165
+ [
1166
+ "Ġan",
1167
+ "d"
1168
+ ],
1169
+ [
1170
+ "à¥Ī",
1171
+ "à¤Ĥ"
1172
+ ],
1173
+ [
1174
+ "âĢ",
1175
+ "Ķ"
1176
+ ],
1177
+ [
1178
+ "Ġex",
1179
+ "ecut"
1180
+ ],
1181
+ [
1182
+ "à¥Ĥà¤",
1183
+ "ģ।"
1184
+ ],
1185
+ [
1186
+ "overeig",
1187
+ "n"
1188
+ ],
1189
+ [
1190
+ "'",
1191
+ ";"
1192
+ ],
1193
+ [
1194
+ "0",
1195
+ "0"
1196
+ ],
1197
+ [
1198
+ "C",
1199
+ "a"
1200
+ ],
1201
+ [
1202
+ "D",
1203
+ "e"
1204
+ ],
1205
+ [
1206
+ "E",
1207
+ "n"
1208
+ ],
1209
+ [
1210
+ "F",
1211
+ "or"
1212
+ ],
1213
+ [
1214
+ "G",
1215
+ "I"
1216
+ ],
1217
+ [
1218
+ "H",
1219
+ "in"
1220
+ ],
1221
+ [
1222
+ "H",
1223
+ "ell"
1224
+ ],
1225
+ [
1226
+ "L",
1227
+ "M"
1228
+ ],
1229
+ [
1230
+ "M",
1231
+ "o"
1232
+ ],
1233
+ [
1234
+ "S",
1235
+ "overeign"
1236
+ ],
1237
+ [
1238
+ "T",
1239
+ "as"
1240
+ ],
1241
+ [
1242
+ "a",
1243
+ "g"
1244
+ ],
1245
+ [
1246
+ "a",
1247
+ "r"
1248
+ ],
1249
+ [
1250
+ "a",
1251
+ "t"
1252
+ ],
1253
+ [
1254
+ "b",
1255
+ "u"
1256
+ ],
1257
+ [
1258
+ "c",
1259
+ "h"
1260
+ ],
1261
+ [
1262
+ "c",
1263
+ "o"
1264
+ ],
1265
+ [
1266
+ "c",
1267
+ "t"
1268
+ ],
1269
+ [
1270
+ "c",
1271
+ "al"
1272
+ ],
1273
+ [
1274
+ "c",
1275
+ "cess"
1276
+ ],
1277
+ [
1278
+ "d",
1279
+ "i"
1280
+ ],
1281
+ [
1282
+ "d",
1283
+ "el"
1284
+ ],
1285
+ [
1286
+ "e",
1287
+ "d"
1288
+ ],
1289
+ [
1290
+ "e",
1291
+ "n"
1292
+ ],
1293
+ [
1294
+ "e",
1295
+ "Tas"
1296
+ ],
1297
+ [
1298
+ "f",
1299
+ "i"
1300
+ ],
1301
+ [
1302
+ "f",
1303
+ "r"
1304
+ ],
1305
+ [
1306
+ "g",
1307
+ "l"
1308
+ ],
1309
+ [
1310
+ "h",
1311
+ "ig"
1312
+ ],
1313
+ [
1314
+ "i",
1315
+ "l"
1316
+ ],
1317
+ [
1318
+ "i",
1319
+ "v"
1320
+ ],
1321
+ [
1322
+ "i",
1323
+ "fi"
1324
+ ],
1325
+ [
1326
+ "j",
1327
+ "s"
1328
+ ],
1329
+ [
1330
+ "l",
1331
+ "ly"
1332
+ ],
1333
+ [
1334
+ "n",
1335
+ "at"
1336
+ ],
1337
+ [
1338
+ "n",
1339
+ "ct"
1340
+ ],
1341
+ [
1342
+ "o",
1343
+ "s"
1344
+ ],
1345
+ [
1346
+ "o",
1347
+ "us"
1348
+ ],
1349
+ [
1350
+ "o",
1351
+ "cess"
1352
+ ],
1353
+ [
1354
+ "o",
1355
+ "Mo"
1356
+ ],
1357
+ [
1358
+ "p",
1359
+ "r"
1360
+ ],
1361
+ [
1362
+ "p",
1363
+ "er"
1364
+ ],
1365
+ [
1366
+ "p",
1367
+ "ec"
1368
+ ],
1369
+ [
1370
+ "q",
1371
+ "u"
1372
+ ],
1373
+ [
1374
+ "r",
1375
+ "an"
1376
+ ],
1377
+ [
1378
+ "r",
1379
+ "ov"
1380
+ ],
1381
+ [
1382
+ "s",
1383
+ "ig"
1384
+ ],
1385
+ [
1386
+ "s",
1387
+ "form"
1388
+ ],
1389
+ [
1390
+ "t",
1391
+ "w"
1392
+ ],
1393
+ [
1394
+ "t",
1395
+ "ur"
1396
+ ],
1397
+ [
1398
+ "t",
1399
+ "ell"
1400
+ ],
1401
+ [
1402
+ "u",
1403
+ "ccess"
1404
+ ],
1405
+ [
1406
+ "v",
1407
+ "an"
1408
+ ],
1409
+ [
1410
+ "¯",
1411
+ "त"
1412
+ ],
1413
+ [
1414
+ "¼",
1415
+ "à¥Ģ"
1416
+ ],
1417
+ [
1418
+ "Ġ",
1419
+ "!"
1420
+ ],
1421
+ [
1422
+ "Ġ",
1423
+ "\""
1424
+ ],
1425
+ [
1426
+ "Ġ",
1427
+ "#"
1428
+ ],
1429
+ [
1430
+ "Ġ",
1431
+ "$"
1432
+ ],
1433
+ [
1434
+ "Ġ",
1435
+ "%"
1436
+ ],
1437
+ [
1438
+ "Ġ",
1439
+ "&"
1440
+ ],
1441
+ [
1442
+ "Ġ",
1443
+ ")"
1444
+ ],
1445
+ [
1446
+ "Ġ",
1447
+ "*"
1448
+ ],
1449
+ [
1450
+ "Ġ",
1451
+ "+"
1452
+ ],
1453
+ [
1454
+ "Ġ",
1455
+ ","
1456
+ ],
1457
+ [
1458
+ "Ġ",
1459
+ "-"
1460
+ ],
1461
+ [
1462
+ "Ġ",
1463
+ "."
1464
+ ],
1465
+ [
1466
+ "Ġ",
1467
+ "/"
1468
+ ],
1469
+ [
1470
+ "Ġ",
1471
+ "0"
1472
+ ],
1473
+ [
1474
+ "Ġ",
1475
+ "2"
1476
+ ],
1477
+ [
1478
+ "Ġ",
1479
+ "3"
1480
+ ],
1481
+ [
1482
+ "Ġ",
1483
+ "4"
1484
+ ],
1485
+ [
1486
+ "Ġ",
1487
+ "5"
1488
+ ],
1489
+ [
1490
+ "Ġ",
1491
+ "6"
1492
+ ],
1493
+ [
1494
+ "Ġ",
1495
+ "7"
1496
+ ],
1497
+ [
1498
+ "Ġ",
1499
+ "8"
1500
+ ],
1501
+ [
1502
+ "Ġ",
1503
+ "9"
1504
+ ],
1505
+ [
1506
+ "Ġ",
1507
+ ":"
1508
+ ],
1509
+ [
1510
+ "Ġ",
1511
+ ";"
1512
+ ],
1513
+ [
1514
+ "Ġ",
1515
+ "<"
1516
+ ],
1517
+ [
1518
+ "Ġ",
1519
+ "="
1520
+ ],
1521
+ [
1522
+ "Ġ",
1523
+ ">"
1524
+ ],
1525
+ [
1526
+ "Ġ",
1527
+ "?"
1528
+ ],
1529
+ [
1530
+ "Ġ",
1531
+ "@"
1532
+ ],
1533
+ [
1534
+ "Ġ",
1535
+ "I"
1536
+ ],
1537
+ [
1538
+ "Ġ",
1539
+ "["
1540
+ ],
1541
+ [
1542
+ "Ġ",
1543
+ "]"
1544
+ ],
1545
+ [
1546
+ "Ġ",
1547
+ "^"
1548
+ ],
1549
+ [
1550
+ "Ġ",
1551
+ "_"
1552
+ ],
1553
+ [
1554
+ "Ġ",
1555
+ "in"
1556
+ ],
1557
+ [
1558
+ "Ġ",
1559
+ "de"
1560
+ ],
1561
+ [
1562
+ "Ġ",
1563
+ "is"
1564
+ ],
1565
+ [
1566
+ "Ġ",
1567
+ "De"
1568
+ ],
1569
+ [
1570
+ "Ġ",
1571
+ "En"
1572
+ ],
1573
+ [
1574
+ "Ġ",
1575
+ "Hin"
1576
+ ],
1577
+ [
1578
+ "Ġ",
1579
+ "bu"
1580
+ ],
1581
+ [
1582
+ "Ġ",
1583
+ "co"
1584
+ ],
1585
+ [
1586
+ "Ġ",
1587
+ "fr"
1588
+ ],
1589
+ [
1590
+ "Ġ",
1591
+ "hig"
1592
+ ],
1593
+ [
1594
+ "Ġ",
1595
+ "js"
1596
+ ],
1597
+ [
1598
+ "Ġ",
1599
+ "nat"
1600
+ ],
1601
+ [
1602
+ "Ġ",
1603
+ "os"
1604
+ ],
1605
+ [
1606
+ "Ġ",
1607
+ "pr"
1608
+ ],
1609
+ [
1610
+ "à¤",
1611
+ "¡"
1612
+ ],
1613
+ [
1614
+ "à¤",
1615
+ "£"
1616
+ ],
1617
+ [
1618
+ "à¤",
1619
+ "ª"
1620
+ ],
1621
+ [
1622
+ "à¤",
1623
+ "¸"
1624
+ ],
1625
+ [
1626
+ "à¤",
1627
+ "ĥ"
1628
+ ],
1629
+ [
1630
+ "à¤",
1631
+ "ħ"
1632
+ ],
1633
+ [
1634
+ "à¤",
1635
+ "ĵ"
1636
+ ],
1637
+ [
1638
+ "à¤",
1639
+ "ľ"
1640
+ ],
1641
+ [
1642
+ "à¤",
1643
+ "ŀ"
1644
+ ],
1645
+ [
1646
+ "à¤",
1647
+ "Ń"
1648
+ ],
1649
+ [
1650
+ "à¤",
1651
+ "¯à¤¤"
1652
+ ],
1653
+ [
1654
+ "à¤",
1655
+ "¼à¥Ģ"
1656
+ ],
1657
+ [
1658
+ "Ġà¤",
1659
+ "¡"
1660
+ ],
1661
+ [
1662
+ "Ġà¤",
1663
+ "¢"
1664
+ ],
1665
+ [
1666
+ "Ġà¤",
1667
+ "£"
1668
+ ],
1669
+ [
1670
+ "Ġà¤",
1671
+ "¥"
1672
+ ],
1673
+ [
1674
+ "Ġà¤",
1675
+ "§"
1676
+ ],
1677
+ [
1678
+ "Ġà¤",
1679
+ "¨"
1680
+ ],
1681
+ [
1682
+ "Ġà¤",
1683
+ "ª"
1684
+ ],
1685
+ [
1686
+ "Ġà¤",
1687
+ "«"
1688
+ ],
1689
+ [
1690
+ "Ġà¤",
1691
+ "¯"
1692
+ ],
1693
+ [
1694
+ "Ġà¤",
1695
+ "°"
1696
+ ],
1697
+ [
1698
+ "Ġà¤",
1699
+ "²"
1700
+ ],
1701
+ [
1702
+ "Ġà¤",
1703
+ "³"
1704
+ ],
1705
+ [
1706
+ "Ġà¤",
1707
+ "¶"
1708
+ ],
1709
+ [
1710
+ "Ġà¤",
1711
+ "·"
1712
+ ],
1713
+ [
1714
+ "Ġà¤",
1715
+ "¿"
1716
+ ],
1717
+ [
1718
+ "Ġà¤",
1719
+ "Ĥ"
1720
+ ],
1721
+ [
1722
+ "Ġà¤",
1723
+ "ĥ"
1724
+ ],
1725
+ [
1726
+ "Ġà¤",
1727
+ "Ĩ"
1728
+ ],
1729
+ [
1730
+ "Ġà¤",
1731
+ "ĩ"
1732
+ ],
1733
+ [
1734
+ "Ġà¤",
1735
+ "Ī"
1736
+ ],
1737
+ [
1738
+ "Ġà¤",
1739
+ "ī"
1740
+ ],
1741
+ [
1742
+ "Ġà¤",
1743
+ "Ĭ"
1744
+ ],
1745
+ [
1746
+ "Ġà¤",
1747
+ "ĭ"
1748
+ ],
1749
+ [
1750
+ "Ġà¤",
1751
+ "IJ"
1752
+ ],
1753
+ [
1754
+ "Ġà¤",
1755
+ "ĵ"
1756
+ ],
1757
+ [
1758
+ "Ġà¤",
1759
+ "ĸ"
1760
+ ],
1761
+ [
1762
+ "Ġà¤",
1763
+ "Ĺ"
1764
+ ],
1765
+ [
1766
+ "Ġà¤",
1767
+ "ĺ"
1768
+ ],
1769
+ [
1770
+ "Ġà¤",
1771
+ "Ļ"
1772
+ ],
1773
+ [
1774
+ "Ġà¤",
1775
+ "ļ"
1776
+ ],
1777
+ [
1778
+ "Ġà¤",
1779
+ "Ľ"
1780
+ ],
1781
+ [
1782
+ "Ġà¤",
1783
+ "Ŀ"
1784
+ ],
1785
+ [
1786
+ "Ġà¤",
1787
+ "ŀ"
1788
+ ],
1789
+ [
1790
+ "Ġà¤",
1791
+ "Ł"
1792
+ ],
1793
+ [
1794
+ "Ġà¤",
1795
+ "ł"
1796
+ ],
1797
+ [
1798
+ "à¥",
1799
+ "¦"
1800
+ ],
1801
+ [
1802
+ "à¥",
1803
+ "ĥ"
1804
+ ],
1805
+ [
1806
+ "Ġà¥",
1807
+ "§"
1808
+ ],
1809
+ [
1810
+ "Ġà¥",
1811
+ "¨"
1812
+ ],
1813
+ [
1814
+ "Ġà¥",
1815
+ "©"
1816
+ ],
1817
+ [
1818
+ "Ġà¥",
1819
+ "ª"
1820
+ ],
1821
+ [
1822
+ "Ġà¥",
1823
+ "«"
1824
+ ],
1825
+ [
1826
+ "Ġà¥",
1827
+ "¬"
1828
+ ],
1829
+ [
1830
+ "Ġà¥",
1831
+ "®"
1832
+ ],
1833
+ [
1834
+ "Ġà¥",
1835
+ "¯"
1836
+ ],
1837
+ [
1838
+ "Ġà¥",
1839
+ "Ģ"
1840
+ ],
1841
+ [
1842
+ "Ġà¥",
1843
+ "ģ"
1844
+ ],
1845
+ [
1846
+ "Ġà¥",
1847
+ "Ĥ"
1848
+ ],
1849
+ [
1850
+ "Ġà¥",
1851
+ "ĥ"
1852
+ ],
1853
+ [
1854
+ "Ġà¥",
1855
+ "ħ"
1856
+ ],
1857
+ [
1858
+ "Ġà¥",
1859
+ "ĩ"
1860
+ ],
1861
+ [
1862
+ "Ġà¥",
1863
+ "Ī"
1864
+ ],
1865
+ [
1866
+ "Ġà¥",
1867
+ "ī"
1868
+ ],
1869
+ [
1870
+ "Ġà¥",
1871
+ "ĭ"
1872
+ ],
1873
+ [
1874
+ "Ġà¥",
1875
+ "Į"
1876
+ ],
1877
+ [
1878
+ "Ġà¥",
1879
+ "į"
1880
+ ],
1881
+ [
1882
+ "Ġà¥",
1883
+ "Ń"
1884
+ ],
1885
+ [
1886
+ "or",
1887
+ "k"
1888
+ ],
1889
+ [
1890
+ "or",
1891
+ "ch"
1892
+ ],
1893
+ [
1894
+ "à¤Ĥ",
1895
+ "âĢĶ"
1896
+ ],
1897
+ [
1898
+ "र",
1899
+ "त"
1900
+ ],
1901
+ [
1902
+ "र",
1903
+ "à¤Ń"
1904
+ ],
1905
+ [
1906
+ "ĠA",
1907
+ "ut"
1908
+ ],
1909
+ [
1910
+ "ĠA",
1911
+ "GI"
1912
+ ],
1913
+ [
1914
+ "an",
1915
+ "ce"
1916
+ ],
1917
+ [
1918
+ "Ġà¤ķ",
1919
+ "रत"
1920
+ ],
1921
+ [
1922
+ "à¥ĩ",
1923
+ "!"
1924
+ ],
1925
+ [
1926
+ "à¥ĩ",
1927
+ "à¤Ĥ"
1928
+ ],
1929
+ [
1930
+ "er",
1931
+ "s"
1932
+ ],
1933
+ [
1934
+ "er",
1935
+ "y"
1936
+ ],
1937
+ [
1938
+ "ig",
1939
+ "en"
1940
+ ],
1941
+ [
1942
+ "on",
1943
+ "om"
1944
+ ],
1945
+ [
1946
+ "on",
1947
+ "ing"
1948
+ ],
1949
+ [
1950
+ "व",
1951
+ "त"
1952
+ ],
1953
+ [
1954
+ "व",
1955
+ "न"
1956
+ ],
1957
+ [
1958
+ "व",
1959
+ "ल"
1960
+ ],
1961
+ [
1962
+ "imp",
1963
+ "rov"
1964
+ ],
1965
+ [
1966
+ "el",
1967
+ "f"
1968
+ ],
1969
+ [
1970
+ "ut",
1971
+ "onom"
1972
+ ],
1973
+ [
1974
+ "Ġs",
1975
+ "overeign"
1976
+ ],
1977
+ [
1978
+ "Ġs",
1979
+ "pec"
1980
+ ],
1981
+ [
1982
+ "Ġs",
1983
+ "uccess"
1984
+ ],
1985
+ [
1986
+ "Ġs",
1987
+ "elf"
1988
+ ],
1989
+ [
1990
+ "म",
1991
+ "त"
1992
+ ],
1993
+ [
1994
+ "म",
1995
+ "स"
1996
+ ],
1997
+ [
1998
+ "al",
1999
+ "LM"
2000
+ ],
2001
+ [
2002
+ "ne",
2003
+ "d"
2004
+ ],
2005
+ [
2006
+ "Ġa",
2007
+ "m"
2008
+ ],
2009
+ [
2010
+ "Ġa",
2011
+ "utonom"
2012
+ ],
2013
+ [
2014
+ "ष",
2015
+ "ण"
2016
+ ],
2017
+ [
2018
+ "à��Ĺ",
2019
+ "र"
2020
+ ],
2021
+ [
2022
+ "à¥Ī",
2023
+ "।"
2024
+ ],
2025
+ [
2026
+ "as",
2027
+ "oning"
2028
+ ],
2029
+ [
2030
+ "fu",
2031
+ "lly"
2032
+ ],
2033
+ [
2034
+ "fu",
2035
+ "nct"
2036
+ ],
2037
+ [
2038
+ "is",
2039
+ "h"
2040
+ ],
2041
+ [
2042
+ "ur",
2043
+ "al"
2044
+ ],
2045
+ [
2046
+ "us",
2047
+ "alLM"
2048
+ ],
2049
+ [
2050
+ "Ġ1",
2051
+ "00"
2052
+ ],
2053
+ [
2054
+ "Ġt",
2055
+ "ran"
2056
+ ],
2057
+ [
2058
+ "Ġt",
2059
+ "orch"
2060
+ ],
2061
+ [
2062
+ "Ġne",
2063
+ "tw"
2064
+ ],
2065
+ [
2066
+ "Ġne",
2067
+ "ural"
2068
+ ],
2069
+ [
2070
+ "Ġre",
2071
+ "tur"
2072
+ ],
2073
+ [
2074
+ "Ġre",
2075
+ "asoning"
2076
+ ],
2077
+ [
2078
+ "न",
2079
+ "मस"
2080
+ ],
2081
+ [
2082
+ "Ġà¤ı",
2083
+ "à¤ķ"
2084
+ ],
2085
+ [
2086
+ "form",
2087
+ "ance"
2088
+ ],
2089
+ [
2090
+ "Ġexecut",
2091
+ "ion"
2092
+ ],
2093
+ [
2094
+ "Ġexecut",
2095
+ "eTas"
2096
+ ],
2097
+ [
2098
+ "Ca",
2099
+ "usalLM"
2100
+ ],
2101
+ [
2102
+ "For",
2103
+ "CausalLM"
2104
+ ],
2105
+ [
2106
+ "Hell",
2107
+ "o"
2108
+ ],
2109
+ [
2110
+ "ag",
2111
+ "ar"
2112
+ ],
2113
+ [
2114
+ "cal",
2115
+ "ly"
2116
+ ],
2117
+ [
2118
+ "del",
2119
+ "ForCausalLM"
2120
+ ],
2121
+ [
2122
+ "gl",
2123
+ "ish"
2124
+ ],
2125
+ [
2126
+ "il",
2127
+ "t"
2128
+ ],
2129
+ [
2130
+ "iv",
2131
+ "e"
2132
+ ],
2133
+ [
2134
+ "ifi",
2135
+ "cally"
2136
+ ],
2137
+ [
2138
+ "ocess",
2139
+ "ed"
2140
+ ],
2141
+ [
2142
+ "oMo",
2143
+ "delForCausalLM"
2144
+ ],
2145
+ [
2146
+ "per",
2147
+ "formance"
2148
+ ],
2149
+ [
2150
+ "qu",
2151
+ "ery"
2152
+ ],
2153
+ [
2154
+ "sig",
2155
+ "ned"
2156
+ ],
2157
+ [
2158
+ "sform",
2159
+ "ers"
2160
+ ],
2161
+ [
2162
+ "tell",
2163
+ "igen"
2164
+ ],
2165
+ [
2166
+ "van",
2167
+ "agar"
2168
+ ],
2169
+ [
2170
+ "Ġin",
2171
+ "telligen"
2172
+ ],
2173
+ [
2174
+ "Ġde",
2175
+ "signed"
2176
+ ],
2177
+ [
2178
+ "ĠDe",
2179
+ "vanagar"
2180
+ ],
2181
+ [
2182
+ "ĠEn",
2183
+ "glish"
2184
+ ],
2185
+ [
2186
+ "ĠHin",
2187
+ "di"
2188
+ ],
2189
+ [
2190
+ "Ġbu",
2191
+ "ilt"
2192
+ ],
2193
+ [
2194
+ "Ġco",
2195
+ "de"
2196
+ ],
2197
+ [
2198
+ "Ġfr",
2199
+ "om"
2200
+ ],
2201
+ [
2202
+ "Ġhig",
2203
+ "h"
2204
+ ],
2205
+ [
2206
+ "Ġjs",
2207
+ "on"
2208
+ ],
2209
+ [
2210
+ "Ġnat",
2211
+ "ive"
2212
+ ],
2213
+ [
2214
+ "Ġpr",
2215
+ "ocessed"
2216
+ ],
2217
+ [
2218
+ "़à¥Ģ",
2219
+ "âĢĶ"
2220
+ ],
2221
+ [
2222
+ "ĠAut",
2223
+ "oModelForCausalLM"
2224
+ ],
2225
+ [
2226
+ "improv",
2227
+ "ing"
2228
+ ],
2229
+ [
2230
+ "Ġspec",
2231
+ "ifically"
2232
+ ],
2233
+ [
2234
+ "Ġsuccess",
2235
+ "fully"
2236
+ ],
2237
+ [
2238
+ "Ġautonom",
2239
+ "ous"
2240
+ ],
2241
+ [
2242
+ "funct",
2243
+ "ion"
2244
+ ],
2245
+ [
2246
+ "Ġtran",
2247
+ "sformers"
2248
+ ],
2249
+ [
2250
+ "Ġnetw",
2251
+ "ork"
2252
+ ],
2253
+ [
2254
+ "Ġretur",
2255
+ "n"
2256
+ ],
2257
+ [
2258
+ "ĠexecuteTas",
2259
+ "k"
2260
+ ],
2261
+ [
2262
+ "Ġintelligen",
2263
+ "ce"
2264
+ ],
2265
+ [
2266
+ "ĠDevanagar",
2267
+ "i"
2268
+ ]
2269
+ ]
2270
+ }
2271
+ }
tokenizer_config.json CHANGED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<s>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<pad>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "<|im_start|>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "5": {
44
+ "content": "<|im_end|>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ },
51
+ "6": {
52
+ "content": "<mask|>",
53
+ "lstrip": false,
54
+ "normalized": false,
55
+ "rstrip": false,
56
+ "single_word": false,
57
+ "special": true
58
+ }
59
+ },
60
+ "bos_token": "<s>",
61
+ "clean_up_tokenization_spaces": false,
62
+ "eos_token": "</s>",
63
+ "extra_special_tokens": {},
64
+ "model_max_length": 1000000000000000019884624838656,
65
+ "pad_token": "<pad>",
66
+ "tokenizer_class": "PreTrainedTokenizerFast",
67
+ "unk_token": "<unk>"
68
+ }