Transformers
JingzeShi commited on
Commit
75cdaf0
·
verified ·
1 Parent(s): ed16a20

Upload tokenizer

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,199 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags: []
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+ This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
added_tokens.json ADDED
@@ -0,0 +1,258 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</citation>": 128013,
3
+ "</code>": 128011,
4
+ "</judgement>": 128020,
5
+ "</think>": 128009,
6
+ "</tool_call>": 128005,
7
+ "</tool_response>": 128007,
8
+ "<citation>": 128012,
9
+ "<code>": 128010,
10
+ "<judgement>": 128019,
11
+ "<think>": 128008,
12
+ "<tool_call>": 128004,
13
+ "<tool_response>": 128006,
14
+ "<|begin_of_text|>": 128000,
15
+ "<|box_end|>": 128018,
16
+ "<|box_start|>": 128017,
17
+ "<|conversation_title|>": 128014,
18
+ "<|end_of_text|>": 128001,
19
+ "<|file_sep|>": 128031,
20
+ "<|fim_middle|>": 128027,
21
+ "<|fim_pad|>": 128029,
22
+ "<|fim_prefix|>": 128026,
23
+ "<|fim_suffix|>": 128028,
24
+ "<|im_end|>": 128003,
25
+ "<|im_start|>": 128002,
26
+ "<|image_pad|>": 128024,
27
+ "<|object_ref_end|>": 128016,
28
+ "<|object_ref_start|>": 128015,
29
+ "<|repo_name|>": 128030,
30
+ "<|reserved_special_token_0|>": 128032,
31
+ "<|reserved_special_token_100|>": 128132,
32
+ "<|reserved_special_token_101|>": 128133,
33
+ "<|reserved_special_token_102|>": 128134,
34
+ "<|reserved_special_token_103|>": 128135,
35
+ "<|reserved_special_token_104|>": 128136,
36
+ "<|reserved_special_token_105|>": 128137,
37
+ "<|reserved_special_token_106|>": 128138,
38
+ "<|reserved_special_token_107|>": 128139,
39
+ "<|reserved_special_token_108|>": 128140,
40
+ "<|reserved_special_token_109|>": 128141,
41
+ "<|reserved_special_token_10|>": 128042,
42
+ "<|reserved_special_token_110|>": 128142,
43
+ "<|reserved_special_token_111|>": 128143,
44
+ "<|reserved_special_token_112|>": 128144,
45
+ "<|reserved_special_token_113|>": 128145,
46
+ "<|reserved_special_token_114|>": 128146,
47
+ "<|reserved_special_token_115|>": 128147,
48
+ "<|reserved_special_token_116|>": 128148,
49
+ "<|reserved_special_token_117|>": 128149,
50
+ "<|reserved_special_token_118|>": 128150,
51
+ "<|reserved_special_token_119|>": 128151,
52
+ "<|reserved_special_token_11|>": 128043,
53
+ "<|reserved_special_token_120|>": 128152,
54
+ "<|reserved_special_token_121|>": 128153,
55
+ "<|reserved_special_token_122|>": 128154,
56
+ "<|reserved_special_token_123|>": 128155,
57
+ "<|reserved_special_token_124|>": 128156,
58
+ "<|reserved_special_token_125|>": 128157,
59
+ "<|reserved_special_token_126|>": 128158,
60
+ "<|reserved_special_token_127|>": 128159,
61
+ "<|reserved_special_token_128|>": 128160,
62
+ "<|reserved_special_token_129|>": 128161,
63
+ "<|reserved_special_token_12|>": 128044,
64
+ "<|reserved_special_token_130|>": 128162,
65
+ "<|reserved_special_token_131|>": 128163,
66
+ "<|reserved_special_token_132|>": 128164,
67
+ "<|reserved_special_token_133|>": 128165,
68
+ "<|reserved_special_token_134|>": 128166,
69
+ "<|reserved_special_token_135|>": 128167,
70
+ "<|reserved_special_token_136|>": 128168,
71
+ "<|reserved_special_token_137|>": 128169,
72
+ "<|reserved_special_token_138|>": 128170,
73
+ "<|reserved_special_token_139|>": 128171,
74
+ "<|reserved_special_token_13|>": 128045,
75
+ "<|reserved_special_token_140|>": 128172,
76
+ "<|reserved_special_token_141|>": 128173,
77
+ "<|reserved_special_token_142|>": 128174,
78
+ "<|reserved_special_token_143|>": 128175,
79
+ "<|reserved_special_token_144|>": 128176,
80
+ "<|reserved_special_token_145|>": 128177,
81
+ "<|reserved_special_token_146|>": 128178,
82
+ "<|reserved_special_token_147|>": 128179,
83
+ "<|reserved_special_token_148|>": 128180,
84
+ "<|reserved_special_token_149|>": 128181,
85
+ "<|reserved_special_token_14|>": 128046,
86
+ "<|reserved_special_token_150|>": 128182,
87
+ "<|reserved_special_token_151|>": 128183,
88
+ "<|reserved_special_token_152|>": 128184,
89
+ "<|reserved_special_token_153|>": 128185,
90
+ "<|reserved_special_token_154|>": 128186,
91
+ "<|reserved_special_token_155|>": 128187,
92
+ "<|reserved_special_token_156|>": 128188,
93
+ "<|reserved_special_token_157|>": 128189,
94
+ "<|reserved_special_token_158|>": 128190,
95
+ "<|reserved_special_token_159|>": 128191,
96
+ "<|reserved_special_token_15|>": 128047,
97
+ "<|reserved_special_token_160|>": 128192,
98
+ "<|reserved_special_token_161|>": 128193,
99
+ "<|reserved_special_token_162|>": 128194,
100
+ "<|reserved_special_token_163|>": 128195,
101
+ "<|reserved_special_token_164|>": 128196,
102
+ "<|reserved_special_token_165|>": 128197,
103
+ "<|reserved_special_token_166|>": 128198,
104
+ "<|reserved_special_token_167|>": 128199,
105
+ "<|reserved_special_token_168|>": 128200,
106
+ "<|reserved_special_token_169|>": 128201,
107
+ "<|reserved_special_token_16|>": 128048,
108
+ "<|reserved_special_token_170|>": 128202,
109
+ "<|reserved_special_token_171|>": 128203,
110
+ "<|reserved_special_token_172|>": 128204,
111
+ "<|reserved_special_token_173|>": 128205,
112
+ "<|reserved_special_token_174|>": 128206,
113
+ "<|reserved_special_token_175|>": 128207,
114
+ "<|reserved_special_token_176|>": 128208,
115
+ "<|reserved_special_token_177|>": 128209,
116
+ "<|reserved_special_token_178|>": 128210,
117
+ "<|reserved_special_token_179|>": 128211,
118
+ "<|reserved_special_token_17|>": 128049,
119
+ "<|reserved_special_token_180|>": 128212,
120
+ "<|reserved_special_token_181|>": 128213,
121
+ "<|reserved_special_token_182|>": 128214,
122
+ "<|reserved_special_token_183|>": 128215,
123
+ "<|reserved_special_token_184|>": 128216,
124
+ "<|reserved_special_token_185|>": 128217,
125
+ "<|reserved_special_token_186|>": 128218,
126
+ "<|reserved_special_token_187|>": 128219,
127
+ "<|reserved_special_token_188|>": 128220,
128
+ "<|reserved_special_token_189|>": 128221,
129
+ "<|reserved_special_token_18|>": 128050,
130
+ "<|reserved_special_token_190|>": 128222,
131
+ "<|reserved_special_token_191|>": 128223,
132
+ "<|reserved_special_token_192|>": 128224,
133
+ "<|reserved_special_token_193|>": 128225,
134
+ "<|reserved_special_token_194|>": 128226,
135
+ "<|reserved_special_token_195|>": 128227,
136
+ "<|reserved_special_token_196|>": 128228,
137
+ "<|reserved_special_token_197|>": 128229,
138
+ "<|reserved_special_token_198|>": 128230,
139
+ "<|reserved_special_token_199|>": 128231,
140
+ "<|reserved_special_token_19|>": 128051,
141
+ "<|reserved_special_token_1|>": 128033,
142
+ "<|reserved_special_token_200|>": 128232,
143
+ "<|reserved_special_token_201|>": 128233,
144
+ "<|reserved_special_token_202|>": 128234,
145
+ "<|reserved_special_token_203|>": 128235,
146
+ "<|reserved_special_token_204|>": 128236,
147
+ "<|reserved_special_token_205|>": 128237,
148
+ "<|reserved_special_token_206|>": 128238,
149
+ "<|reserved_special_token_207|>": 128239,
150
+ "<|reserved_special_token_208|>": 128240,
151
+ "<|reserved_special_token_209|>": 128241,
152
+ "<|reserved_special_token_20|>": 128052,
153
+ "<|reserved_special_token_210|>": 128242,
154
+ "<|reserved_special_token_211|>": 128243,
155
+ "<|reserved_special_token_212|>": 128244,
156
+ "<|reserved_special_token_213|>": 128245,
157
+ "<|reserved_special_token_214|>": 128246,
158
+ "<|reserved_special_token_215|>": 128247,
159
+ "<|reserved_special_token_216|>": 128248,
160
+ "<|reserved_special_token_217|>": 128249,
161
+ "<|reserved_special_token_218|>": 128250,
162
+ "<|reserved_special_token_219|>": 128251,
163
+ "<|reserved_special_token_21|>": 128053,
164
+ "<|reserved_special_token_220|>": 128252,
165
+ "<|reserved_special_token_221|>": 128253,
166
+ "<|reserved_special_token_222|>": 128254,
167
+ "<|reserved_special_token_223|>": 128255,
168
+ "<|reserved_special_token_22|>": 128054,
169
+ "<|reserved_special_token_23|>": 128055,
170
+ "<|reserved_special_token_24|>": 128056,
171
+ "<|reserved_special_token_25|>": 128057,
172
+ "<|reserved_special_token_26|>": 128058,
173
+ "<|reserved_special_token_27|>": 128059,
174
+ "<|reserved_special_token_28|>": 128060,
175
+ "<|reserved_special_token_29|>": 128061,
176
+ "<|reserved_special_token_2|>": 128034,
177
+ "<|reserved_special_token_30|>": 128062,
178
+ "<|reserved_special_token_31|>": 128063,
179
+ "<|reserved_special_token_32|>": 128064,
180
+ "<|reserved_special_token_33|>": 128065,
181
+ "<|reserved_special_token_34|>": 128066,
182
+ "<|reserved_special_token_35|>": 128067,
183
+ "<|reserved_special_token_36|>": 128068,
184
+ "<|reserved_special_token_37|>": 128069,
185
+ "<|reserved_special_token_38|>": 128070,
186
+ "<|reserved_special_token_39|>": 128071,
187
+ "<|reserved_special_token_3|>": 128035,
188
+ "<|reserved_special_token_40|>": 128072,
189
+ "<|reserved_special_token_41|>": 128073,
190
+ "<|reserved_special_token_42|>": 128074,
191
+ "<|reserved_special_token_43|>": 128075,
192
+ "<|reserved_special_token_44|>": 128076,
193
+ "<|reserved_special_token_45|>": 128077,
194
+ "<|reserved_special_token_46|>": 128078,
195
+ "<|reserved_special_token_47|>": 128079,
196
+ "<|reserved_special_token_48|>": 128080,
197
+ "<|reserved_special_token_49|>": 128081,
198
+ "<|reserved_special_token_4|>": 128036,
199
+ "<|reserved_special_token_50|>": 128082,
200
+ "<|reserved_special_token_51|>": 128083,
201
+ "<|reserved_special_token_52|>": 128084,
202
+ "<|reserved_special_token_53|>": 128085,
203
+ "<|reserved_special_token_54|>": 128086,
204
+ "<|reserved_special_token_55|>": 128087,
205
+ "<|reserved_special_token_56|>": 128088,
206
+ "<|reserved_special_token_57|>": 128089,
207
+ "<|reserved_special_token_58|>": 128090,
208
+ "<|reserved_special_token_59|>": 128091,
209
+ "<|reserved_special_token_5|>": 128037,
210
+ "<|reserved_special_token_60|>": 128092,
211
+ "<|reserved_special_token_61|>": 128093,
212
+ "<|reserved_special_token_62|>": 128094,
213
+ "<|reserved_special_token_63|>": 128095,
214
+ "<|reserved_special_token_64|>": 128096,
215
+ "<|reserved_special_token_65|>": 128097,
216
+ "<|reserved_special_token_66|>": 128098,
217
+ "<|reserved_special_token_67|>": 128099,
218
+ "<|reserved_special_token_68|>": 128100,
219
+ "<|reserved_special_token_69|>": 128101,
220
+ "<|reserved_special_token_6|>": 128038,
221
+ "<|reserved_special_token_70|>": 128102,
222
+ "<|reserved_special_token_71|>": 128103,
223
+ "<|reserved_special_token_72|>": 128104,
224
+ "<|reserved_special_token_73|>": 128105,
225
+ "<|reserved_special_token_74|>": 128106,
226
+ "<|reserved_special_token_75|>": 128107,
227
+ "<|reserved_special_token_76|>": 128108,
228
+ "<|reserved_special_token_77|>": 128109,
229
+ "<|reserved_special_token_78|>": 128110,
230
+ "<|reserved_special_token_79|>": 128111,
231
+ "<|reserved_special_token_7|>": 128039,
232
+ "<|reserved_special_token_80|>": 128112,
233
+ "<|reserved_special_token_81|>": 128113,
234
+ "<|reserved_special_token_82|>": 128114,
235
+ "<|reserved_special_token_83|>": 128115,
236
+ "<|reserved_special_token_84|>": 128116,
237
+ "<|reserved_special_token_85|>": 128117,
238
+ "<|reserved_special_token_86|>": 128118,
239
+ "<|reserved_special_token_87|>": 128119,
240
+ "<|reserved_special_token_88|>": 128120,
241
+ "<|reserved_special_token_89|>": 128121,
242
+ "<|reserved_special_token_8|>": 128040,
243
+ "<|reserved_special_token_90|>": 128122,
244
+ "<|reserved_special_token_91|>": 128123,
245
+ "<|reserved_special_token_92|>": 128124,
246
+ "<|reserved_special_token_93|>": 128125,
247
+ "<|reserved_special_token_94|>": 128126,
248
+ "<|reserved_special_token_95|>": 128127,
249
+ "<|reserved_special_token_96|>": 128128,
250
+ "<|reserved_special_token_97|>": 128129,
251
+ "<|reserved_special_token_98|>": 128130,
252
+ "<|reserved_special_token_99|>": 128131,
253
+ "<|reserved_special_token_9|>": 128041,
254
+ "<|video_pad|>": 128025,
255
+ "<|vision_end|>": 128022,
256
+ "<|vision_pad|>": 128023,
257
+ "<|vision_start|>": 128021
258
+ }
chat_template.jinja ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {# ───── defaults ───── #}
2
+ {%- if enable_thinking is not defined -%}
3
+ {%- set enable_thinking = true -%}
4
+ {%- endif -%}
5
+
6
+ {%- if not documents is defined -%}
7
+ {%- set documents = none -%}
8
+ {%- endif -%}
9
+
10
+ {%- if not tools is defined -%}
11
+ {%- set tools = none -%}
12
+ {%- endif -%}
13
+
14
+ {%- if not xml_tools is defined -%}
15
+ {%- set xml_tools = none -%}
16
+ {%- endif -%}
17
+
18
+ {%- if not python_tools is defined -%}
19
+ {%- set python_tools = none -%}
20
+ {%- endif -%}
21
+
22
+ {# ───── reasoning mode ───── #}
23
+ {%- if enable_thinking -%}
24
+ {%- set reasoning_mode = "/think" -%}
25
+ {%- else -%}
26
+ {%- set reasoning_mode = "/no_think" -%}
27
+ {%- endif -%}
28
+
29
+ {# ───── system header ───── #}
30
+ {%- if messages[0].role == 'system' -%}
31
+ {%- set system_message = messages[0].content -%}
32
+ {%- if "/no_think" in system_message -%}
33
+ {%- set reasoning_mode = "/no_think" -%}
34
+ {%- elif "/think" in system_message -%}
35
+ {%- set reasoning_mode = "/think" -%}
36
+ {%- endif -%}
37
+ {%- set custom_instructions = system_message.replace("/no_think", "").replace("/think", "").rstrip() -%}
38
+ {%- endif -%}
39
+
40
+ {# ───── build system message ───── #}
41
+ {{- '<|im_start|>system\n' -}}
42
+
43
+ {%- if "/system_override" in system_message -%}
44
+ {{- custom_instructions.replace("/system_override", "").rstrip() -}}
45
+ {{- "<|im_end|>\n" -}}
46
+ {%- else -%}
47
+ {{- "## Metadata\n\n" -}}
48
+ {{- "Knowledge Cutoff Date: June 2025\n" -}}
49
+ {%- set today = strftime_now("%d %B %Y") -%}
50
+ {{- "Today Date: " ~ today ~ "\n" -}}
51
+ {{- "Reasoning Mode: " + reasoning_mode + "\n\n" -}}
52
+
53
+ {{- "## Custom Instructions\n\n" -}}
54
+ {%- if custom_instructions -%}
55
+ {{- custom_instructions + "\n\n" -}}
56
+ {%- elif reasoning_mode == "/think" -%}
57
+ {{- "You are Doge, an AI assistant created by the SmallDoge Team. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> Thought section </think> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.\n\n" -}}
58
+ {%- else -%}
59
+ {{- "You are Doge, an AI assistant created by the SmallDoge Team.\n\n" -}}
60
+ {%- endif -%}
61
+
62
+ {# ───── Document Support ───── #}
63
+ {%- if documents is not none -%}
64
+ {{- "### Documents\n\n" -}}
65
+ {{- "You have access to the following documents to answer the user's questions:\n\n" -}}
66
+ {%- for doc in documents -%}
67
+ {%- if doc.title is defined -%}
68
+ {{- "**Title**: " ~ doc.title ~ "\n" -}}
69
+ {%- endif -%}
70
+ {{- "**Content**: " ~ doc.text ~ "\n\n" -}}
71
+ {%- endfor -%}
72
+ {{- "If the documents don't contain relevant information, rely on your general knowledge but acknowledge when you're doing so.\n\n" -}}
73
+ {%- endif -%}
74
+
75
+ {# ───── Tool Support ───── #}
76
+ {%- if xml_tools or python_tools or tools -%}
77
+ {{- "### Tools\n\n" -}}
78
+ {%- if xml_tools or tools -%}
79
+ {%- if tools -%}
80
+ {%- set xml_tools = tools -%}
81
+ {%- endif -%}
82
+ {%- set ns = namespace(xml_tool_string="You may call one or more functions to assist with the user query.\nYou are provided with function signatures within <tools></tools> XML tags:\n\n<tools>\n") -%}
83
+ {%- for tool in xml_tools[:] -%} {# The slicing makes sure that xml_tools is a list #}
84
+ {%- set ns.xml_tool_string = ns.xml_tool_string ~ (tool | string) ~ "\n" -%}
85
+ {%- endfor -%}
86
+ {%- set xml_tool_string = ns.xml_tool_string + "</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>" -%}
87
+ {{- xml_tool_string -}}
88
+ {%- endif -%}
89
+ {%- if python_tools -%}
90
+ {%- set ns = namespace(python_tool_string="When you send a message containing Python code between '<code>' and '</code>' tags, it will be executed in a stateful Jupyter notebook environment, and you will then be given the output to continued reasoning in an agentic loop.\n\nYou can use the following tools in your python code like regular functions:\n<tools>\n") -%}
91
+ {%- for tool in python_tools[:] -%} {# The slicing makes sure that python_tools is a list #}
92
+ {%- set ns.python_tool_string = ns.python_tool_string ~ (tool | string) ~ "\n" -%}
93
+ {%- endfor -%}
94
+ {%- set python_tool_string = ns.python_tool_string + "</tools>\n\nThe state persists between code executions: so variables that you define in one step are still available thereafter." -%}
95
+ {{- python_tool_string -}}
96
+ {%- endif -%}
97
+ {{- "\n\n" -}}
98
+ {%- endif -%}
99
+ {%- endif -%}
100
+
101
+ {{- '<|im_end|>\n' -}}
102
+
103
+ {# ───── Message loop ───── #}
104
+ {%- for message in messages -%}
105
+ {%- set content = message.content if message.content is string else "" -%}
106
+ {%- if message.role == "user" -%}
107
+ {{ "<|im_start|>" + message.role + "\n" + content + "<|im_end|>\n" }}
108
+ {%- elif message.role == "assistant" -%}
109
+ {% generation %}
110
+ {%- if reasoning_mode == "/think" -%}
111
+ {{ "<|im_start|>assistant\n" + content.lstrip("\n") + "<|im_end|>\n" }}
112
+ {%- else -%}
113
+ {{ "<|im_start|>assistant\n" + "<think>\n\n</think>\n" + content.lstrip("\n") + "<|im_end|>\n" }}
114
+ {%- endif -%}
115
+ {% endgeneration %}
116
+ {%- elif message.role == "tool" -%}
117
+ {{ "<|im_start|>" + "user\n" + content + "<|im_end|>\n" }}
118
+ {%- endif -%}
119
+ {%- endfor -%}
120
+
121
+ {# ───── Generation prompt ───── #}
122
+ {%- if add_generation_prompt -%}
123
+ {%- if enable_thinking -%}
124
+ {{- '<|im_start|>assistant\n' -}}
125
+ {%- else -%}
126
+ {{- '<|im_start|>assistant\n<think>\n\n</think>\n\n' -}}
127
+ {%- endif -%}
128
+ {%- endif -%}
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|begin_of_text|>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|im_end|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<|end_of_text|>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:47a2732154db3c5893e460b2ea810422106e59a0e5689d7285b118fb1eb02b04
3
+ size 17209485
tokenizer_config.json ADDED
@@ -0,0 +1,2069 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_prefix_space": false,
4
+ "added_tokens_decoder": {
5
+ "128000": {
6
+ "content": "<|begin_of_text|>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "128001": {
14
+ "content": "<|end_of_text|>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "128002": {
22
+ "content": "<|im_start|>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ },
29
+ "128003": {
30
+ "content": "<|im_end|>",
31
+ "lstrip": false,
32
+ "normalized": false,
33
+ "rstrip": false,
34
+ "single_word": false,
35
+ "special": true
36
+ },
37
+ "128004": {
38
+ "content": "<tool_call>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false,
43
+ "special": true
44
+ },
45
+ "128005": {
46
+ "content": "</tool_call>",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false,
51
+ "special": true
52
+ },
53
+ "128006": {
54
+ "content": "<tool_response>",
55
+ "lstrip": false,
56
+ "normalized": false,
57
+ "rstrip": false,
58
+ "single_word": false,
59
+ "special": true
60
+ },
61
+ "128007": {
62
+ "content": "</tool_response>",
63
+ "lstrip": false,
64
+ "normalized": false,
65
+ "rstrip": false,
66
+ "single_word": false,
67
+ "special": true
68
+ },
69
+ "128008": {
70
+ "content": "<think>",
71
+ "lstrip": false,
72
+ "normalized": false,
73
+ "rstrip": false,
74
+ "single_word": false,
75
+ "special": true
76
+ },
77
+ "128009": {
78
+ "content": "</think>",
79
+ "lstrip": false,
80
+ "normalized": false,
81
+ "rstrip": false,
82
+ "single_word": false,
83
+ "special": true
84
+ },
85
+ "128010": {
86
+ "content": "<code>",
87
+ "lstrip": false,
88
+ "normalized": false,
89
+ "rstrip": false,
90
+ "single_word": false,
91
+ "special": true
92
+ },
93
+ "128011": {
94
+ "content": "</code>",
95
+ "lstrip": false,
96
+ "normalized": false,
97
+ "rstrip": false,
98
+ "single_word": false,
99
+ "special": true
100
+ },
101
+ "128012": {
102
+ "content": "<citation>",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false,
107
+ "special": true
108
+ },
109
+ "128013": {
110
+ "content": "</citation>",
111
+ "lstrip": false,
112
+ "normalized": false,
113
+ "rstrip": false,
114
+ "single_word": false,
115
+ "special": true
116
+ },
117
+ "128014": {
118
+ "content": "<|conversation_title|>",
119
+ "lstrip": false,
120
+ "normalized": false,
121
+ "rstrip": false,
122
+ "single_word": false,
123
+ "special": true
124
+ },
125
+ "128015": {
126
+ "content": "<|object_ref_start|>",
127
+ "lstrip": false,
128
+ "normalized": false,
129
+ "rstrip": false,
130
+ "single_word": false,
131
+ "special": true
132
+ },
133
+ "128016": {
134
+ "content": "<|object_ref_end|>",
135
+ "lstrip": false,
136
+ "normalized": false,
137
+ "rstrip": false,
138
+ "single_word": false,
139
+ "special": true
140
+ },
141
+ "128017": {
142
+ "content": "<|box_start|>",
143
+ "lstrip": false,
144
+ "normalized": false,
145
+ "rstrip": false,
146
+ "single_word": false,
147
+ "special": true
148
+ },
149
+ "128018": {
150
+ "content": "<|box_end|>",
151
+ "lstrip": false,
152
+ "normalized": false,
153
+ "rstrip": false,
154
+ "single_word": false,
155
+ "special": true
156
+ },
157
+ "128019": {
158
+ "content": "<judgement>",
159
+ "lstrip": false,
160
+ "normalized": false,
161
+ "rstrip": false,
162
+ "single_word": false,
163
+ "special": true
164
+ },
165
+ "128020": {
166
+ "content": "</judgement>",
167
+ "lstrip": false,
168
+ "normalized": false,
169
+ "rstrip": false,
170
+ "single_word": false,
171
+ "special": true
172
+ },
173
+ "128021": {
174
+ "content": "<|vision_start|>",
175
+ "lstrip": false,
176
+ "normalized": false,
177
+ "rstrip": false,
178
+ "single_word": false,
179
+ "special": true
180
+ },
181
+ "128022": {
182
+ "content": "<|vision_end|>",
183
+ "lstrip": false,
184
+ "normalized": false,
185
+ "rstrip": false,
186
+ "single_word": false,
187
+ "special": true
188
+ },
189
+ "128023": {
190
+ "content": "<|vision_pad|>",
191
+ "lstrip": false,
192
+ "normalized": false,
193
+ "rstrip": false,
194
+ "single_word": false,
195
+ "special": true
196
+ },
197
+ "128024": {
198
+ "content": "<|image_pad|>",
199
+ "lstrip": false,
200
+ "normalized": false,
201
+ "rstrip": false,
202
+ "single_word": false,
203
+ "special": true
204
+ },
205
+ "128025": {
206
+ "content": "<|video_pad|>",
207
+ "lstrip": false,
208
+ "normalized": false,
209
+ "rstrip": false,
210
+ "single_word": false,
211
+ "special": true
212
+ },
213
+ "128026": {
214
+ "content": "<|fim_prefix|>",
215
+ "lstrip": false,
216
+ "normalized": false,
217
+ "rstrip": false,
218
+ "single_word": false,
219
+ "special": true
220
+ },
221
+ "128027": {
222
+ "content": "<|fim_middle|>",
223
+ "lstrip": false,
224
+ "normalized": false,
225
+ "rstrip": false,
226
+ "single_word": false,
227
+ "special": true
228
+ },
229
+ "128028": {
230
+ "content": "<|fim_suffix|>",
231
+ "lstrip": false,
232
+ "normalized": false,
233
+ "rstrip": false,
234
+ "single_word": false,
235
+ "special": true
236
+ },
237
+ "128029": {
238
+ "content": "<|fim_pad|>",
239
+ "lstrip": false,
240
+ "normalized": false,
241
+ "rstrip": false,
242
+ "single_word": false,
243
+ "special": true
244
+ },
245
+ "128030": {
246
+ "content": "<|repo_name|>",
247
+ "lstrip": false,
248
+ "normalized": false,
249
+ "rstrip": false,
250
+ "single_word": false,
251
+ "special": true
252
+ },
253
+ "128031": {
254
+ "content": "<|file_sep|>",
255
+ "lstrip": false,
256
+ "normalized": false,
257
+ "rstrip": false,
258
+ "single_word": false,
259
+ "special": true
260
+ },
261
+ "128032": {
262
+ "content": "<|reserved_special_token_0|>",
263
+ "lstrip": false,
264
+ "normalized": false,
265
+ "rstrip": false,
266
+ "single_word": false,
267
+ "special": true
268
+ },
269
+ "128033": {
270
+ "content": "<|reserved_special_token_1|>",
271
+ "lstrip": false,
272
+ "normalized": false,
273
+ "rstrip": false,
274
+ "single_word": false,
275
+ "special": true
276
+ },
277
+ "128034": {
278
+ "content": "<|reserved_special_token_2|>",
279
+ "lstrip": false,
280
+ "normalized": false,
281
+ "rstrip": false,
282
+ "single_word": false,
283
+ "special": true
284
+ },
285
+ "128035": {
286
+ "content": "<|reserved_special_token_3|>",
287
+ "lstrip": false,
288
+ "normalized": false,
289
+ "rstrip": false,
290
+ "single_word": false,
291
+ "special": true
292
+ },
293
+ "128036": {
294
+ "content": "<|reserved_special_token_4|>",
295
+ "lstrip": false,
296
+ "normalized": false,
297
+ "rstrip": false,
298
+ "single_word": false,
299
+ "special": true
300
+ },
301
+ "128037": {
302
+ "content": "<|reserved_special_token_5|>",
303
+ "lstrip": false,
304
+ "normalized": false,
305
+ "rstrip": false,
306
+ "single_word": false,
307
+ "special": true
308
+ },
309
+ "128038": {
310
+ "content": "<|reserved_special_token_6|>",
311
+ "lstrip": false,
312
+ "normalized": false,
313
+ "rstrip": false,
314
+ "single_word": false,
315
+ "special": true
316
+ },
317
+ "128039": {
318
+ "content": "<|reserved_special_token_7|>",
319
+ "lstrip": false,
320
+ "normalized": false,
321
+ "rstrip": false,
322
+ "single_word": false,
323
+ "special": true
324
+ },
325
+ "128040": {
326
+ "content": "<|reserved_special_token_8|>",
327
+ "lstrip": false,
328
+ "normalized": false,
329
+ "rstrip": false,
330
+ "single_word": false,
331
+ "special": true
332
+ },
333
+ "128041": {
334
+ "content": "<|reserved_special_token_9|>",
335
+ "lstrip": false,
336
+ "normalized": false,
337
+ "rstrip": false,
338
+ "single_word": false,
339
+ "special": true
340
+ },
341
+ "128042": {
342
+ "content": "<|reserved_special_token_10|>",
343
+ "lstrip": false,
344
+ "normalized": false,
345
+ "rstrip": false,
346
+ "single_word": false,
347
+ "special": true
348
+ },
349
+ "128043": {
350
+ "content": "<|reserved_special_token_11|>",
351
+ "lstrip": false,
352
+ "normalized": false,
353
+ "rstrip": false,
354
+ "single_word": false,
355
+ "special": true
356
+ },
357
+ "128044": {
358
+ "content": "<|reserved_special_token_12|>",
359
+ "lstrip": false,
360
+ "normalized": false,
361
+ "rstrip": false,
362
+ "single_word": false,
363
+ "special": true
364
+ },
365
+ "128045": {
366
+ "content": "<|reserved_special_token_13|>",
367
+ "lstrip": false,
368
+ "normalized": false,
369
+ "rstrip": false,
370
+ "single_word": false,
371
+ "special": true
372
+ },
373
+ "128046": {
374
+ "content": "<|reserved_special_token_14|>",
375
+ "lstrip": false,
376
+ "normalized": false,
377
+ "rstrip": false,
378
+ "single_word": false,
379
+ "special": true
380
+ },
381
+ "128047": {
382
+ "content": "<|reserved_special_token_15|>",
383
+ "lstrip": false,
384
+ "normalized": false,
385
+ "rstrip": false,
386
+ "single_word": false,
387
+ "special": true
388
+ },
389
+ "128048": {
390
+ "content": "<|reserved_special_token_16|>",
391
+ "lstrip": false,
392
+ "normalized": false,
393
+ "rstrip": false,
394
+ "single_word": false,
395
+ "special": true
396
+ },
397
+ "128049": {
398
+ "content": "<|reserved_special_token_17|>",
399
+ "lstrip": false,
400
+ "normalized": false,
401
+ "rstrip": false,
402
+ "single_word": false,
403
+ "special": true
404
+ },
405
+ "128050": {
406
+ "content": "<|reserved_special_token_18|>",
407
+ "lstrip": false,
408
+ "normalized": false,
409
+ "rstrip": false,
410
+ "single_word": false,
411
+ "special": true
412
+ },
413
+ "128051": {
414
+ "content": "<|reserved_special_token_19|>",
415
+ "lstrip": false,
416
+ "normalized": false,
417
+ "rstrip": false,
418
+ "single_word": false,
419
+ "special": true
420
+ },
421
+ "128052": {
422
+ "content": "<|reserved_special_token_20|>",
423
+ "lstrip": false,
424
+ "normalized": false,
425
+ "rstrip": false,
426
+ "single_word": false,
427
+ "special": true
428
+ },
429
+ "128053": {
430
+ "content": "<|reserved_special_token_21|>",
431
+ "lstrip": false,
432
+ "normalized": false,
433
+ "rstrip": false,
434
+ "single_word": false,
435
+ "special": true
436
+ },
437
+ "128054": {
438
+ "content": "<|reserved_special_token_22|>",
439
+ "lstrip": false,
440
+ "normalized": false,
441
+ "rstrip": false,
442
+ "single_word": false,
443
+ "special": true
444
+ },
445
+ "128055": {
446
+ "content": "<|reserved_special_token_23|>",
447
+ "lstrip": false,
448
+ "normalized": false,
449
+ "rstrip": false,
450
+ "single_word": false,
451
+ "special": true
452
+ },
453
+ "128056": {
454
+ "content": "<|reserved_special_token_24|>",
455
+ "lstrip": false,
456
+ "normalized": false,
457
+ "rstrip": false,
458
+ "single_word": false,
459
+ "special": true
460
+ },
461
+ "128057": {
462
+ "content": "<|reserved_special_token_25|>",
463
+ "lstrip": false,
464
+ "normalized": false,
465
+ "rstrip": false,
466
+ "single_word": false,
467
+ "special": true
468
+ },
469
+ "128058": {
470
+ "content": "<|reserved_special_token_26|>",
471
+ "lstrip": false,
472
+ "normalized": false,
473
+ "rstrip": false,
474
+ "single_word": false,
475
+ "special": true
476
+ },
477
+ "128059": {
478
+ "content": "<|reserved_special_token_27|>",
479
+ "lstrip": false,
480
+ "normalized": false,
481
+ "rstrip": false,
482
+ "single_word": false,
483
+ "special": true
484
+ },
485
+ "128060": {
486
+ "content": "<|reserved_special_token_28|>",
487
+ "lstrip": false,
488
+ "normalized": false,
489
+ "rstrip": false,
490
+ "single_word": false,
491
+ "special": true
492
+ },
493
+ "128061": {
494
+ "content": "<|reserved_special_token_29|>",
495
+ "lstrip": false,
496
+ "normalized": false,
497
+ "rstrip": false,
498
+ "single_word": false,
499
+ "special": true
500
+ },
501
+ "128062": {
502
+ "content": "<|reserved_special_token_30|>",
503
+ "lstrip": false,
504
+ "normalized": false,
505
+ "rstrip": false,
506
+ "single_word": false,
507
+ "special": true
508
+ },
509
+ "128063": {
510
+ "content": "<|reserved_special_token_31|>",
511
+ "lstrip": false,
512
+ "normalized": false,
513
+ "rstrip": false,
514
+ "single_word": false,
515
+ "special": true
516
+ },
517
+ "128064": {
518
+ "content": "<|reserved_special_token_32|>",
519
+ "lstrip": false,
520
+ "normalized": false,
521
+ "rstrip": false,
522
+ "single_word": false,
523
+ "special": true
524
+ },
525
+ "128065": {
526
+ "content": "<|reserved_special_token_33|>",
527
+ "lstrip": false,
528
+ "normalized": false,
529
+ "rstrip": false,
530
+ "single_word": false,
531
+ "special": true
532
+ },
533
+ "128066": {
534
+ "content": "<|reserved_special_token_34|>",
535
+ "lstrip": false,
536
+ "normalized": false,
537
+ "rstrip": false,
538
+ "single_word": false,
539
+ "special": true
540
+ },
541
+ "128067": {
542
+ "content": "<|reserved_special_token_35|>",
543
+ "lstrip": false,
544
+ "normalized": false,
545
+ "rstrip": false,
546
+ "single_word": false,
547
+ "special": true
548
+ },
549
+ "128068": {
550
+ "content": "<|reserved_special_token_36|>",
551
+ "lstrip": false,
552
+ "normalized": false,
553
+ "rstrip": false,
554
+ "single_word": false,
555
+ "special": true
556
+ },
557
+ "128069": {
558
+ "content": "<|reserved_special_token_37|>",
559
+ "lstrip": false,
560
+ "normalized": false,
561
+ "rstrip": false,
562
+ "single_word": false,
563
+ "special": true
564
+ },
565
+ "128070": {
566
+ "content": "<|reserved_special_token_38|>",
567
+ "lstrip": false,
568
+ "normalized": false,
569
+ "rstrip": false,
570
+ "single_word": false,
571
+ "special": true
572
+ },
573
+ "128071": {
574
+ "content": "<|reserved_special_token_39|>",
575
+ "lstrip": false,
576
+ "normalized": false,
577
+ "rstrip": false,
578
+ "single_word": false,
579
+ "special": true
580
+ },
581
+ "128072": {
582
+ "content": "<|reserved_special_token_40|>",
583
+ "lstrip": false,
584
+ "normalized": false,
585
+ "rstrip": false,
586
+ "single_word": false,
587
+ "special": true
588
+ },
589
+ "128073": {
590
+ "content": "<|reserved_special_token_41|>",
591
+ "lstrip": false,
592
+ "normalized": false,
593
+ "rstrip": false,
594
+ "single_word": false,
595
+ "special": true
596
+ },
597
+ "128074": {
598
+ "content": "<|reserved_special_token_42|>",
599
+ "lstrip": false,
600
+ "normalized": false,
601
+ "rstrip": false,
602
+ "single_word": false,
603
+ "special": true
604
+ },
605
+ "128075": {
606
+ "content": "<|reserved_special_token_43|>",
607
+ "lstrip": false,
608
+ "normalized": false,
609
+ "rstrip": false,
610
+ "single_word": false,
611
+ "special": true
612
+ },
613
+ "128076": {
614
+ "content": "<|reserved_special_token_44|>",
615
+ "lstrip": false,
616
+ "normalized": false,
617
+ "rstrip": false,
618
+ "single_word": false,
619
+ "special": true
620
+ },
621
+ "128077": {
622
+ "content": "<|reserved_special_token_45|>",
623
+ "lstrip": false,
624
+ "normalized": false,
625
+ "rstrip": false,
626
+ "single_word": false,
627
+ "special": true
628
+ },
629
+ "128078": {
630
+ "content": "<|reserved_special_token_46|>",
631
+ "lstrip": false,
632
+ "normalized": false,
633
+ "rstrip": false,
634
+ "single_word": false,
635
+ "special": true
636
+ },
637
+ "128079": {
638
+ "content": "<|reserved_special_token_47|>",
639
+ "lstrip": false,
640
+ "normalized": false,
641
+ "rstrip": false,
642
+ "single_word": false,
643
+ "special": true
644
+ },
645
+ "128080": {
646
+ "content": "<|reserved_special_token_48|>",
647
+ "lstrip": false,
648
+ "normalized": false,
649
+ "rstrip": false,
650
+ "single_word": false,
651
+ "special": true
652
+ },
653
+ "128081": {
654
+ "content": "<|reserved_special_token_49|>",
655
+ "lstrip": false,
656
+ "normalized": false,
657
+ "rstrip": false,
658
+ "single_word": false,
659
+ "special": true
660
+ },
661
+ "128082": {
662
+ "content": "<|reserved_special_token_50|>",
663
+ "lstrip": false,
664
+ "normalized": false,
665
+ "rstrip": false,
666
+ "single_word": false,
667
+ "special": true
668
+ },
669
+ "128083": {
670
+ "content": "<|reserved_special_token_51|>",
671
+ "lstrip": false,
672
+ "normalized": false,
673
+ "rstrip": false,
674
+ "single_word": false,
675
+ "special": true
676
+ },
677
+ "128084": {
678
+ "content": "<|reserved_special_token_52|>",
679
+ "lstrip": false,
680
+ "normalized": false,
681
+ "rstrip": false,
682
+ "single_word": false,
683
+ "special": true
684
+ },
685
+ "128085": {
686
+ "content": "<|reserved_special_token_53|>",
687
+ "lstrip": false,
688
+ "normalized": false,
689
+ "rstrip": false,
690
+ "single_word": false,
691
+ "special": true
692
+ },
693
+ "128086": {
694
+ "content": "<|reserved_special_token_54|>",
695
+ "lstrip": false,
696
+ "normalized": false,
697
+ "rstrip": false,
698
+ "single_word": false,
699
+ "special": true
700
+ },
701
+ "128087": {
702
+ "content": "<|reserved_special_token_55|>",
703
+ "lstrip": false,
704
+ "normalized": false,
705
+ "rstrip": false,
706
+ "single_word": false,
707
+ "special": true
708
+ },
709
+ "128088": {
710
+ "content": "<|reserved_special_token_56|>",
711
+ "lstrip": false,
712
+ "normalized": false,
713
+ "rstrip": false,
714
+ "single_word": false,
715
+ "special": true
716
+ },
717
+ "128089": {
718
+ "content": "<|reserved_special_token_57|>",
719
+ "lstrip": false,
720
+ "normalized": false,
721
+ "rstrip": false,
722
+ "single_word": false,
723
+ "special": true
724
+ },
725
+ "128090": {
726
+ "content": "<|reserved_special_token_58|>",
727
+ "lstrip": false,
728
+ "normalized": false,
729
+ "rstrip": false,
730
+ "single_word": false,
731
+ "special": true
732
+ },
733
+ "128091": {
734
+ "content": "<|reserved_special_token_59|>",
735
+ "lstrip": false,
736
+ "normalized": false,
737
+ "rstrip": false,
738
+ "single_word": false,
739
+ "special": true
740
+ },
741
+ "128092": {
742
+ "content": "<|reserved_special_token_60|>",
743
+ "lstrip": false,
744
+ "normalized": false,
745
+ "rstrip": false,
746
+ "single_word": false,
747
+ "special": true
748
+ },
749
+ "128093": {
750
+ "content": "<|reserved_special_token_61|>",
751
+ "lstrip": false,
752
+ "normalized": false,
753
+ "rstrip": false,
754
+ "single_word": false,
755
+ "special": true
756
+ },
757
+ "128094": {
758
+ "content": "<|reserved_special_token_62|>",
759
+ "lstrip": false,
760
+ "normalized": false,
761
+ "rstrip": false,
762
+ "single_word": false,
763
+ "special": true
764
+ },
765
+ "128095": {
766
+ "content": "<|reserved_special_token_63|>",
767
+ "lstrip": false,
768
+ "normalized": false,
769
+ "rstrip": false,
770
+ "single_word": false,
771
+ "special": true
772
+ },
773
+ "128096": {
774
+ "content": "<|reserved_special_token_64|>",
775
+ "lstrip": false,
776
+ "normalized": false,
777
+ "rstrip": false,
778
+ "single_word": false,
779
+ "special": true
780
+ },
781
+ "128097": {
782
+ "content": "<|reserved_special_token_65|>",
783
+ "lstrip": false,
784
+ "normalized": false,
785
+ "rstrip": false,
786
+ "single_word": false,
787
+ "special": true
788
+ },
789
+ "128098": {
790
+ "content": "<|reserved_special_token_66|>",
791
+ "lstrip": false,
792
+ "normalized": false,
793
+ "rstrip": false,
794
+ "single_word": false,
795
+ "special": true
796
+ },
797
+ "128099": {
798
+ "content": "<|reserved_special_token_67|>",
799
+ "lstrip": false,
800
+ "normalized": false,
801
+ "rstrip": false,
802
+ "single_word": false,
803
+ "special": true
804
+ },
805
+ "128100": {
806
+ "content": "<|reserved_special_token_68|>",
807
+ "lstrip": false,
808
+ "normalized": false,
809
+ "rstrip": false,
810
+ "single_word": false,
811
+ "special": true
812
+ },
813
+ "128101": {
814
+ "content": "<|reserved_special_token_69|>",
815
+ "lstrip": false,
816
+ "normalized": false,
817
+ "rstrip": false,
818
+ "single_word": false,
819
+ "special": true
820
+ },
821
+ "128102": {
822
+ "content": "<|reserved_special_token_70|>",
823
+ "lstrip": false,
824
+ "normalized": false,
825
+ "rstrip": false,
826
+ "single_word": false,
827
+ "special": true
828
+ },
829
+ "128103": {
830
+ "content": "<|reserved_special_token_71|>",
831
+ "lstrip": false,
832
+ "normalized": false,
833
+ "rstrip": false,
834
+ "single_word": false,
835
+ "special": true
836
+ },
837
+ "128104": {
838
+ "content": "<|reserved_special_token_72|>",
839
+ "lstrip": false,
840
+ "normalized": false,
841
+ "rstrip": false,
842
+ "single_word": false,
843
+ "special": true
844
+ },
845
+ "128105": {
846
+ "content": "<|reserved_special_token_73|>",
847
+ "lstrip": false,
848
+ "normalized": false,
849
+ "rstrip": false,
850
+ "single_word": false,
851
+ "special": true
852
+ },
853
+ "128106": {
854
+ "content": "<|reserved_special_token_74|>",
855
+ "lstrip": false,
856
+ "normalized": false,
857
+ "rstrip": false,
858
+ "single_word": false,
859
+ "special": true
860
+ },
861
+ "128107": {
862
+ "content": "<|reserved_special_token_75|>",
863
+ "lstrip": false,
864
+ "normalized": false,
865
+ "rstrip": false,
866
+ "single_word": false,
867
+ "special": true
868
+ },
869
+ "128108": {
870
+ "content": "<|reserved_special_token_76|>",
871
+ "lstrip": false,
872
+ "normalized": false,
873
+ "rstrip": false,
874
+ "single_word": false,
875
+ "special": true
876
+ },
877
+ "128109": {
878
+ "content": "<|reserved_special_token_77|>",
879
+ "lstrip": false,
880
+ "normalized": false,
881
+ "rstrip": false,
882
+ "single_word": false,
883
+ "special": true
884
+ },
885
+ "128110": {
886
+ "content": "<|reserved_special_token_78|>",
887
+ "lstrip": false,
888
+ "normalized": false,
889
+ "rstrip": false,
890
+ "single_word": false,
891
+ "special": true
892
+ },
893
+ "128111": {
894
+ "content": "<|reserved_special_token_79|>",
895
+ "lstrip": false,
896
+ "normalized": false,
897
+ "rstrip": false,
898
+ "single_word": false,
899
+ "special": true
900
+ },
901
+ "128112": {
902
+ "content": "<|reserved_special_token_80|>",
903
+ "lstrip": false,
904
+ "normalized": false,
905
+ "rstrip": false,
906
+ "single_word": false,
907
+ "special": true
908
+ },
909
+ "128113": {
910
+ "content": "<|reserved_special_token_81|>",
911
+ "lstrip": false,
912
+ "normalized": false,
913
+ "rstrip": false,
914
+ "single_word": false,
915
+ "special": true
916
+ },
917
+ "128114": {
918
+ "content": "<|reserved_special_token_82|>",
919
+ "lstrip": false,
920
+ "normalized": false,
921
+ "rstrip": false,
922
+ "single_word": false,
923
+ "special": true
924
+ },
925
+ "128115": {
926
+ "content": "<|reserved_special_token_83|>",
927
+ "lstrip": false,
928
+ "normalized": false,
929
+ "rstrip": false,
930
+ "single_word": false,
931
+ "special": true
932
+ },
933
+ "128116": {
934
+ "content": "<|reserved_special_token_84|>",
935
+ "lstrip": false,
936
+ "normalized": false,
937
+ "rstrip": false,
938
+ "single_word": false,
939
+ "special": true
940
+ },
941
+ "128117": {
942
+ "content": "<|reserved_special_token_85|>",
943
+ "lstrip": false,
944
+ "normalized": false,
945
+ "rstrip": false,
946
+ "single_word": false,
947
+ "special": true
948
+ },
949
+ "128118": {
950
+ "content": "<|reserved_special_token_86|>",
951
+ "lstrip": false,
952
+ "normalized": false,
953
+ "rstrip": false,
954
+ "single_word": false,
955
+ "special": true
956
+ },
957
+ "128119": {
958
+ "content": "<|reserved_special_token_87|>",
959
+ "lstrip": false,
960
+ "normalized": false,
961
+ "rstrip": false,
962
+ "single_word": false,
963
+ "special": true
964
+ },
965
+ "128120": {
966
+ "content": "<|reserved_special_token_88|>",
967
+ "lstrip": false,
968
+ "normalized": false,
969
+ "rstrip": false,
970
+ "single_word": false,
971
+ "special": true
972
+ },
973
+ "128121": {
974
+ "content": "<|reserved_special_token_89|>",
975
+ "lstrip": false,
976
+ "normalized": false,
977
+ "rstrip": false,
978
+ "single_word": false,
979
+ "special": true
980
+ },
981
+ "128122": {
982
+ "content": "<|reserved_special_token_90|>",
983
+ "lstrip": false,
984
+ "normalized": false,
985
+ "rstrip": false,
986
+ "single_word": false,
987
+ "special": true
988
+ },
989
+ "128123": {
990
+ "content": "<|reserved_special_token_91|>",
991
+ "lstrip": false,
992
+ "normalized": false,
993
+ "rstrip": false,
994
+ "single_word": false,
995
+ "special": true
996
+ },
997
+ "128124": {
998
+ "content": "<|reserved_special_token_92|>",
999
+ "lstrip": false,
1000
+ "normalized": false,
1001
+ "rstrip": false,
1002
+ "single_word": false,
1003
+ "special": true
1004
+ },
1005
+ "128125": {
1006
+ "content": "<|reserved_special_token_93|>",
1007
+ "lstrip": false,
1008
+ "normalized": false,
1009
+ "rstrip": false,
1010
+ "single_word": false,
1011
+ "special": true
1012
+ },
1013
+ "128126": {
1014
+ "content": "<|reserved_special_token_94|>",
1015
+ "lstrip": false,
1016
+ "normalized": false,
1017
+ "rstrip": false,
1018
+ "single_word": false,
1019
+ "special": true
1020
+ },
1021
+ "128127": {
1022
+ "content": "<|reserved_special_token_95|>",
1023
+ "lstrip": false,
1024
+ "normalized": false,
1025
+ "rstrip": false,
1026
+ "single_word": false,
1027
+ "special": true
1028
+ },
1029
+ "128128": {
1030
+ "content": "<|reserved_special_token_96|>",
1031
+ "lstrip": false,
1032
+ "normalized": false,
1033
+ "rstrip": false,
1034
+ "single_word": false,
1035
+ "special": true
1036
+ },
1037
+ "128129": {
1038
+ "content": "<|reserved_special_token_97|>",
1039
+ "lstrip": false,
1040
+ "normalized": false,
1041
+ "rstrip": false,
1042
+ "single_word": false,
1043
+ "special": true
1044
+ },
1045
+ "128130": {
1046
+ "content": "<|reserved_special_token_98|>",
1047
+ "lstrip": false,
1048
+ "normalized": false,
1049
+ "rstrip": false,
1050
+ "single_word": false,
1051
+ "special": true
1052
+ },
1053
+ "128131": {
1054
+ "content": "<|reserved_special_token_99|>",
1055
+ "lstrip": false,
1056
+ "normalized": false,
1057
+ "rstrip": false,
1058
+ "single_word": false,
1059
+ "special": true
1060
+ },
1061
+ "128132": {
1062
+ "content": "<|reserved_special_token_100|>",
1063
+ "lstrip": false,
1064
+ "normalized": false,
1065
+ "rstrip": false,
1066
+ "single_word": false,
1067
+ "special": true
1068
+ },
1069
+ "128133": {
1070
+ "content": "<|reserved_special_token_101|>",
1071
+ "lstrip": false,
1072
+ "normalized": false,
1073
+ "rstrip": false,
1074
+ "single_word": false,
1075
+ "special": true
1076
+ },
1077
+ "128134": {
1078
+ "content": "<|reserved_special_token_102|>",
1079
+ "lstrip": false,
1080
+ "normalized": false,
1081
+ "rstrip": false,
1082
+ "single_word": false,
1083
+ "special": true
1084
+ },
1085
+ "128135": {
1086
+ "content": "<|reserved_special_token_103|>",
1087
+ "lstrip": false,
1088
+ "normalized": false,
1089
+ "rstrip": false,
1090
+ "single_word": false,
1091
+ "special": true
1092
+ },
1093
+ "128136": {
1094
+ "content": "<|reserved_special_token_104|>",
1095
+ "lstrip": false,
1096
+ "normalized": false,
1097
+ "rstrip": false,
1098
+ "single_word": false,
1099
+ "special": true
1100
+ },
1101
+ "128137": {
1102
+ "content": "<|reserved_special_token_105|>",
1103
+ "lstrip": false,
1104
+ "normalized": false,
1105
+ "rstrip": false,
1106
+ "single_word": false,
1107
+ "special": true
1108
+ },
1109
+ "128138": {
1110
+ "content": "<|reserved_special_token_106|>",
1111
+ "lstrip": false,
1112
+ "normalized": false,
1113
+ "rstrip": false,
1114
+ "single_word": false,
1115
+ "special": true
1116
+ },
1117
+ "128139": {
1118
+ "content": "<|reserved_special_token_107|>",
1119
+ "lstrip": false,
1120
+ "normalized": false,
1121
+ "rstrip": false,
1122
+ "single_word": false,
1123
+ "special": true
1124
+ },
1125
+ "128140": {
1126
+ "content": "<|reserved_special_token_108|>",
1127
+ "lstrip": false,
1128
+ "normalized": false,
1129
+ "rstrip": false,
1130
+ "single_word": false,
1131
+ "special": true
1132
+ },
1133
+ "128141": {
1134
+ "content": "<|reserved_special_token_109|>",
1135
+ "lstrip": false,
1136
+ "normalized": false,
1137
+ "rstrip": false,
1138
+ "single_word": false,
1139
+ "special": true
1140
+ },
1141
+ "128142": {
1142
+ "content": "<|reserved_special_token_110|>",
1143
+ "lstrip": false,
1144
+ "normalized": false,
1145
+ "rstrip": false,
1146
+ "single_word": false,
1147
+ "special": true
1148
+ },
1149
+ "128143": {
1150
+ "content": "<|reserved_special_token_111|>",
1151
+ "lstrip": false,
1152
+ "normalized": false,
1153
+ "rstrip": false,
1154
+ "single_word": false,
1155
+ "special": true
1156
+ },
1157
+ "128144": {
1158
+ "content": "<|reserved_special_token_112|>",
1159
+ "lstrip": false,
1160
+ "normalized": false,
1161
+ "rstrip": false,
1162
+ "single_word": false,
1163
+ "special": true
1164
+ },
1165
+ "128145": {
1166
+ "content": "<|reserved_special_token_113|>",
1167
+ "lstrip": false,
1168
+ "normalized": false,
1169
+ "rstrip": false,
1170
+ "single_word": false,
1171
+ "special": true
1172
+ },
1173
+ "128146": {
1174
+ "content": "<|reserved_special_token_114|>",
1175
+ "lstrip": false,
1176
+ "normalized": false,
1177
+ "rstrip": false,
1178
+ "single_word": false,
1179
+ "special": true
1180
+ },
1181
+ "128147": {
1182
+ "content": "<|reserved_special_token_115|>",
1183
+ "lstrip": false,
1184
+ "normalized": false,
1185
+ "rstrip": false,
1186
+ "single_word": false,
1187
+ "special": true
1188
+ },
1189
+ "128148": {
1190
+ "content": "<|reserved_special_token_116|>",
1191
+ "lstrip": false,
1192
+ "normalized": false,
1193
+ "rstrip": false,
1194
+ "single_word": false,
1195
+ "special": true
1196
+ },
1197
+ "128149": {
1198
+ "content": "<|reserved_special_token_117|>",
1199
+ "lstrip": false,
1200
+ "normalized": false,
1201
+ "rstrip": false,
1202
+ "single_word": false,
1203
+ "special": true
1204
+ },
1205
+ "128150": {
1206
+ "content": "<|reserved_special_token_118|>",
1207
+ "lstrip": false,
1208
+ "normalized": false,
1209
+ "rstrip": false,
1210
+ "single_word": false,
1211
+ "special": true
1212
+ },
1213
+ "128151": {
1214
+ "content": "<|reserved_special_token_119|>",
1215
+ "lstrip": false,
1216
+ "normalized": false,
1217
+ "rstrip": false,
1218
+ "single_word": false,
1219
+ "special": true
1220
+ },
1221
+ "128152": {
1222
+ "content": "<|reserved_special_token_120|>",
1223
+ "lstrip": false,
1224
+ "normalized": false,
1225
+ "rstrip": false,
1226
+ "single_word": false,
1227
+ "special": true
1228
+ },
1229
+ "128153": {
1230
+ "content": "<|reserved_special_token_121|>",
1231
+ "lstrip": false,
1232
+ "normalized": false,
1233
+ "rstrip": false,
1234
+ "single_word": false,
1235
+ "special": true
1236
+ },
1237
+ "128154": {
1238
+ "content": "<|reserved_special_token_122|>",
1239
+ "lstrip": false,
1240
+ "normalized": false,
1241
+ "rstrip": false,
1242
+ "single_word": false,
1243
+ "special": true
1244
+ },
1245
+ "128155": {
1246
+ "content": "<|reserved_special_token_123|>",
1247
+ "lstrip": false,
1248
+ "normalized": false,
1249
+ "rstrip": false,
1250
+ "single_word": false,
1251
+ "special": true
1252
+ },
1253
+ "128156": {
1254
+ "content": "<|reserved_special_token_124|>",
1255
+ "lstrip": false,
1256
+ "normalized": false,
1257
+ "rstrip": false,
1258
+ "single_word": false,
1259
+ "special": true
1260
+ },
1261
+ "128157": {
1262
+ "content": "<|reserved_special_token_125|>",
1263
+ "lstrip": false,
1264
+ "normalized": false,
1265
+ "rstrip": false,
1266
+ "single_word": false,
1267
+ "special": true
1268
+ },
1269
+ "128158": {
1270
+ "content": "<|reserved_special_token_126|>",
1271
+ "lstrip": false,
1272
+ "normalized": false,
1273
+ "rstrip": false,
1274
+ "single_word": false,
1275
+ "special": true
1276
+ },
1277
+ "128159": {
1278
+ "content": "<|reserved_special_token_127|>",
1279
+ "lstrip": false,
1280
+ "normalized": false,
1281
+ "rstrip": false,
1282
+ "single_word": false,
1283
+ "special": true
1284
+ },
1285
+ "128160": {
1286
+ "content": "<|reserved_special_token_128|>",
1287
+ "lstrip": false,
1288
+ "normalized": false,
1289
+ "rstrip": false,
1290
+ "single_word": false,
1291
+ "special": true
1292
+ },
1293
+ "128161": {
1294
+ "content": "<|reserved_special_token_129|>",
1295
+ "lstrip": false,
1296
+ "normalized": false,
1297
+ "rstrip": false,
1298
+ "single_word": false,
1299
+ "special": true
1300
+ },
1301
+ "128162": {
1302
+ "content": "<|reserved_special_token_130|>",
1303
+ "lstrip": false,
1304
+ "normalized": false,
1305
+ "rstrip": false,
1306
+ "single_word": false,
1307
+ "special": true
1308
+ },
1309
+ "128163": {
1310
+ "content": "<|reserved_special_token_131|>",
1311
+ "lstrip": false,
1312
+ "normalized": false,
1313
+ "rstrip": false,
1314
+ "single_word": false,
1315
+ "special": true
1316
+ },
1317
+ "128164": {
1318
+ "content": "<|reserved_special_token_132|>",
1319
+ "lstrip": false,
1320
+ "normalized": false,
1321
+ "rstrip": false,
1322
+ "single_word": false,
1323
+ "special": true
1324
+ },
1325
+ "128165": {
1326
+ "content": "<|reserved_special_token_133|>",
1327
+ "lstrip": false,
1328
+ "normalized": false,
1329
+ "rstrip": false,
1330
+ "single_word": false,
1331
+ "special": true
1332
+ },
1333
+ "128166": {
1334
+ "content": "<|reserved_special_token_134|>",
1335
+ "lstrip": false,
1336
+ "normalized": false,
1337
+ "rstrip": false,
1338
+ "single_word": false,
1339
+ "special": true
1340
+ },
1341
+ "128167": {
1342
+ "content": "<|reserved_special_token_135|>",
1343
+ "lstrip": false,
1344
+ "normalized": false,
1345
+ "rstrip": false,
1346
+ "single_word": false,
1347
+ "special": true
1348
+ },
1349
+ "128168": {
1350
+ "content": "<|reserved_special_token_136|>",
1351
+ "lstrip": false,
1352
+ "normalized": false,
1353
+ "rstrip": false,
1354
+ "single_word": false,
1355
+ "special": true
1356
+ },
1357
+ "128169": {
1358
+ "content": "<|reserved_special_token_137|>",
1359
+ "lstrip": false,
1360
+ "normalized": false,
1361
+ "rstrip": false,
1362
+ "single_word": false,
1363
+ "special": true
1364
+ },
1365
+ "128170": {
1366
+ "content": "<|reserved_special_token_138|>",
1367
+ "lstrip": false,
1368
+ "normalized": false,
1369
+ "rstrip": false,
1370
+ "single_word": false,
1371
+ "special": true
1372
+ },
1373
+ "128171": {
1374
+ "content": "<|reserved_special_token_139|>",
1375
+ "lstrip": false,
1376
+ "normalized": false,
1377
+ "rstrip": false,
1378
+ "single_word": false,
1379
+ "special": true
1380
+ },
1381
+ "128172": {
1382
+ "content": "<|reserved_special_token_140|>",
1383
+ "lstrip": false,
1384
+ "normalized": false,
1385
+ "rstrip": false,
1386
+ "single_word": false,
1387
+ "special": true
1388
+ },
1389
+ "128173": {
1390
+ "content": "<|reserved_special_token_141|>",
1391
+ "lstrip": false,
1392
+ "normalized": false,
1393
+ "rstrip": false,
1394
+ "single_word": false,
1395
+ "special": true
1396
+ },
1397
+ "128174": {
1398
+ "content": "<|reserved_special_token_142|>",
1399
+ "lstrip": false,
1400
+ "normalized": false,
1401
+ "rstrip": false,
1402
+ "single_word": false,
1403
+ "special": true
1404
+ },
1405
+ "128175": {
1406
+ "content": "<|reserved_special_token_143|>",
1407
+ "lstrip": false,
1408
+ "normalized": false,
1409
+ "rstrip": false,
1410
+ "single_word": false,
1411
+ "special": true
1412
+ },
1413
+ "128176": {
1414
+ "content": "<|reserved_special_token_144|>",
1415
+ "lstrip": false,
1416
+ "normalized": false,
1417
+ "rstrip": false,
1418
+ "single_word": false,
1419
+ "special": true
1420
+ },
1421
+ "128177": {
1422
+ "content": "<|reserved_special_token_145|>",
1423
+ "lstrip": false,
1424
+ "normalized": false,
1425
+ "rstrip": false,
1426
+ "single_word": false,
1427
+ "special": true
1428
+ },
1429
+ "128178": {
1430
+ "content": "<|reserved_special_token_146|>",
1431
+ "lstrip": false,
1432
+ "normalized": false,
1433
+ "rstrip": false,
1434
+ "single_word": false,
1435
+ "special": true
1436
+ },
1437
+ "128179": {
1438
+ "content": "<|reserved_special_token_147|>",
1439
+ "lstrip": false,
1440
+ "normalized": false,
1441
+ "rstrip": false,
1442
+ "single_word": false,
1443
+ "special": true
1444
+ },
1445
+ "128180": {
1446
+ "content": "<|reserved_special_token_148|>",
1447
+ "lstrip": false,
1448
+ "normalized": false,
1449
+ "rstrip": false,
1450
+ "single_word": false,
1451
+ "special": true
1452
+ },
1453
+ "128181": {
1454
+ "content": "<|reserved_special_token_149|>",
1455
+ "lstrip": false,
1456
+ "normalized": false,
1457
+ "rstrip": false,
1458
+ "single_word": false,
1459
+ "special": true
1460
+ },
1461
+ "128182": {
1462
+ "content": "<|reserved_special_token_150|>",
1463
+ "lstrip": false,
1464
+ "normalized": false,
1465
+ "rstrip": false,
1466
+ "single_word": false,
1467
+ "special": true
1468
+ },
1469
+ "128183": {
1470
+ "content": "<|reserved_special_token_151|>",
1471
+ "lstrip": false,
1472
+ "normalized": false,
1473
+ "rstrip": false,
1474
+ "single_word": false,
1475
+ "special": true
1476
+ },
1477
+ "128184": {
1478
+ "content": "<|reserved_special_token_152|>",
1479
+ "lstrip": false,
1480
+ "normalized": false,
1481
+ "rstrip": false,
1482
+ "single_word": false,
1483
+ "special": true
1484
+ },
1485
+ "128185": {
1486
+ "content": "<|reserved_special_token_153|>",
1487
+ "lstrip": false,
1488
+ "normalized": false,
1489
+ "rstrip": false,
1490
+ "single_word": false,
1491
+ "special": true
1492
+ },
1493
+ "128186": {
1494
+ "content": "<|reserved_special_token_154|>",
1495
+ "lstrip": false,
1496
+ "normalized": false,
1497
+ "rstrip": false,
1498
+ "single_word": false,
1499
+ "special": true
1500
+ },
1501
+ "128187": {
1502
+ "content": "<|reserved_special_token_155|>",
1503
+ "lstrip": false,
1504
+ "normalized": false,
1505
+ "rstrip": false,
1506
+ "single_word": false,
1507
+ "special": true
1508
+ },
1509
+ "128188": {
1510
+ "content": "<|reserved_special_token_156|>",
1511
+ "lstrip": false,
1512
+ "normalized": false,
1513
+ "rstrip": false,
1514
+ "single_word": false,
1515
+ "special": true
1516
+ },
1517
+ "128189": {
1518
+ "content": "<|reserved_special_token_157|>",
1519
+ "lstrip": false,
1520
+ "normalized": false,
1521
+ "rstrip": false,
1522
+ "single_word": false,
1523
+ "special": true
1524
+ },
1525
+ "128190": {
1526
+ "content": "<|reserved_special_token_158|>",
1527
+ "lstrip": false,
1528
+ "normalized": false,
1529
+ "rstrip": false,
1530
+ "single_word": false,
1531
+ "special": true
1532
+ },
1533
+ "128191": {
1534
+ "content": "<|reserved_special_token_159|>",
1535
+ "lstrip": false,
1536
+ "normalized": false,
1537
+ "rstrip": false,
1538
+ "single_word": false,
1539
+ "special": true
1540
+ },
1541
+ "128192": {
1542
+ "content": "<|reserved_special_token_160|>",
1543
+ "lstrip": false,
1544
+ "normalized": false,
1545
+ "rstrip": false,
1546
+ "single_word": false,
1547
+ "special": true
1548
+ },
1549
+ "128193": {
1550
+ "content": "<|reserved_special_token_161|>",
1551
+ "lstrip": false,
1552
+ "normalized": false,
1553
+ "rstrip": false,
1554
+ "single_word": false,
1555
+ "special": true
1556
+ },
1557
+ "128194": {
1558
+ "content": "<|reserved_special_token_162|>",
1559
+ "lstrip": false,
1560
+ "normalized": false,
1561
+ "rstrip": false,
1562
+ "single_word": false,
1563
+ "special": true
1564
+ },
1565
+ "128195": {
1566
+ "content": "<|reserved_special_token_163|>",
1567
+ "lstrip": false,
1568
+ "normalized": false,
1569
+ "rstrip": false,
1570
+ "single_word": false,
1571
+ "special": true
1572
+ },
1573
+ "128196": {
1574
+ "content": "<|reserved_special_token_164|>",
1575
+ "lstrip": false,
1576
+ "normalized": false,
1577
+ "rstrip": false,
1578
+ "single_word": false,
1579
+ "special": true
1580
+ },
1581
+ "128197": {
1582
+ "content": "<|reserved_special_token_165|>",
1583
+ "lstrip": false,
1584
+ "normalized": false,
1585
+ "rstrip": false,
1586
+ "single_word": false,
1587
+ "special": true
1588
+ },
1589
+ "128198": {
1590
+ "content": "<|reserved_special_token_166|>",
1591
+ "lstrip": false,
1592
+ "normalized": false,
1593
+ "rstrip": false,
1594
+ "single_word": false,
1595
+ "special": true
1596
+ },
1597
+ "128199": {
1598
+ "content": "<|reserved_special_token_167|>",
1599
+ "lstrip": false,
1600
+ "normalized": false,
1601
+ "rstrip": false,
1602
+ "single_word": false,
1603
+ "special": true
1604
+ },
1605
+ "128200": {
1606
+ "content": "<|reserved_special_token_168|>",
1607
+ "lstrip": false,
1608
+ "normalized": false,
1609
+ "rstrip": false,
1610
+ "single_word": false,
1611
+ "special": true
1612
+ },
1613
+ "128201": {
1614
+ "content": "<|reserved_special_token_169|>",
1615
+ "lstrip": false,
1616
+ "normalized": false,
1617
+ "rstrip": false,
1618
+ "single_word": false,
1619
+ "special": true
1620
+ },
1621
+ "128202": {
1622
+ "content": "<|reserved_special_token_170|>",
1623
+ "lstrip": false,
1624
+ "normalized": false,
1625
+ "rstrip": false,
1626
+ "single_word": false,
1627
+ "special": true
1628
+ },
1629
+ "128203": {
1630
+ "content": "<|reserved_special_token_171|>",
1631
+ "lstrip": false,
1632
+ "normalized": false,
1633
+ "rstrip": false,
1634
+ "single_word": false,
1635
+ "special": true
1636
+ },
1637
+ "128204": {
1638
+ "content": "<|reserved_special_token_172|>",
1639
+ "lstrip": false,
1640
+ "normalized": false,
1641
+ "rstrip": false,
1642
+ "single_word": false,
1643
+ "special": true
1644
+ },
1645
+ "128205": {
1646
+ "content": "<|reserved_special_token_173|>",
1647
+ "lstrip": false,
1648
+ "normalized": false,
1649
+ "rstrip": false,
1650
+ "single_word": false,
1651
+ "special": true
1652
+ },
1653
+ "128206": {
1654
+ "content": "<|reserved_special_token_174|>",
1655
+ "lstrip": false,
1656
+ "normalized": false,
1657
+ "rstrip": false,
1658
+ "single_word": false,
1659
+ "special": true
1660
+ },
1661
+ "128207": {
1662
+ "content": "<|reserved_special_token_175|>",
1663
+ "lstrip": false,
1664
+ "normalized": false,
1665
+ "rstrip": false,
1666
+ "single_word": false,
1667
+ "special": true
1668
+ },
1669
+ "128208": {
1670
+ "content": "<|reserved_special_token_176|>",
1671
+ "lstrip": false,
1672
+ "normalized": false,
1673
+ "rstrip": false,
1674
+ "single_word": false,
1675
+ "special": true
1676
+ },
1677
+ "128209": {
1678
+ "content": "<|reserved_special_token_177|>",
1679
+ "lstrip": false,
1680
+ "normalized": false,
1681
+ "rstrip": false,
1682
+ "single_word": false,
1683
+ "special": true
1684
+ },
1685
+ "128210": {
1686
+ "content": "<|reserved_special_token_178|>",
1687
+ "lstrip": false,
1688
+ "normalized": false,
1689
+ "rstrip": false,
1690
+ "single_word": false,
1691
+ "special": true
1692
+ },
1693
+ "128211": {
1694
+ "content": "<|reserved_special_token_179|>",
1695
+ "lstrip": false,
1696
+ "normalized": false,
1697
+ "rstrip": false,
1698
+ "single_word": false,
1699
+ "special": true
1700
+ },
1701
+ "128212": {
1702
+ "content": "<|reserved_special_token_180|>",
1703
+ "lstrip": false,
1704
+ "normalized": false,
1705
+ "rstrip": false,
1706
+ "single_word": false,
1707
+ "special": true
1708
+ },
1709
+ "128213": {
1710
+ "content": "<|reserved_special_token_181|>",
1711
+ "lstrip": false,
1712
+ "normalized": false,
1713
+ "rstrip": false,
1714
+ "single_word": false,
1715
+ "special": true
1716
+ },
1717
+ "128214": {
1718
+ "content": "<|reserved_special_token_182|>",
1719
+ "lstrip": false,
1720
+ "normalized": false,
1721
+ "rstrip": false,
1722
+ "single_word": false,
1723
+ "special": true
1724
+ },
1725
+ "128215": {
1726
+ "content": "<|reserved_special_token_183|>",
1727
+ "lstrip": false,
1728
+ "normalized": false,
1729
+ "rstrip": false,
1730
+ "single_word": false,
1731
+ "special": true
1732
+ },
1733
+ "128216": {
1734
+ "content": "<|reserved_special_token_184|>",
1735
+ "lstrip": false,
1736
+ "normalized": false,
1737
+ "rstrip": false,
1738
+ "single_word": false,
1739
+ "special": true
1740
+ },
1741
+ "128217": {
1742
+ "content": "<|reserved_special_token_185|>",
1743
+ "lstrip": false,
1744
+ "normalized": false,
1745
+ "rstrip": false,
1746
+ "single_word": false,
1747
+ "special": true
1748
+ },
1749
+ "128218": {
1750
+ "content": "<|reserved_special_token_186|>",
1751
+ "lstrip": false,
1752
+ "normalized": false,
1753
+ "rstrip": false,
1754
+ "single_word": false,
1755
+ "special": true
1756
+ },
1757
+ "128219": {
1758
+ "content": "<|reserved_special_token_187|>",
1759
+ "lstrip": false,
1760
+ "normalized": false,
1761
+ "rstrip": false,
1762
+ "single_word": false,
1763
+ "special": true
1764
+ },
1765
+ "128220": {
1766
+ "content": "<|reserved_special_token_188|>",
1767
+ "lstrip": false,
1768
+ "normalized": false,
1769
+ "rstrip": false,
1770
+ "single_word": false,
1771
+ "special": true
1772
+ },
1773
+ "128221": {
1774
+ "content": "<|reserved_special_token_189|>",
1775
+ "lstrip": false,
1776
+ "normalized": false,
1777
+ "rstrip": false,
1778
+ "single_word": false,
1779
+ "special": true
1780
+ },
1781
+ "128222": {
1782
+ "content": "<|reserved_special_token_190|>",
1783
+ "lstrip": false,
1784
+ "normalized": false,
1785
+ "rstrip": false,
1786
+ "single_word": false,
1787
+ "special": true
1788
+ },
1789
+ "128223": {
1790
+ "content": "<|reserved_special_token_191|>",
1791
+ "lstrip": false,
1792
+ "normalized": false,
1793
+ "rstrip": false,
1794
+ "single_word": false,
1795
+ "special": true
1796
+ },
1797
+ "128224": {
1798
+ "content": "<|reserved_special_token_192|>",
1799
+ "lstrip": false,
1800
+ "normalized": false,
1801
+ "rstrip": false,
1802
+ "single_word": false,
1803
+ "special": true
1804
+ },
1805
+ "128225": {
1806
+ "content": "<|reserved_special_token_193|>",
1807
+ "lstrip": false,
1808
+ "normalized": false,
1809
+ "rstrip": false,
1810
+ "single_word": false,
1811
+ "special": true
1812
+ },
1813
+ "128226": {
1814
+ "content": "<|reserved_special_token_194|>",
1815
+ "lstrip": false,
1816
+ "normalized": false,
1817
+ "rstrip": false,
1818
+ "single_word": false,
1819
+ "special": true
1820
+ },
1821
+ "128227": {
1822
+ "content": "<|reserved_special_token_195|>",
1823
+ "lstrip": false,
1824
+ "normalized": false,
1825
+ "rstrip": false,
1826
+ "single_word": false,
1827
+ "special": true
1828
+ },
1829
+ "128228": {
1830
+ "content": "<|reserved_special_token_196|>",
1831
+ "lstrip": false,
1832
+ "normalized": false,
1833
+ "rstrip": false,
1834
+ "single_word": false,
1835
+ "special": true
1836
+ },
1837
+ "128229": {
1838
+ "content": "<|reserved_special_token_197|>",
1839
+ "lstrip": false,
1840
+ "normalized": false,
1841
+ "rstrip": false,
1842
+ "single_word": false,
1843
+ "special": true
1844
+ },
1845
+ "128230": {
1846
+ "content": "<|reserved_special_token_198|>",
1847
+ "lstrip": false,
1848
+ "normalized": false,
1849
+ "rstrip": false,
1850
+ "single_word": false,
1851
+ "special": true
1852
+ },
1853
+ "128231": {
1854
+ "content": "<|reserved_special_token_199|>",
1855
+ "lstrip": false,
1856
+ "normalized": false,
1857
+ "rstrip": false,
1858
+ "single_word": false,
1859
+ "special": true
1860
+ },
1861
+ "128232": {
1862
+ "content": "<|reserved_special_token_200|>",
1863
+ "lstrip": false,
1864
+ "normalized": false,
1865
+ "rstrip": false,
1866
+ "single_word": false,
1867
+ "special": true
1868
+ },
1869
+ "128233": {
1870
+ "content": "<|reserved_special_token_201|>",
1871
+ "lstrip": false,
1872
+ "normalized": false,
1873
+ "rstrip": false,
1874
+ "single_word": false,
1875
+ "special": true
1876
+ },
1877
+ "128234": {
1878
+ "content": "<|reserved_special_token_202|>",
1879
+ "lstrip": false,
1880
+ "normalized": false,
1881
+ "rstrip": false,
1882
+ "single_word": false,
1883
+ "special": true
1884
+ },
1885
+ "128235": {
1886
+ "content": "<|reserved_special_token_203|>",
1887
+ "lstrip": false,
1888
+ "normalized": false,
1889
+ "rstrip": false,
1890
+ "single_word": false,
1891
+ "special": true
1892
+ },
1893
+ "128236": {
1894
+ "content": "<|reserved_special_token_204|>",
1895
+ "lstrip": false,
1896
+ "normalized": false,
1897
+ "rstrip": false,
1898
+ "single_word": false,
1899
+ "special": true
1900
+ },
1901
+ "128237": {
1902
+ "content": "<|reserved_special_token_205|>",
1903
+ "lstrip": false,
1904
+ "normalized": false,
1905
+ "rstrip": false,
1906
+ "single_word": false,
1907
+ "special": true
1908
+ },
1909
+ "128238": {
1910
+ "content": "<|reserved_special_token_206|>",
1911
+ "lstrip": false,
1912
+ "normalized": false,
1913
+ "rstrip": false,
1914
+ "single_word": false,
1915
+ "special": true
1916
+ },
1917
+ "128239": {
1918
+ "content": "<|reserved_special_token_207|>",
1919
+ "lstrip": false,
1920
+ "normalized": false,
1921
+ "rstrip": false,
1922
+ "single_word": false,
1923
+ "special": true
1924
+ },
1925
+ "128240": {
1926
+ "content": "<|reserved_special_token_208|>",
1927
+ "lstrip": false,
1928
+ "normalized": false,
1929
+ "rstrip": false,
1930
+ "single_word": false,
1931
+ "special": true
1932
+ },
1933
+ "128241": {
1934
+ "content": "<|reserved_special_token_209|>",
1935
+ "lstrip": false,
1936
+ "normalized": false,
1937
+ "rstrip": false,
1938
+ "single_word": false,
1939
+ "special": true
1940
+ },
1941
+ "128242": {
1942
+ "content": "<|reserved_special_token_210|>",
1943
+ "lstrip": false,
1944
+ "normalized": false,
1945
+ "rstrip": false,
1946
+ "single_word": false,
1947
+ "special": true
1948
+ },
1949
+ "128243": {
1950
+ "content": "<|reserved_special_token_211|>",
1951
+ "lstrip": false,
1952
+ "normalized": false,
1953
+ "rstrip": false,
1954
+ "single_word": false,
1955
+ "special": true
1956
+ },
1957
+ "128244": {
1958
+ "content": "<|reserved_special_token_212|>",
1959
+ "lstrip": false,
1960
+ "normalized": false,
1961
+ "rstrip": false,
1962
+ "single_word": false,
1963
+ "special": true
1964
+ },
1965
+ "128245": {
1966
+ "content": "<|reserved_special_token_213|>",
1967
+ "lstrip": false,
1968
+ "normalized": false,
1969
+ "rstrip": false,
1970
+ "single_word": false,
1971
+ "special": true
1972
+ },
1973
+ "128246": {
1974
+ "content": "<|reserved_special_token_214|>",
1975
+ "lstrip": false,
1976
+ "normalized": false,
1977
+ "rstrip": false,
1978
+ "single_word": false,
1979
+ "special": true
1980
+ },
1981
+ "128247": {
1982
+ "content": "<|reserved_special_token_215|>",
1983
+ "lstrip": false,
1984
+ "normalized": false,
1985
+ "rstrip": false,
1986
+ "single_word": false,
1987
+ "special": true
1988
+ },
1989
+ "128248": {
1990
+ "content": "<|reserved_special_token_216|>",
1991
+ "lstrip": false,
1992
+ "normalized": false,
1993
+ "rstrip": false,
1994
+ "single_word": false,
1995
+ "special": true
1996
+ },
1997
+ "128249": {
1998
+ "content": "<|reserved_special_token_217|>",
1999
+ "lstrip": false,
2000
+ "normalized": false,
2001
+ "rstrip": false,
2002
+ "single_word": false,
2003
+ "special": true
2004
+ },
2005
+ "128250": {
2006
+ "content": "<|reserved_special_token_218|>",
2007
+ "lstrip": false,
2008
+ "normalized": false,
2009
+ "rstrip": false,
2010
+ "single_word": false,
2011
+ "special": true
2012
+ },
2013
+ "128251": {
2014
+ "content": "<|reserved_special_token_219|>",
2015
+ "lstrip": false,
2016
+ "normalized": false,
2017
+ "rstrip": false,
2018
+ "single_word": false,
2019
+ "special": true
2020
+ },
2021
+ "128252": {
2022
+ "content": "<|reserved_special_token_220|>",
2023
+ "lstrip": false,
2024
+ "normalized": false,
2025
+ "rstrip": false,
2026
+ "single_word": false,
2027
+ "special": true
2028
+ },
2029
+ "128253": {
2030
+ "content": "<|reserved_special_token_221|>",
2031
+ "lstrip": false,
2032
+ "normalized": false,
2033
+ "rstrip": false,
2034
+ "single_word": false,
2035
+ "special": true
2036
+ },
2037
+ "128254": {
2038
+ "content": "<|reserved_special_token_222|>",
2039
+ "lstrip": false,
2040
+ "normalized": false,
2041
+ "rstrip": false,
2042
+ "single_word": false,
2043
+ "special": true
2044
+ },
2045
+ "128255": {
2046
+ "content": "<|reserved_special_token_223|>",
2047
+ "lstrip": false,
2048
+ "normalized": false,
2049
+ "rstrip": false,
2050
+ "single_word": false,
2051
+ "special": true
2052
+ }
2053
+ },
2054
+ "bos_token": "<|begin_of_text|>",
2055
+ "clean_up_tokenization_spaces": true,
2056
+ "eos_token": "<|im_end|>",
2057
+ "errors": "replace",
2058
+ "extra_special_tokens": {},
2059
+ "model_input_names": [
2060
+ "input_ids",
2061
+ "attention_mask"
2062
+ ],
2063
+ "model_max_length": 131072,
2064
+ "pad_token": "<|end_of_text|>",
2065
+ "padding_side": "left",
2066
+ "split_special_tokens": false,
2067
+ "tokenizer_class": "TFMTokenizer",
2068
+ "unk_token": null
2069
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff