Hugging Face's logo

CollectionStudio
/
umt5-base

PyTorch
Model card Files
xet
 Community
CollectionStudio ArthurZ HF Staff commited on
Commit
39d83d1
·
0 Parent(s):

Duplicate from google/umt5-base

Browse files

Co-authored-by: Arthur Zucker <ArthurZ@users.noreply.huggingface.co>

Files changed (9) hide show
  1. .gitattributes +36 -0
  2. README.md +130 -0
  3. config.json +32 -0
  4. generation_config.json +7 -0
  5. pytorch_model.bin +3 -0
  6. special_tokens_map.json +308 -0
  7. spiece.model +3 -0
  8. tokenizer.json +3 -0
  9. tokenizer_config.json +314 -0
.gitattributes ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - multilingual
4
+ - af
5
+ - am
6
+ - ar
7
+ - az
8
+ - be
9
+ - bg
10
+ - bn
11
+ - ca
12
+ - ceb
13
+ - co
14
+ - cs
15
+ - cy
16
+ - da
17
+ - de
18
+ - el
19
+ - en
20
+ - eo
21
+ - es
22
+ - et
23
+ - eu
24
+ - fa
25
+ - fi
26
+ - fil
27
+ - fr
28
+ - fy
29
+ - ga
30
+ - gd
31
+ - gl
32
+ - gu
33
+ - ha
34
+ - haw
35
+ - hi
36
+ - hmn
37
+ - ht
38
+ - hu
39
+ - hy
40
+ - ig
41
+ - is
42
+ - it
43
+ - iw
44
+ - ja
45
+ - jv
46
+ - ka
47
+ - kk
48
+ - km
49
+ - kn
50
+ - ko
51
+ - ku
52
+ - ky
53
+ - la
54
+ - lb
55
+ - lo
56
+ - lt
57
+ - lv
58
+ - mg
59
+ - mi
60
+ - mk
61
+ - ml
62
+ - mn
63
+ - mr
64
+ - ms
65
+ - mt
66
+ - my
67
+ - ne
68
+ - nl
69
+ - no
70
+ - ny
71
+ - pa
72
+ - pl
73
+ - ps
74
+ - pt
75
+ - ro
76
+ - ru
77
+ - sd
78
+ - si
79
+ - sk
80
+ - sl
81
+ - sm
82
+ - sn
83
+ - so
84
+ - sq
85
+ - sr
86
+ - st
87
+ - su
88
+ - sv
89
+ - sw
90
+ - ta
91
+ - te
92
+ - tg
93
+ - th
94
+ - tr
95
+ - uk
96
+ - und
97
+ - ur
98
+ - uz
99
+ - vi
100
+ - xh
101
+ - yi
102
+ - yo
103
+ - zh
104
+ - zu
105
+ datasets:
106
+ - mc4
107
+
108
+ license: apache-2.0
109
+ ---
110
+
111
+ [Google's UMT5](https://github.com/google-research/multilingual-t5)
112
+
113
+ UMT5 is pretrained on the an updated version of [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) corpus, covering 107 languages:
114
+
115
+ Afrikaans, Albanian, Amharic, Arabic, Armenian, Azerbaijani, Basque, Belarusian, Bengali, Bulgarian, Burmese, Catalan, Cebuano, Chichewa, Chinese, Corsican, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Igbo, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish, Kyrgyz, Lao, Latin, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan, Scottish Gaelic, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sotho, Spanish, Sundanese, Swahili, Swedish, Tajik, Tamil, Telugu, Thai, Turkish, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, West Frisian, Xhosa, Yiddish, Yoruba, Zulu.
116
+
117
+ **Note**: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.
118
+
119
+ Pretraining Dataset: [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual)
120
+
121
+ Other Community Checkpoints: [here](https://huggingface.co/models?search=umt5)
122
+
123
+ Paper: [UniMax, Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining](https://openreview.net/forum?id=kXwdL1cWOAi)
124
+
125
+ Authors: *by Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, Noah Constant*
126
+
127
+
128
+ ## Abstract
129
+
130
+ *Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.*
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "ArthurZ/umt5-base",
3
+ "architectures": [
4
+ "UMT5ForConditionalGeneration"
5
+ ],
6
+ "d_ff": 2048,
7
+ "d_kv": 64,
8
+ "d_model": 768,
9
+ "decoder_start_token_id": 0,
10
+ "dense_act_fn": "gelu_new",
11
+ "dropout_rate": 0.1,
12
+ "eos_token_id": 1,
13
+ "feed_forward_proj": "gated-gelu",
14
+ "initializer_factor": 1.0,
15
+ "is_encoder_decoder": true,
16
+ "is_gated_act": true,
17
+ "layer_norm_epsilon": 1e-06,
18
+ "num_decoder_layers": 12,
19
+ "num_heads": 12,
20
+ "num_layers": 12,
21
+ "output_past": true,
22
+ "pad_token_id": 0,
23
+ "relative_attention_max_distance": 128,
24
+ "relative_attention_num_buckets": 32,
25
+ "scalable_attention": true,
26
+ "tie_word_embeddings": false,
27
+ "tokenizer_class": "T5Tokenizer",
28
+ "torch_dtype": "float32",
29
+ "transformers_version": "4.31.0.dev0",
30
+ "use_cache": true,
31
+ "vocab_size": 256384
32
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "decoder_start_token_id": 0,
4
+ "eos_token_id": 1,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "4.31.0.dev0"
7
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c48f7036095f1a02ee817d1b131de443bfe126b45ff76133ef5ae685bac9f061
3
+ size 2368275277
special_tokens_map.json ADDED
@@ -0,0 +1,308 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<extra_id_299>",
4
+ "<extra_id_298>",
5
+ "<extra_id_297>",
6
+ "<extra_id_296>",
7
+ "<extra_id_295>",
8
+ "<extra_id_294>",
9
+ "<extra_id_293>",
10
+ "<extra_id_292>",
11
+ "<extra_id_291>",
12
+ "<extra_id_290>",
13
+ "<extra_id_289>",
14
+ "<extra_id_288>",
15
+ "<extra_id_287>",
16
+ "<extra_id_286>",
17
+ "<extra_id_285>",
18
+ "<extra_id_284>",
19
+ "<extra_id_283>",
20
+ "<extra_id_282>",
21
+ "<extra_id_281>",
22
+ "<extra_id_280>",
23
+ "<extra_id_279>",
24
+ "<extra_id_278>",
25
+ "<extra_id_277>",
26
+ "<extra_id_276>",
27
+ "<extra_id_275>",
28
+ "<extra_id_274>",
29
+ "<extra_id_273>",
30
+ "<extra_id_272>",
31
+ "<extra_id_271>",
32
+ "<extra_id_270>",
33
+ "<extra_id_269>",
34
+ "<extra_id_268>",
35
+ "<extra_id_267>",
36
+ "<extra_id_266>",
37
+ "<extra_id_265>",
38
+ "<extra_id_264>",
39
+ "<extra_id_263>",
40
+ "<extra_id_262>",
41
+ "<extra_id_261>",
42
+ "<extra_id_260>",
43
+ "<extra_id_259>",
44
+ "<extra_id_258>",
45
+ "<extra_id_257>",
46
+ "<extra_id_256>",
47
+ "<extra_id_255>",
48
+ "<extra_id_254>",
49
+ "<extra_id_253>",
50
+ "<extra_id_252>",
51
+ "<extra_id_251>",
52
+ "<extra_id_250>",
53
+ "<extra_id_249>",
54
+ "<extra_id_248>",
55
+ "<extra_id_247>",
56
+ "<extra_id_246>",
57
+ "<extra_id_245>",
58
+ "<extra_id_244>",
59
+ "<extra_id_243>",
60
+ "<extra_id_242>",
61
+ "<extra_id_241>",
62
+ "<extra_id_240>",
63
+ "<extra_id_239>",
64
+ "<extra_id_238>",
65
+ "<extra_id_237>",
66
+ "<extra_id_236>",
67
+ "<extra_id_235>",
68
+ "<extra_id_234>",
69
+ "<extra_id_233>",
70
+ "<extra_id_232>",
71
+ "<extra_id_231>",
72
+ "<extra_id_230>",
73
+ "<extra_id_229>",
74
+ "<extra_id_228>",
75
+ "<extra_id_227>",
76
+ "<extra_id_226>",
77
+ "<extra_id_225>",
78
+ "<extra_id_224>",
79
+ "<extra_id_223>",
80
+ "<extra_id_222>",
81
+ "<extra_id_221>",
82
+ "<extra_id_220>",
83
+ "<extra_id_219>",
84
+ "<extra_id_218>",
85
+ "<extra_id_217>",
86
+ "<extra_id_216>",
87
+ "<extra_id_215>",
88
+ "<extra_id_214>",
89
+ "<extra_id_213>",
90
+ "<extra_id_212>",
91
+ "<extra_id_211>",
92
+ "<extra_id_210>",
93
+ "<extra_id_209>",
94
+ "<extra_id_208>",
95
+ "<extra_id_207>",
96
+ "<extra_id_206>",
97
+ "<extra_id_205>",
98
+ "<extra_id_204>",
99
+ "<extra_id_203>",
100
+ "<extra_id_202>",
101
+ "<extra_id_201>",
102
+ "<extra_id_200>",
103
+ "<extra_id_199>",
104
+ "<extra_id_198>",
105
+ "<extra_id_197>",
106
+ "<extra_id_196>",
107
+ "<extra_id_195>",
108
+ "<extra_id_194>",
109
+ "<extra_id_193>",
110
+ "<extra_id_192>",
111
+ "<extra_id_191>",
112
+ "<extra_id_190>",
113
+ "<extra_id_189>",
114
+ "<extra_id_188>",
115
+ "<extra_id_187>",
116
+ "<extra_id_186>",
117
+ "<extra_id_185>",
118
+ "<extra_id_184>",
119
+ "<extra_id_183>",
120
+ "<extra_id_182>",
121
+ "<extra_id_181>",
122
+ "<extra_id_180>",
123
+ "<extra_id_179>",
124
+ "<extra_id_178>",
125
+ "<extra_id_177>",
126
+ "<extra_id_176>",
127
+ "<extra_id_175>",
128
+ "<extra_id_174>",
129
+ "<extra_id_173>",
130
+ "<extra_id_172>",
131
+ "<extra_id_171>",
132
+ "<extra_id_170>",
133
+ "<extra_id_169>",
134
+ "<extra_id_168>",
135
+ "<extra_id_167>",
136
+ "<extra_id_166>",
137
+ "<extra_id_165>",
138
+ "<extra_id_164>",
139
+ "<extra_id_163>",
140
+ "<extra_id_162>",
141
+ "<extra_id_161>",
142
+ "<extra_id_160>",
143
+ "<extra_id_159>",
144
+ "<extra_id_158>",
145
+ "<extra_id_157>",
146
+ "<extra_id_156>",
147
+ "<extra_id_155>",
148
+ "<extra_id_154>",
149
+ "<extra_id_153>",
150
+ "<extra_id_152>",
151
+ "<extra_id_151>",
152
+ "<extra_id_150>",
153
+ "<extra_id_149>",
154
+ "<extra_id_148>",
155
+ "<extra_id_147>",
156
+ "<extra_id_146>",
157
+ "<extra_id_145>",
158
+ "<extra_id_144>",
159
+ "<extra_id_143>",
160
+ "<extra_id_142>",
161
+ "<extra_id_141>",
162
+ "<extra_id_140>",
163
+ "<extra_id_139>",
164
+ "<extra_id_138>",
165
+ "<extra_id_137>",
166
+ "<extra_id_136>",
167
+ "<extra_id_135>",
168
+ "<extra_id_134>",
169
+ "<extra_id_133>",
170
+ "<extra_id_132>",
171
+ "<extra_id_131>",
172
+ "<extra_id_130>",
173
+ "<extra_id_129>",
174
+ "<extra_id_128>",
175
+ "<extra_id_127>",
176
+ "<extra_id_126>",
177
+ "<extra_id_125>",
178
+ "<extra_id_124>",
179
+ "<extra_id_123>",
180
+ "<extra_id_122>",
181
+ "<extra_id_121>",
182
+ "<extra_id_120>",
183
+ "<extra_id_119>",
184
+ "<extra_id_118>",
185
+ "<extra_id_117>",
186
+ "<extra_id_116>",
187
+ "<extra_id_115>",
188
+ "<extra_id_114>",
189
+ "<extra_id_113>",
190
+ "<extra_id_112>",
191
+ "<extra_id_111>",
192
+ "<extra_id_110>",
193
+ "<extra_id_109>",
194
+ "<extra_id_108>",
195
+ "<extra_id_107>",
196
+ "<extra_id_106>",
197
+ "<extra_id_105>",
198
+ "<extra_id_104>",
199
+ "<extra_id_103>",
200
+ "<extra_id_102>",
201
+ "<extra_id_101>",
202
+ "<extra_id_100>",
203
+ "<extra_id_99>",
204
+ "<extra_id_98>",
205
+ "<extra_id_97>",
206
+ "<extra_id_96>",
207
+ "<extra_id_95>",
208
+ "<extra_id_94>",
209
+ "<extra_id_93>",
210
+ "<extra_id_92>",
211
+ "<extra_id_91>",
212
+ "<extra_id_90>",
213
+ "<extra_id_89>",
214
+ "<extra_id_88>",
215
+ "<extra_id_87>",
216
+ "<extra_id_86>",
217
+ "<extra_id_85>",
218
+ "<extra_id_84>",
219
+ "<extra_id_83>",
220
+ "<extra_id_82>",
221
+ "<extra_id_81>",
222
+ "<extra_id_80>",
223
+ "<extra_id_79>",
224
+ "<extra_id_78>",
225
+ "<extra_id_77>",
226
+ "<extra_id_76>",
227
+ "<extra_id_75>",
228
+ "<extra_id_74>",
229
+ "<extra_id_73>",
230
+ "<extra_id_72>",
231
+ "<extra_id_71>",
232
+ "<extra_id_70>",
233
+ "<extra_id_69>",
234
+ "<extra_id_68>",
235
+ "<extra_id_67>",
236
+ "<extra_id_66>",
237
+ "<extra_id_65>",
238
+ "<extra_id_64>",
239
+ "<extra_id_63>",
240
+ "<extra_id_62>",
241
+ "<extra_id_61>",
242
+ "<extra_id_60>",
243
+ "<extra_id_59>",
244
+ "<extra_id_58>",
245
+ "<extra_id_57>",
246
+ "<extra_id_56>",
247
+ "<extra_id_55>",
248
+ "<extra_id_54>",
249
+ "<extra_id_53>",
250
+ "<extra_id_52>",
251
+ "<extra_id_51>",
252
+ "<extra_id_50>",
253
+ "<extra_id_49>",
254
+ "<extra_id_48>",
255
+ "<extra_id_47>",
256
+ "<extra_id_46>",
257
+ "<extra_id_45>",
258
+ "<extra_id_44>",
259
+ "<extra_id_43>",
260
+ "<extra_id_42>",
261
+ "<extra_id_41>",
262
+ "<extra_id_40>",
263
+ "<extra_id_39>",
264
+ "<extra_id_38>",
265
+ "<extra_id_37>",
266
+ "<extra_id_36>",
267
+ "<extra_id_35>",
268
+ "<extra_id_34>",
269
+ "<extra_id_33>",
270
+ "<extra_id_32>",
271
+ "<extra_id_31>",
272
+ "<extra_id_30>",
273
+ "<extra_id_29>",
274
+ "<extra_id_28>",
275
+ "<extra_id_27>",
276
+ "<extra_id_26>",
277
+ "<extra_id_25>",
278
+ "<extra_id_24>",
279
+ "<extra_id_23>",
280
+ "<extra_id_22>",
281
+ "<extra_id_21>",
282
+ "<extra_id_20>",
283
+ "<extra_id_19>",
284
+ "<extra_id_18>",
285
+ "<extra_id_17>",
286
+ "<extra_id_16>",
287
+ "<extra_id_15>",
288
+ "<extra_id_14>",
289
+ "<extra_id_13>",
290
+ "<extra_id_12>",
291
+ "<extra_id_11>",
292
+ "<extra_id_10>",
293
+ "<extra_id_9>",
294
+ "<extra_id_8>",
295
+ "<extra_id_7>",
296
+ "<extra_id_6>",
297
+ "<extra_id_5>",
298
+ "<extra_id_4>",
299
+ "<extra_id_3>",
300
+ "<extra_id_2>",
301
+ "<extra_id_1>",
302
+ "<extra_id_0>"
303
+ ],
304
+ "bos_token": "<s>",
305
+ "eos_token": "</s>",
306
+ "pad_token": "<pad>",
307
+ "unk_token": "<unk>"
308
+ }
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e3909a67b780650b35cf529ac782ad2b6b26e6d1f849d3fbb6a872905f452458
3
+ size 4548313
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af904105ce1071b1202bba0059a841f4a7b85b48b6ec179c4948e3483476e0dd
3
+ size 16853013
tokenizer_config.json ADDED
@@ -0,0 +1,314 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<extra_id_0>",
4
+ "<extra_id_1>",
5
+ "<extra_id_2>",
6
+ "<extra_id_3>",
7
+ "<extra_id_4>",
8
+ "<extra_id_5>",
9
+ "<extra_id_6>",
10
+ "<extra_id_7>",
11
+ "<extra_id_8>",
12
+ "<extra_id_9>",
13
+ "<extra_id_10>",
14
+ "<extra_id_11>",
15
+ "<extra_id_12>",
16
+ "<extra_id_13>",
17
+ "<extra_id_14>",
18
+ "<extra_id_15>",
19
+ "<extra_id_16>",
20
+ "<extra_id_17>",
21
+ "<extra_id_18>",
22
+ "<extra_id_19>",
23
+ "<extra_id_20>",
24
+ "<extra_id_21>",
25
+ "<extra_id_22>",
26
+ "<extra_id_23>",
27
+ "<extra_id_24>",
28
+ "<extra_id_25>",
29
+ "<extra_id_26>",
30
+ "<extra_id_27>",
31
+ "<extra_id_28>",
32
+ "<extra_id_29>",
33
+ "<extra_id_30>",
34
+ "<extra_id_31>",
35
+ "<extra_id_32>",
36
+ "<extra_id_33>",
37
+ "<extra_id_34>",
38
+ "<extra_id_35>",
39
+ "<extra_id_36>",
40
+ "<extra_id_37>",
41
+ "<extra_id_38>",
42
+ "<extra_id_39>",
43
+ "<extra_id_40>",
44
+ "<extra_id_41>",
45
+ "<extra_id_42>",
46
+ "<extra_id_43>",
47
+ "<extra_id_44>",
48
+ "<extra_id_45>",
49
+ "<extra_id_46>",
50
+ "<extra_id_47>",
51
+ "<extra_id_48>",
52
+ "<extra_id_49>",
53
+ "<extra_id_50>",
54
+ "<extra_id_51>",
55
+ "<extra_id_52>",
56
+ "<extra_id_53>",
57
+ "<extra_id_54>",
58
+ "<extra_id_55>",
59
+ "<extra_id_56>",
60
+ "<extra_id_57>",
61
+ "<extra_id_58>",
62
+ "<extra_id_59>",
63
+ "<extra_id_60>",
64
+ "<extra_id_61>",
65
+ "<extra_id_62>",
66
+ "<extra_id_63>",
67
+ "<extra_id_64>",
68
+ "<extra_id_65>",
69
+ "<extra_id_66>",
70
+ "<extra_id_67>",
71
+ "<extra_id_68>",
72
+ "<extra_id_69>",
73
+ "<extra_id_70>",
74
+ "<extra_id_71>",
75
+ "<extra_id_72>",
76
+ "<extra_id_73>",
77
+ "<extra_id_74>",
78
+ "<extra_id_75>",
79
+ "<extra_id_76>",
80
+ "<extra_id_77>",
81
+ "<extra_id_78>",
82
+ "<extra_id_79>",
83
+ "<extra_id_80>",
84
+ "<extra_id_81>",
85
+ "<extra_id_82>",
86
+ "<extra_id_83>",
87
+ "<extra_id_84>",
88
+ "<extra_id_85>",
89
+ "<extra_id_86>",
90
+ "<extra_id_87>",
91
+ "<extra_id_88>",
92
+ "<extra_id_89>",
93
+ "<extra_id_90>",
94
+ "<extra_id_91>",
95
+ "<extra_id_92>",
96
+ "<extra_id_93>",
97
+ "<extra_id_94>",
98
+ "<extra_id_95>",
99
+ "<extra_id_96>",
100
+ "<extra_id_97>",
101
+ "<extra_id_98>",
102
+ "<extra_id_99>",
103
+ "<extra_id_100>",
104
+ "<extra_id_101>",
105
+ "<extra_id_102>",
106
+ "<extra_id_103>",
107
+ "<extra_id_104>",
108
+ "<extra_id_105>",
109
+ "<extra_id_106>",
110
+ "<extra_id_107>",
111
+ "<extra_id_108>",
112
+ "<extra_id_109>",
113
+ "<extra_id_110>",
114
+ "<extra_id_111>",
115
+ "<extra_id_112>",
116
+ "<extra_id_113>",
117
+ "<extra_id_114>",
118
+ "<extra_id_115>",
119
+ "<extra_id_116>",
120
+ "<extra_id_117>",
121
+ "<extra_id_118>",
122
+ "<extra_id_119>",
123
+ "<extra_id_120>",
124
+ "<extra_id_121>",
125
+ "<extra_id_122>",
126
+ "<extra_id_123>",
127
+ "<extra_id_124>",
128
+ "<extra_id_125>",
129
+ "<extra_id_126>",
130
+ "<extra_id_127>",
131
+ "<extra_id_128>",
132
+ "<extra_id_129>",
133
+ "<extra_id_130>",
134
+ "<extra_id_131>",
135
+ "<extra_id_132>",
136
+ "<extra_id_133>",
137
+ "<extra_id_134>",
138
+ "<extra_id_135>",
139
+ "<extra_id_136>",
140
+ "<extra_id_137>",
141
+ "<extra_id_138>",
142
+ "<extra_id_139>",
143
+ "<extra_id_140>",
144
+ "<extra_id_141>",
145
+ "<extra_id_142>",
146
+ "<extra_id_143>",
147
+ "<extra_id_144>",
148
+ "<extra_id_145>",
149
+ "<extra_id_146>",
150
+ "<extra_id_147>",
151
+ "<extra_id_148>",
152
+ "<extra_id_149>",
153
+ "<extra_id_150>",
154
+ "<extra_id_151>",
155
+ "<extra_id_152>",
156
+ "<extra_id_153>",
157
+ "<extra_id_154>",
158
+ "<extra_id_155>",
159
+ "<extra_id_156>",
160
+ "<extra_id_157>",
161
+ "<extra_id_158>",
162
+ "<extra_id_159>",
163
+ "<extra_id_160>",
164
+ "<extra_id_161>",
165
+ "<extra_id_162>",
166
+ "<extra_id_163>",
167
+ "<extra_id_164>",
168
+ "<extra_id_165>",
169
+ "<extra_id_166>",
170
+ "<extra_id_167>",
171
+ "<extra_id_168>",
172
+ "<extra_id_169>",
173
+ "<extra_id_170>",
174
+ "<extra_id_171>",
175
+ "<extra_id_172>",
176
+ "<extra_id_173>",
177
+ "<extra_id_174>",
178
+ "<extra_id_175>",
179
+ "<extra_id_176>",
180
+ "<extra_id_177>",
181
+ "<extra_id_178>",
182
+ "<extra_id_179>",
183
+ "<extra_id_180>",
184
+ "<extra_id_181>",
185
+ "<extra_id_182>",
186
+ "<extra_id_183>",
187
+ "<extra_id_184>",
188
+ "<extra_id_185>",
189
+ "<extra_id_186>",
190
+ "<extra_id_187>",
191
+ "<extra_id_188>",
192
+ "<extra_id_189>",
193
+ "<extra_id_190>",
194
+ "<extra_id_191>",
195
+ "<extra_id_192>",
196
+ "<extra_id_193>",
197
+ "<extra_id_194>",
198
+ "<extra_id_195>",
199
+ "<extra_id_196>",
200
+ "<extra_id_197>",
201
+ "<extra_id_198>",
202
+ "<extra_id_199>",
203
+ "<extra_id_200>",
204
+ "<extra_id_201>",
205
+ "<extra_id_202>",
206
+ "<extra_id_203>",
207
+ "<extra_id_204>",
208
+ "<extra_id_205>",
209
+ "<extra_id_206>",
210
+ "<extra_id_207>",
211
+ "<extra_id_208>",
212
+ "<extra_id_209>",
213
+ "<extra_id_210>",
214
+ "<extra_id_211>",
215
+ "<extra_id_212>",
216
+ "<extra_id_213>",
217
+ "<extra_id_214>",
218
+ "<extra_id_215>",
219
+ "<extra_id_216>",
220
+ "<extra_id_217>",
221
+ "<extra_id_218>",
222
+ "<extra_id_219>",
223
+ "<extra_id_220>",
224
+ "<extra_id_221>",
225
+ "<extra_id_222>",
226
+ "<extra_id_223>",
227
+ "<extra_id_224>",
228
+ "<extra_id_225>",
229
+ "<extra_id_226>",
230
+ "<extra_id_227>",
231
+ "<extra_id_228>",
232
+ "<extra_id_229>",
233
+ "<extra_id_230>",
234
+ "<extra_id_231>",
235
+ "<extra_id_232>",
236
+ "<extra_id_233>",
237
+ "<extra_id_234>",
238
+ "<extra_id_235>",
239
+ "<extra_id_236>",
240
+ "<extra_id_237>",
241
+ "<extra_id_238>",
242
+ "<extra_id_239>",
243
+ "<extra_id_240>",
244
+ "<extra_id_241>",
245
+ "<extra_id_242>",
246
+ "<extra_id_243>",
247
+ "<extra_id_244>",
248
+ "<extra_id_245>",
249
+ "<extra_id_246>",
250
+ "<extra_id_247>",
251
+ "<extra_id_248>",
252
+ "<extra_id_249>",
253
+ "<extra_id_250>",
254
+ "<extra_id_251>",
255
+ "<extra_id_252>",
256
+ "<extra_id_253>",
257
+ "<extra_id_254>",
258
+ "<extra_id_255>",
259
+ "<extra_id_256>",
260
+ "<extra_id_257>",
261
+ "<extra_id_258>",
262
+ "<extra_id_259>",
263
+ "<extra_id_260>",
264
+ "<extra_id_261>",
265
+ "<extra_id_262>",
266
+ "<extra_id_263>",
267
+ "<extra_id_264>",
268
+ "<extra_id_265>",
269
+ "<extra_id_266>",
270
+ "<extra_id_267>",
271
+ "<extra_id_268>",
272
+ "<extra_id_269>",
273
+ "<extra_id_270>",
274
+ "<extra_id_271>",
275
+ "<extra_id_272>",
276
+ "<extra_id_273>",
277
+ "<extra_id_274>",
278
+ "<extra_id_275>",
279
+ "<extra_id_276>",
280
+ "<extra_id_277>",
281
+ "<extra_id_278>",
282
+ "<extra_id_279>",
283
+ "<extra_id_280>",
284
+ "<extra_id_281>",
285
+ "<extra_id_282>",
286
+ "<extra_id_283>",
287
+ "<extra_id_284>",
288
+ "<extra_id_285>",
289
+ "<extra_id_286>",
290
+ "<extra_id_287>",
291
+ "<extra_id_288>",
292
+ "<extra_id_289>",
293
+ "<extra_id_290>",
294
+ "<extra_id_291>",
295
+ "<extra_id_292>",
296
+ "<extra_id_293>",
297
+ "<extra_id_294>",
298
+ "<extra_id_295>",
299
+ "<extra_id_296>",
300
+ "<extra_id_297>",
301
+ "<extra_id_298>",
302
+ "<extra_id_299>"
303
+ ],
304
+ "bos_token": "<s>",
305
+ "clean_up_tokenization_spaces": true,
306
+ "eos_token": "</s>",
307
+ "extra_ids": 300,
308
+ "model_max_length": 1000000000000000019884624838656,
309
+ "pad_token": "<pad>",
310
+ "sp_model_kwargs": {},
311
+ "spaces_between_special_tokens": false,
312
+ "tokenizer_class": "T5Tokenizer",
313
+ "unk_token": "<unk>"
314
+ }