Hugging Face's logo

42dot
/
42dot_LLM-SFT-1.3B

Text Generation
Transformers
PyTorch
Safetensors
English
Korean
llama
causal-lm
42dot_llm
text-generation-inference
Model card Files
xet
 Community
3

Adding `safetensors` variant of this model

#1
by ykhwang - opened
base: refs/heads/main
←
from: refs/pr/1
Discussion Files changed
+16
-502
Files changed (9) hide show
  1. README.md +13 -13
  2. added_tokens.json +1 -45
  3. asset/42dot-SFT-vs.png +0 -0
  4. asset/42dot_LLM_vs_score.png +0 -0
  5. asset/42dot_llm_ko_score_white_background.png +0 -0
  6. asset/Ko-Score.png +0 -0
  7. special_tokens_map.json +0 -46
  8. tokenizer.json +0 -396
  9. tokenizer_config.json +2 -2
README.md CHANGED
@@ -10,15 +10,15 @@ tags:
10
  - 42dot_llm
11
  license: cc-by-nc-4.0
12
  ---
13
- # 42dot_LLM-SFT-1.3B
14
 
15
- **42dot LLM-SFT** is a large language model (LLM) developed by [**42dot**](https://42dot.ai/) which is trained to follow natural language instructions.
16
- 42dot LLM-SFT is a part of **42dot LLM**, and derived from **42dot LLM-PLM** by supervised fine-tuning (SFT). This repository contains a 1.3B-parameter version.
17
 
18
  ## Model Description
19
 
20
  ### Hyperparameters
21
- As same as 42dot LLM-PLM, the model is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
22
 
23
  | Params | Layers | Attention heads | Hidden size | FFN size | Max. length\* |
24
  | -- | -- | -- | -- | -- | -- |
@@ -30,45 +30,45 @@ As same as 42dot LLM-PLM, the model is built upon a Transformer decoder architec
30
  Fine-tuning took about 112 GPU hours (in NVIDIA A100). For the training dataset, we manually constructed (question or insturuction) and response pairs, which can either be single- or multi-turn.
31
 
32
  ### Evaluation
33
- Inspired by recent attempts like [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/#how-good-is-vicuna), we evaluate 42dot LLM-SFT with other proprietary/open-sourced chatbots using GPT-4 for assessing various aspects of responses. The evaluation dataset consists of 121 prompts over 10 categories. The sample of the evaluation dataset and prompt template can be downloaded from our [GitHub repo](https://github.com/42dot/42dot_LLM).
34
 
35
  - Baselines:
36
  - [ChatGPT](https://chat.openai.com/) using GPT-3.5-turbo and GPT-4
37
  - [Bard](https://bard.google.com/)
38
  - [KORani-v2-13B](https://huggingface.co/KRAFTON/KORani-v1-13B)
39
 
40
- | Model | GPT-3.5 | GPT-4 | Bard | KORani | 42dot LLM-SFT |
41
  | :-- |:-------:|:--------:|:--------:|:------:|:---------:|
42
  | Params | Unknown | Unknown | Unknown | 13B | 1.3B |
43
 
44
  <figure align="center">
45
- <img src="https://huggingface.co/42dot/42dot_LLM-SFT-1.3B/resolve/main/asset/42dot_llm_ko_score_white_background.png"/>
46
  <figcaption><b>Response quality evaluation result</b></figcaption>
47
  </figure>
48
 
49
  <figure align="center">
50
- <img src="https://huggingface.co/42dot/42dot_LLM-SFT-1.3B/resolve/main/asset/42dot_LLM_vs_score.png"/>
51
- <figcaption><b>Comparison between proprietary chatbots and 42dot LLM-SFT</b></figcaption>
52
  </figure>
53
 
54
 
55
  ## Limitations and Ethical Considerations
56
- 42dot LLM-SFT shares a number of well-known limitations of other LLMs. For example, it may generate false and misinformative content since 42dot LLM-SFT is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot LLM-SFT may generate toxic, harmful, and biased content due to the use of web-available training data in the pre-training phase. We strongly suggest that 42dot LLM-SFT users should be aware of those limitations and take necessary steps to mitigate those issues.
57
 
58
  ## Disclaimer
59
  The contents generated by 42dot LLM series ("42dot LLM") do not necessarily reflect the views or opinions of 42dot Inc. ("42dot"). 42dot disclaims any and all liability to any part for any direct, indirect, implied, punitive, special, incidental, or other consequential damages arising from any use of the 42dot LLM and its generated contents.
60
 
61
  ## License
62
- The 42dot LLM-SFT is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).
63
 
64
  ## Citation
65
 
66
  ```
67
  @misc{42dot2023llm,
68
  title={42dot LLM: A Series of Large Language Model by 42dot},
69
- author={42dot Inc.},
70
  year={2023},
71
  url = {https://github.com/42dot/42dot_LLM},
72
- version = {1.0.0},
73
  }
74
  ```
 
10
  - 42dot_llm
11
  license: cc-by-nc-4.0
12
  ---
13
+ # 42dot-SFT-1.3B
14
 
15
+ **42dot-SFT** is a large language model (LLM) developed by [**42dot**](https://42dot.ai/) which is trained to follow natural language instructions.
16
+ 42dot-SFT is a part of **42dot LLM**, and derived from **42dot-PLM** by supervised fine-tuning (SFT). This repository contains a 1.3B-parameter version.
17
 
18
  ## Model Description
19
 
20
  ### Hyperparameters
21
+ As same as 42dot-PLM, the model is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
22
 
23
  | Params | Layers | Attention heads | Hidden size | FFN size | Max. length\* |
24
  | -- | -- | -- | -- | -- | -- |
 
30
  Fine-tuning took about 112 GPU hours (in NVIDIA A100). For the training dataset, we manually constructed (question or insturuction) and response pairs, which can either be single- or multi-turn.
31
 
32
  ### Evaluation
33
+ Inspired by recent attempts like [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/#how-good-is-vicuna), we evaluate 42dot-PLM with other proprietary/open-sourced chatbots using GPT-4 for assessing various aspects of responses. The evaluation dataset consists of 121 prompts over 10 categories. The sample of the evaluation dataset and prompt template can be downloaded from our [GitHub repo](https://github.com/42dot/42dot_LLM).
34
 
35
  - Baselines:
36
  - [ChatGPT](https://chat.openai.com/) using GPT-3.5-turbo and GPT-4
37
  - [Bard](https://bard.google.com/)
38
  - [KORani-v2-13B](https://huggingface.co/KRAFTON/KORani-v1-13B)
39
 
40
+ | Model | GPT-3.5 | GPT-4 | Bard | KORani | 42dot-SFT |
41
  | :-- |:-------:|:--------:|:--------:|:------:|:---------:|
42
  | Params | Unknown | Unknown | Unknown | 13B | 1.3B |
43
 
44
  <figure align="center">
45
+ <img src="https://huggingface.co/42dot/42dot-sft-1.3b/resolve/main/asset/Ko-Score.png" width="90%" height="70%"/>
46
  <figcaption><b>Response quality evaluation result</b></figcaption>
47
  </figure>
48
 
49
  <figure align="center">
50
+ <img src="https://huggingface.co/42dot/42dot-sft-1.3b/resolve/main/asset/42dot-SFT-vs.png" width="90%" height="70%"/>
51
+ <figcaption><b>Comparison between proprietary chatbots and 42dot-SFT</b></figcaption>
52
  </figure>
53
 
54
 
55
  ## Limitations and Ethical Considerations
56
+ 42dot-SFT shares a number of well-known limitations of other LLMs. For example, it may generate false and misinformative content since 42dot-SFT is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot-SFT may generate toxic, harmful, and biased content due to the use of web-available training data in the pre-training phase. We strongly suggest that 42dot-SFT users should be aware of those limitations and take necessary steps to mitigate those issues.
57
 
58
  ## Disclaimer
59
  The contents generated by 42dot LLM series ("42dot LLM") do not necessarily reflect the views or opinions of 42dot Inc. ("42dot"). 42dot disclaims any and all liability to any part for any direct, indirect, implied, punitive, special, incidental, or other consequential damages arising from any use of the 42dot LLM and its generated contents.
60
 
61
  ## License
62
+ The 42dot-SFT is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).
63
 
64
  ## Citation
65
 
66
  ```
67
  @misc{42dot2023llm,
68
  title={42dot LLM: A Series of Large Language Model by 42dot},
69
+ author={Woo-Jong Ryu and Sang-Kil Park and Jinwoo Park and Seongmin Lee and Yongkeun Hwang},
70
  year={2023},
71
  url = {https://github.com/42dot/42dot_LLM},
72
+ version = {pre-release},
73
  }
74
  ```
added_tokens.json CHANGED
@@ -2,49 +2,5 @@
2
  "<|endoftext|>": 50256,
3
  "<||bos||>": 50257,
4
  "<||pad||>": 50258,
5
- "<||unk||>": 50259,
6
- "<||unused10||>": 50269,
7
- "<||unused11||>": 50270,
8
- "<||unused12||>": 50271,
9
- "<||unused13||>": 50272,
10
- "<||unused14||>": 50273,
11
- "<||unused15||>": 50274,
12
- "<||unused16||>": 50275,
13
- "<||unused17||>": 50276,
14
- "<||unused18||>": 50277,
15
- "<||unused19||>": 50278,
16
- "<||unused1||>": 50260,
17
- "<||unused20||>": 50279,
18
- "<||unused21||>": 50280,
19
- "<||unused22||>": 50281,
20
- "<||unused23||>": 50282,
21
- "<||unused24||>": 50283,
22
- "<||unused25||>": 50284,
23
- "<||unused26||>": 50285,
24
- "<||unused27||>": 50286,
25
- "<||unused28||>": 50287,
26
- "<||unused29||>": 50288,
27
- "<||unused2||>": 50261,
28
- "<||unused30||>": 50289,
29
- "<||unused31||>": 50290,
30
- "<||unused32||>": 50291,
31
- "<||unused33||>": 50292,
32
- "<||unused34||>": 50293,
33
- "<||unused35||>": 50294,
34
- "<||unused36||>": 50295,
35
- "<||unused37||>": 50296,
36
- "<||unused38||>": 50297,
37
- "<||unused39||>": 50298,
38
- "<||unused3||>": 50262,
39
- "<||unused40||>": 50299,
40
- "<||unused41||>": 50300,
41
- "<||unused42||>": 50301,
42
- "<||unused43||>": 50302,
43
- "<||unused44||>": 50303,
44
- "<||unused4||>": 50263,
45
- "<||unused5||>": 50264,
46
- "<||unused6||>": 50265,
47
- "<||unused7||>": 50266,
48
- "<||unused8||>": 50267,
49
- "<||unused9||>": 50268
50
  }
 
2
  "<|endoftext|>": 50256,
3
  "<||bos||>": 50257,
4
  "<||pad||>": 50258,
5
+ "<||unk||>": 50259
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  }
asset/42dot-SFT-vs.png ADDED
asset/42dot_LLM_vs_score.png DELETED
Binary file (92.5 kB)
 
asset/42dot_llm_ko_score_white_background.png DELETED
Binary file (57.7 kB)
 
asset/Ko-Score.png ADDED
special_tokens_map.json CHANGED
@@ -1,50 +1,4 @@
1
  {
2
- "additional_special_tokens": [
3
- "<||unused1||>",
4
- "<||unused2||>",
5
- "<||unused3||>",
6
- "<||unused4||>",
7
- "<||unused5||>",
8
- "<||unused6||>",
9
- "<||unused7||>",
10
- "<||unused8||>",
11
- "<||unused9||>",
12
- "<||unused10||>",
13
- "<||unused11||>",
14
- "<||unused12||>",
15
- "<||unused13||>",
16
- "<||unused14||>",
17
- "<||unused15||>",
18
- "<||unused16||>",
19
- "<||unused17||>",
20
- "<||unused18||>",
21
- "<||unused19||>",
22
- "<||unused20||>",
23
- "<||unused21||>",
24
- "<||unused22||>",
25
- "<||unused23||>",
26
- "<||unused24||>",
27
- "<||unused25||>",
28
- "<||unused26||>",
29
- "<||unused27||>",
30
- "<||unused28||>",
31
- "<||unused29||>",
32
- "<||unused30||>",
33
- "<||unused31||>",
34
- "<||unused32||>",
35
- "<||unused33||>",
36
- "<||unused34||>",
37
- "<||unused35||>",
38
- "<||unused36||>",
39
- "<||unused37||>",
40
- "<||unused38||>",
41
- "<||unused39||>",
42
- "<||unused40||>",
43
- "<||unused41||>",
44
- "<||unused42||>",
45
- "<||unused43||>",
46
- "<||unused44||>"
47
- ],
48
  "bos_token": "<||bos||>",
49
  "eos_token": {
50
  "content": "<|endoftext|>",
 
1
  {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  "bos_token": "<||bos||>",
3
  "eos_token": {
4
  "content": "<|endoftext|>",
tokenizer.json CHANGED
@@ -38,402 +38,6 @@
38
  "rstrip": false,
39
  "normalized": false,
40
  "special": true
41
- },
42
- {
43
- "id": 50260,
44
- "content": "<||unused1||>",
45
- "single_word": false,
46
- "lstrip": false,
47
- "rstrip": false,
48
- "normalized": false,
49
- "special": true
50
- },
51
- {
52
- "id": 50261,
53
- "content": "<||unused2||>",
54
- "single_word": false,
55
- "lstrip": false,
56
- "rstrip": false,
57
- "normalized": false,
58
- "special": true
59
- },
60
- {
61
- "id": 50262,
62
- "content": "<||unused3||>",
63
- "single_word": false,
64
- "lstrip": false,
65
- "rstrip": false,
66
- "normalized": false,
67
- "special": true
68
- },
69
- {
70
- "id": 50263,
71
- "content": "<||unused4||>",
72
- "single_word": false,
73
- "lstrip": false,
74
- "rstrip": false,
75
- "normalized": false,
76
- "special": true
77
- },
78
- {
79
- "id": 50264,
80
- "content": "<||unused5||>",
81
- "single_word": false,
82
- "lstrip": false,
83
- "rstrip": false,
84
- "normalized": false,
85
- "special": true
86
- },
87
- {
88
- "id": 50265,
89
- "content": "<||unused6||>",
90
- "single_word": false,
91
- "lstrip": false,
92
- "rstrip": false,
93
- "normalized": false,
94
- "special": true
95
- },
96
- {
97
- "id": 50266,
98
- "content": "<||unused7||>",
99
- "single_word": false,
100
- "lstrip": false,
101
- "rstrip": false,
102
- "normalized": false,
103
- "special": true
104
- },
105
- {
106
- "id": 50267,
107
- "content": "<||unused8||>",
108
- "single_word": false,
109
- "lstrip": false,
110
- "rstrip": false,
111
- "normalized": false,
112
- "special": true
113
- },
114
- {
115
- "id": 50268,
116
- "content": "<||unused9||>",
117
- "single_word": false,
118
- "lstrip": false,
119
- "rstrip": false,
120
- "normalized": false,
121
- "special": true
122
- },
123
- {
124
- "id": 50269,
125
- "content": "<||unused10||>",
126
- "single_word": false,
127
- "lstrip": false,
128
- "rstrip": false,
129
- "normalized": false,
130
- "special": true
131
- },
132
- {
133
- "id": 50270,
134
- "content": "<||unused11||>",
135
- "single_word": false,
136
- "lstrip": false,
137
- "rstrip": false,
138
- "normalized": false,
139
- "special": true
140
- },
141
- {
142
- "id": 50271,
143
- "content": "<||unused12||>",
144
- "single_word": false,
145
- "lstrip": false,
146
- "rstrip": false,
147
- "normalized": false,
148
- "special": true
149
- },
150
- {
151
- "id": 50272,
152
- "content": "<||unused13||>",
153
- "single_word": false,
154
- "lstrip": false,
155
- "rstrip": false,
156
- "normalized": false,
157
- "special": true
158
- },
159
- {
160
- "id": 50273,
161
- "content": "<||unused14||>",
162
- "single_word": false,
163
- "lstrip": false,
164
- "rstrip": false,
165
- "normalized": false,
166
- "special": true
167
- },
168
- {
169
- "id": 50274,
170
- "content": "<||unused15||>",
171
- "single_word": false,
172
- "lstrip": false,
173
- "rstrip": false,
174
- "normalized": false,
175
- "special": true
176
- },
177
- {
178
- "id": 50275,
179
- "content": "<||unused16||>",
180
- "single_word": false,
181
- "lstrip": false,
182
- "rstrip": false,
183
- "normalized": false,
184
- "special": true
185
- },
186
- {
187
- "id": 50276,
188
- "content": "<||unused17||>",
189
- "single_word": false,
190
- "lstrip": false,
191
- "rstrip": false,
192
- "normalized": false,
193
- "special": true
194
- },
195
- {
196
- "id": 50277,
197
- "content": "<||unused18||>",
198
- "single_word": false,
199
- "lstrip": false,
200
- "rstrip": false,
201
- "normalized": false,
202
- "special": true
203
- },
204
- {
205
- "id": 50278,
206
- "content": "<||unused19||>",
207
- "single_word": false,
208
- "lstrip": false,
209
- "rstrip": false,
210
- "normalized": false,
211
- "special": true
212
- },
213
- {
214
- "id": 50279,
215
- "content": "<||unused20||>",
216
- "single_word": false,
217
- "lstrip": false,
218
- "rstrip": false,
219
- "normalized": false,
220
- "special": true
221
- },
222
- {
223
- "id": 50280,
224
- "content": "<||unused21||>",
225
- "single_word": false,
226
- "lstrip": false,
227
- "rstrip": false,
228
- "normalized": false,
229
- "special": true
230
- },
231
- {
232
- "id": 50281,
233
- "content": "<||unused22||>",
234
- "single_word": false,
235
- "lstrip": false,
236
- "rstrip": false,
237
- "normalized": false,
238
- "special": true
239
- },
240
- {
241
- "id": 50282,
242
- "content": "<||unused23||>",
243
- "single_word": false,
244
- "lstrip": false,
245
- "rstrip": false,
246
- "normalized": false,
247
- "special": true
248
- },
249
- {
250
- "id": 50283,
251
- "content": "<||unused24||>",
252
- "single_word": false,
253
- "lstrip": false,
254
- "rstrip": false,
255
- "normalized": false,
256
- "special": true
257
- },
258
- {
259
- "id": 50284,
260
- "content": "<||unused25||>",
261
- "single_word": false,
262
- "lstrip": false,
263
- "rstrip": false,
264
- "normalized": false,
265
- "special": true
266
- },
267
- {
268
- "id": 50285,
269
- "content": "<||unused26||>",
270
- "single_word": false,
271
- "lstrip": false,
272
- "rstrip": false,
273
- "normalized": false,
274
- "special": true
275
- },
276
- {
277
- "id": 50286,
278
- "content": "<||unused27||>",
279
- "single_word": false,
280
- "lstrip": false,
281
- "rstrip": false,
282
- "normalized": false,
283
- "special": true
284
- },
285
- {
286
- "id": 50287,
287
- "content": "<||unused28||>",
288
- "single_word": false,
289
- "lstrip": false,
290
- "rstrip": false,
291
- "normalized": false,
292
- "special": true
293
- },
294
- {
295
- "id": 50288,
296
- "content": "<||unused29||>",
297
- "single_word": false,
298
- "lstrip": false,
299
- "rstrip": false,
300
- "normalized": false,
301
- "special": true
302
- },
303
- {
304
- "id": 50289,
305
- "content": "<||unused30||>",
306
- "single_word": false,
307
- "lstrip": false,
308
- "rstrip": false,
309
- "normalized": false,
310
- "special": true
311
- },
312
- {
313
- "id": 50290,
314
- "content": "<||unused31||>",
315
- "single_word": false,
316
- "lstrip": false,
317
- "rstrip": false,
318
- "normalized": false,
319
- "special": true
320
- },
321
- {
322
- "id": 50291,
323
- "content": "<||unused32||>",
324
- "single_word": false,
325
- "lstrip": false,
326
- "rstrip": false,
327
- "normalized": false,
328
- "special": true
329
- },
330
- {
331
- "id": 50292,
332
- "content": "<||unused33||>",
333
- "single_word": false,
334
- "lstrip": false,
335
- "rstrip": false,
336
- "normalized": false,
337
- "special": true
338
- },
339
- {
340
- "id": 50293,
341
- "content": "<||unused34||>",
342
- "single_word": false,
343
- "lstrip": false,
344
- "rstrip": false,
345
- "normalized": false,
346
- "special": true
347
- },
348
- {
349
- "id": 50294,
350
- "content": "<||unused35||>",
351
- "single_word": false,
352
- "lstrip": false,
353
- "rstrip": false,
354
- "normalized": false,
355
- "special": true
356
- },
357
- {
358
- "id": 50295,
359
- "content": "<||unused36||>",
360
- "single_word": false,
361
- "lstrip": false,
362
- "rstrip": false,
363
- "normalized": false,
364
- "special": true
365
- },
366
- {
367
- "id": 50296,
368
- "content": "<||unused37||>",
369
- "single_word": false,
370
- "lstrip": false,
371
- "rstrip": false,
372
- "normalized": false,
373
- "special": true
374
- },
375
- {
376
- "id": 50297,
377
- "content": "<||unused38||>",
378
- "single_word": false,
379
- "lstrip": false,
380
- "rstrip": false,
381
- "normalized": false,
382
- "special": true
383
- },
384
- {
385
- "id": 50298,
386
- "content": "<||unused39||>",
387
- "single_word": false,
388
- "lstrip": false,
389
- "rstrip": false,
390
- "normalized": false,
391
- "special": true
392
- },
393
- {
394
- "id": 50299,
395
- "content": "<||unused40||>",
396
- "single_word": false,
397
- "lstrip": false,
398
- "rstrip": false,
399
- "normalized": false,
400
- "special": true
401
- },
402
- {
403
- "id": 50300,
404
- "content": "<||unused41||>",
405
- "single_word": false,
406
- "lstrip": false,
407
- "rstrip": false,
408
- "normalized": false,
409
- "special": true
410
- },
411
- {
412
- "id": 50301,
413
- "content": "<||unused42||>",
414
- "single_word": false,
415
- "lstrip": false,
416
- "rstrip": false,
417
- "normalized": false,
418
- "special": true
419
- },
420
- {
421
- "id": 50302,
422
- "content": "<||unused43||>",
423
- "single_word": false,
424
- "lstrip": false,
425
- "rstrip": false,
426
- "normalized": false,
427
- "special": true
428
- },
429
- {
430
- "id": 50303,
431
- "content": "<||unused44||>",
432
- "single_word": false,
433
- "lstrip": false,
434
- "rstrip": false,
435
- "normalized": false,
436
- "special": true
437
  }
438
  ],
439
  "normalizer": null,
 
38
  "rstrip": false,
39
  "normalized": false,
40
  "special": true
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  }
42
  ],
43
  "normalizer": null,
tokenizer_config.json CHANGED
@@ -3,7 +3,7 @@
3
  "add_prefix_space": false,
4
  "bos_token": {
5
  "__type": "AddedToken",
6
- "content": "<||bos||>",
7
  "lstrip": false,
8
  "normalized": true,
9
  "rstrip": false,
@@ -25,7 +25,7 @@
25
  "tokenizer_class": "GPT2Tokenizer",
26
  "unk_token": {
27
  "__type": "AddedToken",
28
- "content": "<||unk||>",
29
  "lstrip": false,
30
  "normalized": true,
31
  "rstrip": false,
 
3
  "add_prefix_space": false,
4
  "bos_token": {
5
  "__type": "AddedToken",
6
+ "content": "<|endoftext|>",
7
  "lstrip": false,
8
  "normalized": true,
9
  "rstrip": false,
 
25
  "tokenizer_class": "GPT2Tokenizer",
26
  "unk_token": {
27
  "__type": "AddedToken",
28
+ "content": "<|endoftext|>",
29
  "lstrip": false,
30
  "normalized": true,
31
  "rstrip": false,