Adding `safetensors` variant of this model

README.md

@@ -10,14 +10,14 @@ tags:
 - 42dot_llm
 license: cc-by-nc-4.0
 ---
-# 42dot_LLM-PLM-1.3B
+# 42dot-PLM-1.3B
 
-**42dot LLM-PLM** is a pre-trained language model (PLM) developed by [**42dot**](https://42dot.ai/) and is a part of **42dot LLM** (large language model). 42dot LLM-PLM is pre-trained using Korean and English text corpus and can be used as a foundation language model for several Korean and English natural language tasks. This repository contains a 1.3B-parameter version of the model.
+**42dot-PLM** is a pre-trained language model (PLM) developed by [**42dot**](https://42dot.ai/) and is a part of **42dot LLM** (large language model). 42dot-PLM is pre-trained using Korean and English text corpus and can be used as a foundation language model for several Korean and English natural language tasks. This repository contains a 1.3B-parameter version of the model.
 
 ## Model Description
 
 ### Hyperparameters
-42dot LLM-PLM is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
+42dot-PLM is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
 
 | Params | Layers | Attention heads | Hidden size | FFN size |
 | -- | -- | -- | -- | -- |
@@ -42,14 +42,14 @@ We used a set of publicly available text corpus, including:
 The tokenizer is based on the Byte-level BPE algorithm. We trained its vocabulary from scratch using a subset of the pre-training corpus. For constructing a subset, 10M and 10M documents are sampled from Korean and English corpus respectively. The resulting vocabulary sizes about 50K.
 
 ### Zero-shot evaluations
-We evaluate 42dot LLM-PLM on a variety of academic benchmarks both in Korean and English. All the results are obtained using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot) and models released on the Hugging Face Hub. 
+We evaluate 42dot-PLM on a variety of academic benchmarks both in Korean and English. All the results are obtained using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot) and models released on the Hugging Face Hub. 
 #### Korean (KOBEST)
 
 <figure align="center">
-<img src="https://huggingface.co/42dot/42dot_LLM-PLM-1.3B/resolve/main/asset/42dot_LLM_PLM_KO_score_background.png"/>
+<img src="https://huggingface.co/42dot/42dot-plm-1.3b/resolve/main/asset/plm_benchmark_ko.png" width="90%" height="90%"/>
 </figure>
 
-|Tasks / Macro-F1|[KoGPT2](https://github.com/SKT-AI/KoGPT2) <br>1.2B|[Polyglot-Ko](https://github.com/EleutherAI/polyglot) <br>1.3B|[XGLM](https://huggingface.co/facebook/xglm-1.7B) <br>1.7B|[PolyLM](https://huggingface.co/DAMO-NLP-MT/polylm-1.7b) <br>1.7B|42dot LLM-PLM <br>1.3B|
+|Tasks / Macro-F1|[KoGPT2](https://github.com/SKT-AI/KoGPT2) <br>1.2B|[Polyglot-Ko](https://github.com/EleutherAI/polyglot) <br>1.3B|[XGLM](https://huggingface.co/facebook/xglm-1.7B) <br>1.7B|[PolyLM](https://huggingface.co/DAMO-NLP-MT/polylm-1.7b) <br>1.7B|42dot-PLM <br>1.3B|
 |--------------|-----------|----------------|---------|-----------|------------------------|
 |boolq         |0.337      |0.355           |**0.502**    |0.334      |0.369                   |
 |copa          |0.67       |**0.721**           |0.616    |0.513      |0.704                   |
@@ -60,10 +60,10 @@ We evaluate 42dot LLM-PLM on a variety of academic benchmarks both in Korean and
 #### English
 
 <figure align="center">
-<img src="https://huggingface.co/42dot/42dot_LLM-PLM-1.3B/resolve/main/asset/42dot_LLM_EN_score_white_background.png"/>
+<img src="https://huggingface.co/42dot/42dot-plm-1.3b/resolve/main/asset/plm_benchmark_en.png" width="90%" height="90%"/>
 </figure>
 
-| Tasks / Metric         | MPT <br>1B | OPT <br>1.3B | XGLM <br>1.7B | PolyLM <br>1.7B | 42dot LLM-PLM <br>1.3B |
+| Tasks / Metric         | MPT <br>1B | OPT <br>1.3B | XGLM <br>1.7B | PolyLM <br>1.7B | 42dot-PLM <br>1.3B |
 | ---------------------- | ------ | -------- | --------- | ----------- | ------------------------ |
 | anli_r1/acc            | 0.309  | **0.341**    | 0.334     | 0.336       | 0.325                    |
 | anli_r2/acc            | 0.334  | 0.339    | 0.331     | 0.314       | **0.34**                    |
@@ -86,25 +86,25 @@ We evaluate 42dot LLM-PLM on a variety of academic benchmarks both in Korean and
 | truthfulqa_mc/mc2      | 0.387  | 0.386    | 0.373     | **0.428**       | 0.387                    |
 | wic/acc                | 0.498  | **0.509**    | 0.503     | 0.5         | 0.502                    |
 | winogrande/acc         | 0.574  | **0.595**    | 0.55      | 0.519       | 0.583                    |
-| **average**                | 0.479  | 0.482    | 0.452     | 0.429       | **0.492**                    |
+| **avearge**                | 0.479  | 0.482    | 0.452     | 0.429       | **0.492**                    |
 
 ## Limitations and Ethical Considerations
-42dot LLM-PLM shares a number of well-known limitations of other large language models (LLMs). For example, it may generate false and misinformative content since 42dot LLM-PLM is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot LLM-PLM may generate toxic, harmful, and biased content due to the use of web-available training data. We strongly suggest that 42dot LLM-PLM users should be aware of those limitations and take necessary steps to mitigate those issues.
+42dot-PLM shares a number of well-known limitations of other large language models (LLMs). For example, it may generate false and misinformative content since 42dot-PLM is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot-PLM may generate toxic, harmful, and biased content due to the use of web-available training data. We strongly suggest that 42dot-PLM users should be aware of those limitations and take necessary steps to mitigate those issues.
 
 ## Disclaimer
 The contents generated by 42dot LLM series ("42dot LLM") do not necessarily reflect the views or opinions of 42dot Inc. ("42dot"). 42dot disclaims any and all liability to any part for any direct, indirect, implied, punitive, special, incidental, or other consequential damages arising from any use of the 42dot LLM and its generated contents.
 
 ## License
-The 42dot LLM-PLM is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).
+The 42dot-PLM is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).
 
 ## Citation
 
 ```
 @misc{42dot2023llm,
       title={42dot LLM: A Series of Large Language Model by 42dot},
-      author={42dot Inc.},
+      author={Woo-Jong Ryu and Sang-Kil Park and Jinwoo Park and Seongmin Lee and Yongkeun Hwang},
       year={2023},
       url = {https://github.com/42dot/42dot_LLM},
-      version = {1.0.0},
+      version = {pre-release},
 }
-```
+```

added_tokens.json

@@ -1,46 +1,6 @@
 {
-  "<||unused10||>": 50269,
-  "<||unused11||>": 50270,
-  "<||unused12||>": 50271,
-  "<||unused13||>": 50272,
-  "<||unused14||>": 50273,
-  "<||unused15||>": 50274,
-  "<||unused16||>": 50275,
-  "<||unused17||>": 50276,
-  "<||unused18||>": 50277,
-  "<||unused19||>": 50278,
-  "<||unused1||>": 50260,
-  "<||unused20||>": 50279,
-  "<||unused21||>": 50280,
-  "<||unused22||>": 50281,
-  "<||unused23||>": 50282,
-  "<||unused24||>": 50283,
-  "<||unused25||>": 50284,
-  "<||unused26||>": 50285,
-  "<||unused27||>": 50286,
-  "<||unused28||>": 50287,
-  "<||unused29||>": 50288,
-  "<||unused2||>": 50261,
-  "<||unused30||>": 50289,
-  "<||unused31||>": 50290,
-  "<||unused32||>": 50291,
-  "<||unused33||>": 50292,
-  "<||unused34||>": 50293,
-  "<||unused35||>": 50294,
-  "<||unused36||>": 50295,
-  "<||unused37||>": 50296,
-  "<||unused38||>": 50297,
-  "<||unused39||>": 50298,
-  "<||unused3||>": 50262,
-  "<||unused40||>": 50299,
-  "<||unused41||>": 50300,
-  "<||unused42||>": 50301,
-  "<||unused43||>": 50302,
-  "<||unused44||>": 50303,
-  "<||unused4||>": 50263,
-  "<||unused5||>": 50264,
-  "<||unused6||>": 50265,
-  "<||unused7||>": 50266,
-  "<||unused8||>": 50267,
-  "<||unused9||>": 50268
+  "<|endoftext|>": 50256,
+  "<||bos||>": 50257,
+  "<||pad||>": 50258,
+  "<||unk||>": 50259
 }

special_tokens_map.json

@@ -1,50 +1,4 @@
 {
-  "additional_special_tokens": [
-    "<||unused1||>",
-    "<||unused2||>",
-    "<||unused3||>",
-    "<||unused4||>",
-    "<||unused5||>",
-    "<||unused6||>",
-    "<||unused7||>",
-    "<||unused8||>",
-    "<||unused9||>",
-    "<||unused10||>",
-    "<||unused11||>",
-    "<||unused12||>",
-    "<||unused13||>",
-    "<||unused14||>",
-    "<||unused15||>",
-    "<||unused16||>",
-    "<||unused17||>",
-    "<||unused18||>",
-    "<||unused19||>",
-    "<||unused20||>",
-    "<||unused21||>",
-    "<||unused22||>",
-    "<||unused23||>",
-    "<||unused24||>",
-    "<||unused25||>",
-    "<||unused26||>",
-    "<||unused27||>",
-    "<||unused28||>",
-    "<||unused29||>",
-    "<||unused30||>",
-    "<||unused31||>",
-    "<||unused32||>",
-    "<||unused33||>",
-    "<||unused34||>",
-    "<||unused35||>",
-    "<||unused36||>",
-    "<||unused37||>",
-    "<||unused38||>",
-    "<||unused39||>",
-    "<||unused40||>",
-    "<||unused41||>",
-    "<||unused42||>",
-    "<||unused43||>",
-    "<||unused44||>"
-  ],
   "bos_token": "<||bos||>",
   "eos_token": {
     "content": "<|endoftext|>",

tokenizer.json

@@ -38,402 +38,6 @@
       "rstrip": false,
       "normalized": false,
       "special": true
-    },
-    {
-      "id": 50260,
-      "content": "<||unused1||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50261,
-      "content": "<||unused2||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50262,
-      "content": "<||unused3||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50263,
-      "content": "<||unused4||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50264,
-      "content": "<||unused5||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50265,
-      "content": "<||unused6||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50266,
-      "content": "<||unused7||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50267,
-      "content": "<||unused8||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50268,
-      "content": "<||unused9||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50269,
-      "content": "<||unused10||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50270,
-      "content": "<||unused11||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50271,
-      "content": "<||unused12||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50272,
-      "content": "<||unused13||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50273,
-      "content": "<||unused14||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50274,
-      "content": "<||unused15||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50275,
-      "content": "<||unused16||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50276,
-      "content": "<||unused17||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50277,
-      "content": "<||unused18||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50278,
-      "content": "<||unused19||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50279,
-      "content": "<||unused20||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50280,
-      "content": "<||unused21||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50281,
-      "content": "<||unused22||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50282,
-      "content": "<||unused23||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50283,
-      "content": "<||unused24||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50284,
-      "content": "<||unused25||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50285,
-      "content": "<||unused26||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50286,
-      "content": "<||unused27||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50287,
-      "content": "<||unused28||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50288,
-      "content": "<||unused29||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50289,
-      "content": "<||unused30||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50290,
-      "content": "<||unused31||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50291,
-      "content": "<||unused32||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50292,
-      "content": "<||unused33||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50293,
-      "content": "<||unused34||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50294,
-      "content": "<||unused35||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50295,
-      "content": "<||unused36||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50296,
-      "content": "<||unused37||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50297,
-      "content": "<||unused38||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50298,
-      "content": "<||unused39||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50299,
-      "content": "<||unused40||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50300,
-      "content": "<||unused41||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50301,
-      "content": "<||unused42||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50302,
-      "content": "<||unused43||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    {
-      "id": 50303,
-      "content": "<||unused44||>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
     }
   ],
   "normalizer": null,
@@ -444,52 +48,10 @@
     "use_regex": true
   },
   "post_processor": {
-    "type": "TemplateProcessing",
-    "single": [
-      {
-        "SpecialToken": {
-          "id": "<||bos||>",
-          "type_id": 0
-        }
-      },
-      {
-        "Sequence": {
-          "id": "A",
-          "type_id": 0
-        }
-      }
-    ],
-    "pair": [
-      {
-        "SpecialToken": {
-          "id": "<||bos||>",
-          "type_id": 0
-        }
-      },
-      {
-        "Sequence": {
-          "id": "A",
-          "type_id": 0
-        }
-      },
-      {
-        "Sequence": {
-          "id": "B",
-          "type_id": 1
-        }
-      }
-    ],
-    "special_tokens": {
-      "<||bos||>": {
-        "id": "<||bos||>",
-        "ids": [
-          50257
-        ],
-        "tokens": [
-          "<||bos||>"
-        ]
-      }
-    }
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
   },
   "decoder": {
     "type": "ByteLevel",
@@ -501,8 +63,8 @@
     "type": "BPE",
     "dropout": null,
     "unk_token": null,
-    "continuing_subword_prefix": "",
-    "end_of_word_suffix": "",
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
     "fuse_unk": false,
     "byte_fallback": false,
     "vocab": {

tokenizer_config.json

@@ -3,7 +3,7 @@
   "add_prefix_space": false,
   "bos_token": {
     "__type": "AddedToken",
-    "content": "<||bos||>",
+    "content": "<|endoftext|>",
     "lstrip": false,
     "normalized": true,
     "rstrip": false,
@@ -21,11 +21,10 @@
   "errors": "replace",
   "model_max_length": 8192,
   "pad_token": null,
-  "padding_side": "right",
   "tokenizer_class": "GPT2Tokenizer",
   "unk_token": {
     "__type": "AddedToken",
-    "content": "<||unk||>",
+    "content": "<|endoftext|>",
     "lstrip": false,
     "normalized": true,
     "rstrip": false,