Initial commit

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,74 @@
+---
+language:
+- en
+- ko
+pipeline_tag: text-generation
+tags:
+- pytorch
+- llama
+- causal-lm
+- 42dot_llm
+license: cc-by-nc-4.0
+---
+# 42dot_LLM-SFT-1.3B
+
+**42dot LLM-SFT** is a large language model (LLM) developed by [**42dot**](https://42dot.ai/) which is trained to follow natural language instructions.
+42dot LLM-SFT is a part of **42dot LLM**, and derived from **42dot LLM-PLM** by supervised fine-tuning (SFT). This repository contains a 1.3B-parameter version.
+
+## Model Description
+
+### Hyperparameters
+As same as 42dot LLM-PLM, the model is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
+
+| Params | Layers | Attention heads | Hidden size | FFN size | Max. length\* |
+| -- | -- | -- | -- | -- | -- |
+| 1.3B | 24 | 32 | 2,048 | 5,632 | 4,096 |
+
+(\* unit: tokens)
+### Supervised Fine-tuning
+
+Fine-tuning took about 112 GPU hours (in NVIDIA A100). For the training dataset, we manually constructed  (question or insturuction) and response pairs, which can either be single- or multi-turn. 
+
+### Evaluation
+Inspired by recent attempts like [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/#how-good-is-vicuna), we evaluate 42dot LLM-SFT with other proprietary/open-sourced chatbots using GPT-4 for assessing various aspects of responses. The evaluation dataset consists of 121 prompts over 10 categories. The sample of the evaluation dataset and prompt template can be downloaded from our [GitHub repo](https://github.com/42dot/42dot_LLM).
+
+- Baselines:
+  - [ChatGPT](https://chat.openai.com/) using GPT-3.5-turbo and GPT-4
+  - [Bard](https://bard.google.com/)
+  - [KORani-v2-13B](https://huggingface.co/KRAFTON/KORani-v1-13B)
+
+| Model | GPT-3.5 |  GPT-4   |   Bard   | KORani | 42dot LLM-SFT |
+| :-- |:-------:|:--------:|:--------:|:------:|:---------:|
+| Params | Unknown | Unknown | Unknown |  13B   |   1.3B    |
+
+<figure align="center">
+<img src="https://huggingface.co/42dot/42dot_LLM-SFT-1.3B/resolve/main/asset/42dot_llm_ko_score_white_background.png"/>
+<figcaption><b>Response quality evaluation result</b></figcaption>
+</figure>
+
+<figure align="center">
+<img src="https://huggingface.co/42dot/42dot_LLM-SFT-1.3B/resolve/main/asset/42dot_LLM_vs_score.png"/>
+<figcaption><b>Comparison between proprietary chatbots and 42dot LLM-SFT</b></figcaption>
+</figure>
+
+
+## Limitations and Ethical Considerations
+42dot LLM-SFT shares a number of well-known limitations of other LLMs. For example, it may generate false and misinformative content since 42dot LLM-SFT is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot LLM-SFT may generate toxic, harmful, and biased content due to the use of web-available training data in the pre-training phase. We strongly suggest that 42dot LLM-SFT users should be aware of those limitations and take necessary steps to mitigate those issues.
+
+## Disclaimer
+The contents generated by 42dot LLM series ("42dot LLM") do not necessarily reflect the views or opinions of 42dot Inc. ("42dot"). 42dot disclaims any and all liability to any part for any direct, indirect, implied, punitive, special, incidental, or other consequential damages arising from any use of the 42dot LLM and its generated contents.
+
+## License
+The 42dot LLM-SFT is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).
+
+## Citation
+
+```
+@misc{42dot2023llm,
+      title={42dot LLM: A Series of Large Language Model by 42dot},
+      author={42dot Inc.},
+      year={2023},
+      url = {https://github.com/42dot/42dot_LLM},
+      version = {1.0.0},
+}
+```

added_tokens.json

@@ -0,0 +1,6 @@
+{
+  "<|endoftext|>": 50256,
+  "<||bos||>": 50257,
+  "<||pad||>": 50258,
+  "<||unk||>": 50259
+}

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "42dot-SFT-1.3B",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 50257,
+  "eos_token_id": 50256,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.01,
+  "intermediate_size": 5632,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 32,
+  "pad_token_id": 50258,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "tokenizer_class": "GPT2TokenizerFast",
+  "torch_dtype": "float32",
+  "transformers_version": "4.33.1",
+  "use_cache": false,
+  "vocab_size": 50304
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50257,
+  "eos_token_id": 50256,
+  "pad_token_id": 50258,
+  "transformers_version": "4.33.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed4607610d6b55736588cda7d05e5ea718cb239d856f98681397e428410a1b88
+size 5757108800

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40c4e4a9b32503e3f5e66b6325dde81a368fc27147f62f80e0dca5d5eae6f3d8
+size 5757154933

special_tokens_map.json

@@ -0,0 +1,12 @@
+{
+  "bos_token": "<||bos||>",
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<||pad||>",
+  "unk_token": "<||unk||>"
+}

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "add_bos_token": true,
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": true,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "model_max_length": 8192,
+  "pad_token": null,
+  "padding_side": "right",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}