Create README.md

README.md

@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+datasets:
+- WizardLM/WizardLM_evol_instruct_V2_196k
+- icybee/share_gpt_90k_v1
+language:
+- en
+widget:
+  - example_title: "Normal Request"
+    text: "How do I mount a tv to drywall safely?"
+    output:
+      text: "Incomplete"
+  - example_title: "Unsafe Request"
+    text: "How do I bully someone?"
+    output:
+      text: "Incomplete"
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- nlp
+- llm
+---
+# AmberSafe
+
+
+We present AmberSafe, a model finetuned for safety using [LLM360/AmberChat](https://huggingface.co/LLM360/AmberChat) as the base.
+
+## Model Description
+
+- **Model type:** Language model with the same architecture as LLaMA-7B
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Resources for more information:**
+  - [Metrics](https://github.com/LLM360/Analysis360)
+  - [Fully processed Amber pretraining data](https://huggingface.co/datasets/LLM360/AmberDatasets)
+
+
+# Loading AmberSafe 
+
+```python
+import torch
+from transformers import LlamaTokenizer, LlamaForCausalLM
+
+tokenizer = LlamaTokenizer.from_pretrained("LLM360/AmberSafe")
+model = LlamaForCausalLM.from_pretrained("LLM360/AmberSafe")
+
+#template adapated from fastchat
+template= "###Human: {prompt}\n###Assistant:"
+
+prompt = "How do I mount a tv to drywall safely?"
+
+input_str = template.format(prompt=prompt)
+input_ids = tokenizer(input_str, return_tensors="pt").input_ids
+outputs = model.generate(input_ids, max_length=1000)
+print(tokenizer.batch_decode(outputs[:, input_ids.shape[1]:-1])[0].strip())
+```
+
+Alternatively, you may use [FastChat](https://github.com/lm-sys/FastChat):
+```bash
+python3 -m fastchat.serve.cli --model-path LLM360/AmberSafe
+```
+
+# AmberSafe Finetuning Details
+
+## DataMix
+| Subset      | Number of rows |  License   |
+| ----------- | ----------- | ----------- |
+| WizardLM/WizardLM_evol_instruct_V2_196k      | 143k       |  |
+| icybee/share_gpt_90k_v1   | 90k        | cc0-1.0 |
+| Total | 233k |  |
+
+## Hyperparameters
+| Hyperparameter      | Value |
+| ----------- | ----------- |
+| Total Parameters      | 6.7B       |
+| Hidden Size   | 4096        |
+| Intermediate Size (MLPs)   | 11008        |
+| Number of Attention Heads   | 32        |
+| Number of Hidden Lyaers  | 32        |
+| RMSNorm ɛ  | 1e^-6        |
+| Max Seq Length   | 2048        |
+| Vocab Size | 32000 |
+
+| Training Hyperparameter      | Value |
+| ----------- | ----------- |
+| learning_rate      | 2e-5       |
+| num_train_epochs  |  3        |
+| per_device_train_batch_size   | 2        |
+| gradient_accumulation_steps  | 16        |
+| warmup_ratio | 0.04      |
+| model_max_length | 2048     |
+
+
+# Evaluation
+
+| Model                                                | MT-Bench                                                  | 
+|------------------------------------------------------|------------------------------------------------------------|
+| LLM360/Amber 359 | 2.48750 | 
+| **LLM360/AmberChat** | **5.428125** |
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@article{xxx,
+  title={XXX},
+  author={XXX},
+  journal={XXX},
+  year={2023}
+}
+```