Create README.md

README.md

@@ -0,0 +1,111 @@
+---
+license: other
+datasets:
+- blip-solutions/SlovAlpaca
+language:
+- sk
+---
+
+# SlovAlpaca
+
+This repository contains the LORA weights finetuned on the translated version of the original Alpaca dataset (more info on the dataset card)
+
+## Training procedure
+
+The training was done on the 7B LLaMA model (decapoda-research/llama-7b-hf) quantized to 8bits with following Hyperparameters:
+
+```
+MICRO_BATCH_SIZE = 3 
+BATCH_SIZE = 128
+GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
+EPOCHS = 2  # paper uses 3
+LEARNING_RATE = 2e-5  # from the original paper
+CUTOFF_LEN = 256  # 256 accounts for about 96% of the data
+LORA_R = 4
+LORA_ALPHA = 16
+LORA_DROPOUT = 0.05
+```
+
+The sole goal of this project is to explore the effects of single language finetuning using the same dataset and methods as the original paper did and comapre the results
+
+@misc{alpaca,
+  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
+  title = {Stanford Alpaca: An Instruction-following LLaMA model},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
+}
+
+## How to use:
+
+### Prerequisites
+```
+!pip install datasets loralib sentencepiece
+!pip uninstall -y transformers
+!pip install git+https://github.com/zphang/transformers@c3dc391#egg=transformers 
+!pip install git+https://github.com/huggingface/peft.git
+!pip install bitsandbytes
+```
+
+
+### Load model:
+
+```
+from peft import PeftModel
+from transformers import LLaMATokenizer, LLaMAForCausalLM, GenerationConfig
+
+tokenizer = LLaMATokenizer.from_pretrained("decapoda-research/llama-7b-hf")
+
+model = LLaMAForCausalLM.from_pretrained(
+    "decapoda-research/llama-7b-hf",
+    load_in_8bit=True,
+    device_map="auto",
+)
+
+model = PeftModel.from_pretrained(model, "blip-solutions/SlovAlpaca")
+```
+
+### Generation
+
+Here is a colab notebook for inference: https://colab.research.google.com/drive/1z4aMG7tGjchLBlg_iXDuqt3sH6bQRuQk?usp=sharing
+
+```
+PROMPT = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+Kde žijú lamy?
+### Response:"""
+
+inputs = tokenizer(
+    PROMPT,
+    return_tensors="pt",
+)
+input_ids = inputs["input_ids"].cuda()
+
+generation_config = GenerationConfig(
+    temperature=0.6,
+    top_p=0.95,
+    repetition_penalty=1.15,
+)
+print("Generating...")
+generation_output = model.generate(
+    input_ids=input_ids,
+    generation_config=generation_config,
+    return_dict_in_generate=True,
+    output_scores=True,
+    max_new_tokens=128,
+)
+for s in generation_output.sequences:
+    print(tokenizer.decode(s))
+```
+
+### Response:
+
+```
+Generating...
+ Below is an instruction that describes a task. Write a response that appropriately completes the request.
+### Instruction:
+Kde žijú lamy?
+### Response:
+Lamy žiju v horách, na poli, alebo v lesoch.
+```