Create Initial Model Card for flan-t5-xl-lora

README.md

@@ -0,0 +1,96 @@
+---
+license: apache-2.0
+datasets:
+- tatsu-lab/alpaca
+---
+
+## 🍮 🦙 Flan-Alpaca: Instruction Tuning from Humans and Machines
+
+Our [repository](https://github.com/declare-lab/flan-alpaca) contains code for extending the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)
+synthetic instruction tuning to existing instruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416).
+The pretrained models and demos are available on HuggingFace 🤗 :
+
+| Model                                                                     | Parameters | Training GPUs   |
+|---------------------------------------------------------------------------|------------|-----------------|
+| [Flan-Alpaca-Base](https://huggingface.co/declare-lab/flan-alpaca-base)   | 220M       | 1x A6000        |
+| [Flan-Alpaca-Large](https://huggingface.co/declare-lab/flan-alpaca-large) | 770M       | 1x A6000        |
+| [Flan-Alpaca-XL](https://huggingface.co/declare-lab/flan-alpaca-xl)       | 3B         | 1x A6000        |
+| [Flan-Alpaca-XXL](https://huggingface.co/declare-lab/flan-alpaca-xxl)     | 11B        | 4x A6000 (FSDP) |
+
+### Why?
+
+[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represents an exciting new direction
+to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
+Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
+The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
+However, the original implementation is less accessible due to licensing constraints of the
+underlying [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) model.
+Furthermore, users have noted [potential noise](https://github.com/tloen/alpaca-lora/issues/65) in the synthetic
+dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
+less diverse) instructions such as [Flan-T5](https://arxiv.org/abs/2210.11416).
+
+### Usage
+This uses Huggingface PEFT library for Parameter Efficient Fine Tuning
+
+```
+import torch
+from peft import PeftModel
+from transformers import GenerationConfig
+
+
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+
+BASE_MODEL = "google/flan-t5-xl"
+LORA_WEIGHTS = "declare-lab/flan-alpaca-xl-lora"
+TEMPERATURE = 1.0
+TOP_P = 0.75
+TOP_K = 40
+NUM_BEAMS = 4
+MAX_NEW_TOKENS = 128
+
+if torch.cuda.is_available():
+    device = "cuda"
+else:
+    device = "cpu"
+
+
+if device == "cuda":
+    model = AutoModelForSeq2SeqLM.from_pretrained(
+        BASE_MODEL,
+        device_map="auto",
+    )
+    model = PeftModel.from_pretrained(model, LORA_WEIGHTS, force_download=True)
+else:
+    model = AutoModelForSeq2SeqLM.from_pretrained(
+        BASE_MODEL, device_map={"": device}, low_cpu_mem_usage=True
+    )
+    model = PeftModel.from_pretrained(
+        model,
+        LORA_WEIGHTS,
+        device_map={"": device},
+    )
+
+
+prompt = "Write a short email to show that 42 is the optimal seed for training neural networks"
+
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+input_ids = input_ids.to(device)
+
+generation_config = GenerationConfig(
+    temperature=TEMPERATURE,
+    top_p=TOP_P,
+    top_k=TOP_K,
+    num_beams=NUM_BEAMS,
+)
+generation_output = model.generate(
+    input_ids=input_ids,
+    generation_config=generation_config,
+    return_dict_in_generate=True,
+    output_scores=True,
+    max_new_tokens=MAX_NEW_TOKENS,
+)
+print(tokenizer.batch_decode(generation_output.sequences)[0])
+
+```