mistral-7b-alpaca-qlora

Fine-tuned version of mistralai/Mistral-7B-v0.1 using QLoRA on the Alpaca dataset.

Training details

• Base model: mistralai/Mistral-7B-v0.1
• Dataset: tatsu-lab/alpaca (52k instruction examples)
• Method: QLoRA (4-bit NF4, LoRA r=8)
• Hardware: NVIDIA RTX 3060 12 GB
• Sequence length: 256 tokens
• Epochs: 3
• Training time: 18hrs

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

tokenizer = AutoTokenizer.from_pretrained("4x32/mistral-7b-alpaca-qlora")
base = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "4x32/mistral-7b-alpaca-qlora")

prompt = (
    "Below is an instruction that describes a task. "
    "Write a response that appropriately completes the request.\n\n"
    "### Instruction:\nExplain what a neural network is.\n\n"
    "### Response:\n"
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))