Nagargpt-Finetuned
This repository contains LoRA adapters for the CreitinGameplays/bloom-3b-conversational model, fine-tuned on a Nepali municipality QA and conversational dataset.
Usage
To use this model, load the base model and apply the LoRA adapters:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = "CreitinGameplays/bloom-3b-conversational"
adapter_path = "your-username/nagargpt-finetuned"
tokenizer = AutoTokenizer.from_pretrained(adapter_path)
model = AutoModelForCausalLM.from_pretrained(base_model, load_in_4bit=True)
model = PeftModel.from_pretrained(model, adapter_path)
Training Details
- Base Model:
CreitinGameplays/bloom-3b-conversational - Dataset: Nepali municipality QA and conversational data (
yam3333/main_plus_additional_v2) - Fine-Tuning: QLoRA with 4-bit quantization, LoRA rank=8, alpha=16, dropout=0.05
- Target Modules:
query_key_value - Epochs: 3
- Batch Size: 1 (with gradient accumulation steps=8)
- Learning Rate: 2e-5