yam3333/Trial_version

Nagargpt-Finetuned

This repository contains LoRA adapters for the CreitinGameplays/bloom-3b-conversational model, fine-tuned on a Nepali municipality QA and conversational dataset.

Usage

To use this model, load the base model and apply the LoRA adapters:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "CreitinGameplays/bloom-3b-conversational"
adapter_path = "your-username/nagargpt-finetuned"

tokenizer = AutoTokenizer.from_pretrained(adapter_path)
model = AutoModelForCausalLM.from_pretrained(base_model, load_in_4bit=True)
model = PeftModel.from_pretrained(model, adapter_path)

Training Details

• Base Model: CreitinGameplays/bloom-3b-conversational
• Dataset: Nepali municipality QA and conversational data (yam3333/main_plus_additional_v2)
• Fine-Tuning: QLoRA with 4-bit quantization, LoRA rank=8, alpha=16, dropout=0.05
• Target Modules: query_key_value
• Epochs: 3
• Batch Size: 1 (with gradient accumulation steps=8)
• Learning Rate: 2e-5