| # Nagargpt-Finetuned | |
| This repository contains LoRA adapters for the `CreitinGameplays/bloom-3b-conversational` model, fine-tuned on a Nepali municipality QA and conversational dataset. | |
| ## Usage | |
| To use this model, load the base model and apply the LoRA adapters: | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| from peft import PeftModel | |
| base_model = "CreitinGameplays/bloom-3b-conversational" | |
| adapter_path = "your-username/nagargpt-finetuned" | |
| tokenizer = AutoTokenizer.from_pretrained(adapter_path) | |
| model = AutoModelForCausalLM.from_pretrained(base_model, load_in_4bit=True) | |
| model = PeftModel.from_pretrained(model, adapter_path) | |
| ``` | |
| ## Training Details | |
| - Base Model: `CreitinGameplays/bloom-3b-conversational` | |
| - Dataset: Nepali municipality QA and conversational data (`yam3333/main_plus_additional_v2`) | |
| - Fine-Tuning: QLoRA with 4-bit quantization, LoRA rank=8, alpha=16, dropout=0.05 | |
| - Target Modules: `query_key_value` | |
| - Epochs: 3 | |
| - Batch Size: 1 (with gradient accumulation steps=8) | |
| - Learning Rate: 2e-5 | |