File size: 1,042 Bytes
10c2141 c2a9c79 10c2141 c2a9c79 10c2141 c2a9c79 10c2141 c2a9c79 10c2141 c2a9c79 10c2141 c2a9c79 10c2141 c2a9c79 10c2141 c2a9c79 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# Nagargpt-Finetuned
This repository contains LoRA adapters for the `CreitinGameplays/bloom-3b-conversational` model, fine-tuned on a Nepali municipality QA and conversational dataset.
## Usage
To use this model, load the base model and apply the LoRA adapters:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = "CreitinGameplays/bloom-3b-conversational"
adapter_path = "your-username/nagargpt-finetuned"
tokenizer = AutoTokenizer.from_pretrained(adapter_path)
model = AutoModelForCausalLM.from_pretrained(base_model, load_in_4bit=True)
model = PeftModel.from_pretrained(model, adapter_path)
```
## Training Details
- Base Model: `CreitinGameplays/bloom-3b-conversational`
- Dataset: Nepali municipality QA and conversational data (`yam3333/main_plus_additional_v2`)
- Fine-Tuning: QLoRA with 4-bit quantization, LoRA rank=8, alpha=16, dropout=0.05
- Target Modules: `query_key_value`
- Epochs: 3
- Batch Size: 1 (with gradient accumulation steps=8)
- Learning Rate: 2e-5
|