File size: 1,839 Bytes
40540a4 6242282 40540a4 6242282 40540a4 6242282 0f04eca 8ab3109 6242282 40540a4 6242282 40540a4 6242282 40540a4 6242282 40540a4 6242282 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: apache-2.0
base_model: google/gemma-2b
tags:
- gemma
- lora
- qlora
- instruction-tuning
- unsloth
- transformers
- text-generation
library_name: transformers
---
- **Developed by:** Tushar Kamthe
- **License:** apache-2.0
- **Finetuned from model :** unsloth/gemma-2-2b-it-bnb-4bit
-
# Gemma-2B QLoRA Fine-tuned Model
## Model Description
This model is a fine-tuned version of **Google's Gemma-2B** using the **QLoRA (Quantized Low-Rank Adaptation)** technique. The model was trained using **parameter-efficient fine-tuning (PEFT)**, where only LoRA adapters were trained while keeping the base model weights frozen.
The model is designed for **instruction-following text generation tasks**.
Fine-tuning was performed using:
- QLoRA (4-bit quantization)
- LoRA adapters
- HuggingFace Transformers
- PEFT
- Unsloth for faster training
---
## Base Model
Base model used for training:
google/gemma-2b
---
## Training Details
### Training Method
The model was trained using **QLoRA**, which enables efficient training of large language models by:
- Loading the base model in **4-bit quantized format**
- Training **LoRA adapter weights only**
- Keeping base model weights frozen
This significantly reduces GPU memory requirements.
---
### Training Configuration
| Parameter | Value |
|--------|--------|
| Method | QLoRA |
| Quantization | 4-bit (NF4) |
| LoRA Rank (r) | 16 |
| LoRA Alpha | 64 |
| LoRA Dropout | 0.05 |
| Optimizer | AdamW |
| Precision | bfloat16 |
| Framework | HuggingFace Transformers |
---
### Hardware
Training was performed on:
- GPU: NVIDIA GPU (Colab / Local GPU)
- Framework: PyTorch
- Libraries:
- transformers
- peft
- datasets
- unsloth
---
## Dataset
The model was fine-tuned on a **custom instruction dataset** containing prompt-response pairs.
Dataset format:
|