|
|
--- |
|
|
library_name: peft |
|
|
tags: |
|
|
- axolotl |
|
|
- lora |
|
|
- transformers |
|
|
- gemma |
|
|
pipeline_tag: text-generation |
|
|
base_model: google/gemma-3-27b-it |
|
|
--- |
|
|
|
|
|
# EmoN1 |
|
|
|
|
|
Fine-tuned from [google/gemma-3-27b-it](https://huggingface.co/google/gemma-3-27b-it) using QLoRA. |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Base Model:** google/gemma-3-27b-it |
|
|
- **Method:** QLoRA (4-bit quantization + LoRA) |
|
|
- **LoRA Rank:** 32 |
|
|
- **LoRA Alpha:** 64 |
|
|
- **Sequence Length:** 8192 |
|
|
- **Epochs:** 3 |
|
|
- **Learning Rate:** 2e-4 |
|
|
|
|
|
### Training Results |
|
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|
|:-------------:|:-----:|:----:|:---------------:| |
|
|
| 0.9058 | 1.0 | 63 | 0.8959 | |
|
|
| 0.8279 | 2.0 | 126 | 0.8607 | |
|
|
|
|
|
### Framework Versions |
|
|
|
|
|
- PEFT 0.17.1 |
|
|
- Transformers 4.55.4 |
|
|
- Pytorch 2.7.1+cu126 |
|
|
|
