File size: 2,214 Bytes
eb734c8 b71c602 eb734c8 b71c602 eb734c8 b71c602 eb734c8 b71c602 eb734c8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
base_model: google/gemma-3-4b-it-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- gemma3
- trl
license: apache-2.0
language:
- en
---
# 🧠 Fine-Tuning Gemma 3B on Healthcare Admin Tasks
This repository demonstrates how to fine-tune the instruction-tuned [`google/gemma-3-4b-it`](https://huggingface.co/google/gemma-3-4b-it) model on a custom dataset covering administrative tasks in the healthcare industry.
---
## 📌 Project Overview
We use the [Unsloth](https://github.com/unslothai/unsloth) framework to:
- Load and quantize the base Gemma model in **4-bit precision**.
- Apply **LoRA (Low-Rank Adaptation)** for efficient parameter tuning.
- Train the model using Hugging Face's `trl` library and `SFTTrainer`.
This setup significantly reduces memory footprint and training cost, making it suitable for training on consumer GPUs (e.g. Colab, T4, A100).
---
## 🩺 Dataset: Healthcare Admin
- **Source**: [`xgalaxy/healthcare_admin`](https://huggingface.co/datasets/xgalaxy/healthcare_admin)
- **Format**: ShareGPT-style JSON format with structured `user` and `assistant` roles
- **Coverage**:
- Appointment scheduling, cancellation, and rescheduling
- Edge cases involving follow-ups, missing info, and ambiguous requests
- Multi-turn conversations to emulate real-world interactions
---
## 🛠️ Key Components
### ✅ Model Setup
- `google/gemma-3-4b-it` loaded using Unsloth's `FastModel.from_pretrained()`
- 4-bit quantization enabled via `load_in_4bit=True`
- LoRA adapters injected for memory-efficient tuning
### ✅ Training
- Supervised fine-tuning with `SFTTrainer`
- Batch size simulated using `gradient_accumulation_steps`
- Linear learning rate scheduler with warmup
- Training capped at a fixed number of steps for fast iteration
---
## 🚀 Trained Model
The fine-tuned model is available on Hugging Face:
👉 [xgalaxy/gemma-3](https://huggingface.co/xgalaxy/gemma-3)
---
## 🔗 Resources
- 🔗 [Unsloth GitHub](https://github.com/unslothai/unsloth)
- 📘 [Gemma on Hugging Face](https://huggingface.co/google/gemma-3-4b-it)
- 🗃️ [Healthcare Admin Dataset](https://huggingface.co/datasets/xgalaxy/healthcare_admin)
---
|