File size: 3,080 Bytes
5c53a61 ff1fb53 b51f9e1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
license: mit
language:
- en
base_model:
- TinyLlama/TinyLlama-1.1B-Chat-v1.0
---
Pirate-Language LLM ⚓️
Quantization & Fine-Tuning TinyLlama on a 50k+ Instruction Dataset
📖 Project Overview
This project demonstrates fine-tuning and quantization of the TinyLlama model on a custom dataset of 50k+ instruction-response pairs. The objective was to train a model capable of converting standard English queries into pirate-style responses.
The project highlights how resource-constrained environments (like 4GB GPUs) can still be used effectively to fine-tune large language models using parameter-efficient fine-tuning (LoRA) and quantization (BitsAndBytes).
✨ Key Features
Dataset: 50k+ instruction-response pairs formatted into JSONL for supervised fine-tuning.
Model: TinyLlama fine-tuned with LoRA adapters.
Quantization: 4-bit quantization using BitsAndBytes for reduced memory usage.
Training:
Locally on NVIDIA GTX 1650 (4GB VRAM).
Scaled up to Google Colab T4 (16GB VRAM) for large dataset training.
Optimizations:
Gradient checkpointing
Cosine learning rate scheduler
Mixed precision (bfloat16)
Checkpoint Management: Saved models on Google Drive and Hugging Face Hub to mitigate Colab’s 12-hour GPU session limits.
🚀 Tech Stack
Languages/Frameworks: Python, PyTorch
Libraries: Hugging Face Transformers, TRL, PEFT, LoRA, BitsAndBytes
Compute: CUDA, cuDNN, Google Colab, Local GPU (GTX 1650)
📂 Repository Structure
Pirate-Language-LLM/
│── data/ # Dataset files (JSONL format)
│── notebooks/ # Jupyter notebooks for experiments
│── scripts/ # Training and evaluation scripts
│── checkpoints/ # Saved model checkpoints
│── README.md # Project documentation
│── requirements.txt # Dependencies
🔧 Setup & Installation
Clone the repository:
git clone https://github.com/yourusername/Pirate-Language-LLM.git
cd Pirate-Language-LLM
Install dependencies:
pip install -r requirements.txt
Download dataset (or use your own):
from datasets import load_dataset
dataset = load_dataset("json", data_files="pirate_data_large.jsonl")
Fine-tune the model:
python scripts/train.py
📊 Results
Successfully fine-tuned TinyLlama to translate English into pirate-style text.
Achieved stable training on both local GPU (4GB) and Colab T4 (16GB).
Demonstrated practical quantization + LoRA fine-tuning workflow on limited compute.
🔮 Future Work
Extend the approach to Indian languages, building a voice-to-text and text-to-voice model with Indian accent support.
Pre-quantize larger multilingual models and fine-tune on diverse datasets.
Enable real-time conversational systems with efficient deployment on constrained hardware.
📎 Model & Resources
Hugging Face Model Link
(replace with your link)
🤝 Contribution
Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.
📜 License
This project is licensed under the MIT License.
|