|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- TinyLlama/TinyLlama-1.1B-Chat-v1.0 |
|
|
--- |
|
|
Pirate-Language LLM ⚓️ |
|
|
|
|
|
Quantization & Fine-Tuning TinyLlama on a 50k+ Instruction Dataset |
|
|
|
|
|
📖 Project Overview |
|
|
|
|
|
This project demonstrates fine-tuning and quantization of the TinyLlama model on a custom dataset of 50k+ instruction-response pairs. The objective was to train a model capable of converting standard English queries into pirate-style responses. |
|
|
|
|
|
The project highlights how resource-constrained environments (like 4GB GPUs) can still be used effectively to fine-tune large language models using parameter-efficient fine-tuning (LoRA) and quantization (BitsAndBytes). |
|
|
|
|
|
✨ Key Features |
|
|
|
|
|
Dataset: 50k+ instruction-response pairs formatted into JSONL for supervised fine-tuning. |
|
|
|
|
|
Model: TinyLlama fine-tuned with LoRA adapters. |
|
|
|
|
|
Quantization: 4-bit quantization using BitsAndBytes for reduced memory usage. |
|
|
|
|
|
Training: |
|
|
|
|
|
Locally on NVIDIA GTX 1650 (4GB VRAM). |
|
|
|
|
|
Scaled up to Google Colab T4 (16GB VRAM) for large dataset training. |
|
|
|
|
|
Optimizations: |
|
|
|
|
|
Gradient checkpointing |
|
|
|
|
|
Cosine learning rate scheduler |
|
|
|
|
|
Mixed precision (bfloat16) |
|
|
|
|
|
Checkpoint Management: Saved models on Google Drive and Hugging Face Hub to mitigate Colab’s 12-hour GPU session limits. |
|
|
|
|
|
🚀 Tech Stack |
|
|
|
|
|
Languages/Frameworks: Python, PyTorch |
|
|
|
|
|
Libraries: Hugging Face Transformers, TRL, PEFT, LoRA, BitsAndBytes |
|
|
|
|
|
Compute: CUDA, cuDNN, Google Colab, Local GPU (GTX 1650) |
|
|
|
|
|
📂 Repository Structure |
|
|
Pirate-Language-LLM/ |
|
|
│── data/ # Dataset files (JSONL format) |
|
|
│── notebooks/ # Jupyter notebooks for experiments |
|
|
│── scripts/ # Training and evaluation scripts |
|
|
│── checkpoints/ # Saved model checkpoints |
|
|
│── README.md # Project documentation |
|
|
│── requirements.txt # Dependencies |
|
|
|
|
|
🔧 Setup & Installation |
|
|
|
|
|
Clone the repository: |
|
|
|
|
|
git clone https://github.com/yourusername/Pirate-Language-LLM.git |
|
|
cd Pirate-Language-LLM |
|
|
|
|
|
|
|
|
Install dependencies: |
|
|
|
|
|
pip install -r requirements.txt |
|
|
|
|
|
|
|
|
Download dataset (or use your own): |
|
|
|
|
|
from datasets import load_dataset |
|
|
dataset = load_dataset("json", data_files="pirate_data_large.jsonl") |
|
|
|
|
|
|
|
|
Fine-tune the model: |
|
|
|
|
|
python scripts/train.py |
|
|
|
|
|
📊 Results |
|
|
|
|
|
Successfully fine-tuned TinyLlama to translate English into pirate-style text. |
|
|
|
|
|
Achieved stable training on both local GPU (4GB) and Colab T4 (16GB). |
|
|
|
|
|
Demonstrated practical quantization + LoRA fine-tuning workflow on limited compute. |
|
|
|
|
|
🔮 Future Work |
|
|
|
|
|
Extend the approach to Indian languages, building a voice-to-text and text-to-voice model with Indian accent support. |
|
|
|
|
|
Pre-quantize larger multilingual models and fine-tune on diverse datasets. |
|
|
|
|
|
Enable real-time conversational systems with efficient deployment on constrained hardware. |
|
|
|
|
|
📎 Model & Resources |
|
|
|
|
|
Hugging Face Model Link |
|
|
(replace with your link) |
|
|
|
|
|
🤝 Contribution |
|
|
|
|
|
Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change. |
|
|
|
|
|
📜 License |
|
|
|
|
|
This project is licensed under the MIT License. |
|
|
|
