ydvdhrj/Pirate-LLM

Pirate-Language LLM ⚓️

Quantization & Fine-Tuning TinyLlama on a 50k+ Instruction Dataset

📖 Project Overview

This project demonstrates fine-tuning and quantization of the TinyLlama model on a custom dataset of 50k+ instruction-response pairs. The objective was to train a model capable of converting standard English queries into pirate-style responses.

The project highlights how resource-constrained environments (like 4GB GPUs) can still be used effectively to fine-tune large language models using parameter-efficient fine-tuning (LoRA) and quantization (BitsAndBytes).

✨ Key Features

Dataset: 50k+ instruction-response pairs formatted into JSONL for supervised fine-tuning.

Model: TinyLlama fine-tuned with LoRA adapters.

Quantization: 4-bit quantization using BitsAndBytes for reduced memory usage.

Training:

Locally on NVIDIA GTX 1650 (4GB VRAM).

Scaled up to Google Colab T4 (16GB VRAM) for large dataset training.

Optimizations:

Gradient checkpointing

Cosine learning rate scheduler

Mixed precision (bfloat16)

Checkpoint Management: Saved models on Google Drive and Hugging Face Hub to mitigate Colab’s 12-hour GPU session limits.

🚀 Tech Stack

Languages/Frameworks: Python, PyTorch

Libraries: Hugging Face Transformers, TRL, PEFT, LoRA, BitsAndBytes

Compute: CUDA, cuDNN, Google Colab, Local GPU (GTX 1650)

📂 Repository Structure Pirate-Language-LLM/ │── data/ # Dataset files (JSONL format) │── notebooks/ # Jupyter notebooks for experiments │── scripts/ # Training and evaluation scripts │── checkpoints/ # Saved model checkpoints │── README.md # Project documentation │── requirements.txt # Dependencies

🔧 Setup & Installation

Clone the repository:

git clone https://github.com/yourusername/Pirate-Language-LLM.git cd Pirate-Language-LLM

Install dependencies:

pip install -r requirements.txt

Download dataset (or use your own):

from datasets import load_dataset dataset = load_dataset("json", data_files="pirate_data_large.jsonl")

Fine-tune the model:

python scripts/train.py

📊 Results

Successfully fine-tuned TinyLlama to translate English into pirate-style text.

Achieved stable training on both local GPU (4GB) and Colab T4 (16GB).

Demonstrated practical quantization + LoRA fine-tuning workflow on limited compute.

🔮 Future Work

Extend the approach to Indian languages, building a voice-to-text and text-to-voice model with Indian accent support.

Pre-quantize larger multilingual models and fine-tune on diverse datasets.

Enable real-time conversational systems with efficient deployment on constrained hardware.

📎 Model & Resources

Hugging Face Model Link (replace with your link)

🤝 Contribution

Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.

📜 License

This project is licensed under the MIT License.