# Lab 2 - Fine-Tuning a Large Language Model

## Overview

In this lab we fine-tuned the open-source base model `unsloth/Llama-3.2-3B-Instruct` on Maxime Labonne's FineTome-100k instruction dataset using LoRA (Low-Rank Adaptation) with QLoRA quantization via Unsloth and `trl.SFTTrainer`. The training used the following hyperparameters:

| Hyperparameter | Value |
|----------------|-------|
| `per_device_train_batch_size` | 2 |
| `gradient_accumulation_steps` | 4 |
| `num_train_epochs` | 1 |
| `learning_rate` | 2e-4 |
| `warmup_steps` | 5 |
| `weight_decay` | 0.01 |
| `optimizer` | adamw_8bit |

---

## Evaluation Methodology

We evaluated both the base model and the fine-tuned model on **100 held-out examples** from a 15% test split of FineTome-100k. For each example we used the conversation history as input and treated the last assistant turn as the reference answer.

### Metrics Used

We used **ROUGE scores** (Recall-Oriented Understudy for Gisting Evaluation) to measure the overlap between model-generated responses and reference answers:

- **ROUGE-1**: Measures unigram (single word) overlap between the generated text and reference. Higher scores indicate better word-level similarity.
- **ROUGE-2**: Measures bigram (two consecutive words) overlap. This captures phrase-level similarity and is more sensitive to word order.
- **ROUGE-L**: Measures the longest common subsequence between generated and reference text. This captures sentence-level structure and fluency.

All scores use the F-measure (harmonic mean of precision and recall) and range from 0 to 1, where higher is better.

### Generation Settings

We used **deterministic generation** (greedy decoding with `do_sample=False`) for both models to ensure reproducible and comparable results. Each response was limited to 256 new tokens.

---

## Results

The fine-tuned model clearly outperformed the base model across all ROUGE metrics:

| Metric | Base Model | Fine-Tuned Model | Improvement |
|--------|------------|------------------|-------------|
| ROUGE-1 | 0.4732 | 0.5323 | **+12.5%** |
| ROUGE-2 | 0.2255 | 0.2849 | **+26.4%** |
| ROUGE-L | 0.2856 | 0.3521 | **+23.3%** |

These results demonstrate that even a single epoch of LoRA fine-tuning on FineTome-100k produces substantial improvements in response quality. The largest gain was in ROUGE-2 (+26.4%), indicating that the fine-tuned model better captures phrase-level patterns from the training data.

### Response Length Analysis

| Metric | Base Model | Fine-Tuned Model | Reference |
|--------|------------|------------------|-----------|
| Mean length (words) | 165.0 | 155.7 | 216.3 |
| Median length (words) | 176.0 | 165.5 | 199.0 |

The fine-tuned model produces slightly more concise responses while achieving higher ROUGE scores, suggesting improved information density.

---

## Improving Model Performance

### (a) Model-Centric Approach

A model-centric approach keeps the data fixed and focuses on changing the model architecture, training configuration, or optimization procedure. Below are concrete strategies for further improvement:

#### Hyperparameter Tuning

- **Learning rate**: Sweep over values such as `1e-4`, `2e-4`, `5e-4` to find the optimal learning rate. Our current setting of `2e-4` is a reasonable default but may not be optimal.
- **Learning rate schedule**: Experiment with cosine decay or cosine with warm restarts instead of constant learning rate.
- **Training epochs**: Train for 2–3 epochs with early stopping based on validation loss to potentially improve convergence.
- **Batch size**: Increase effective batch size (via `gradient_accumulation_steps`) within memory constraints for more stable gradients.
- **Warmup steps**: Adjust warmup duration (e.g., 10–100 steps) to improve training stability.
- **Weight decay**: Test different regularization strengths (e.g., `0.001`, `0.01`, `0.1`) to control overfitting.

#### LoRA Configuration

- **Rank (r)**: Increase LoRA rank (e.g., from 16 to 32 or 64) to allow more expressive adapter updates, at the cost of increased memory.
- **Alpha scaling**: Adjust the LoRA alpha parameter to control the magnitude of adapter contributions.
- **Target modules**: Experiment with applying LoRA to different layer types (attention only, MLP layers, or both) and different layer ranges.

#### Model Architecture

- **Base model selection**: Compare different foundation models such as `Llama-3.2-1B-Instruct` (faster inference) or `Llama-3.1-8B-Instruct` (potentially higher quality but slower).
- **Quantization**: Compare 4-bit (QLoRA) vs 8-bit quantization to understand the quality-speed tradeoff.

#### Training Procedure

- **Gradient clipping**: Add gradient clipping to prevent exploding gradients and improve training stability.
- **Mixed precision**: Ensure optimal use of mixed precision training for faster iteration.

### (b) Data-Centric Approach

A data-centric approach keeps the model and training loop mostly fixed and focuses on improving or extending the training data. Below are concrete strategies:

#### Data Quality Improvements

- **Filter low-quality examples**: Remove very short, unclear, or noisy instruction-response pairs from FineTome-100k to increase average signal per batch.
- **Deduplicate**: Remove near-duplicate examples that may cause the model to overfit to specific patterns.
- **Balance task types**: If the target application focuses on specific capabilities (e.g., reasoning, coding, explanation), up-sample those categories and down-sample less relevant ones.

#### Additional Data Sources

Augment FineTome-100k with other high-quality open-source instruction datasets:

| Dataset | Focus Area | Potential Benefit |
|---------|------------|-------------------|
| OpenAssistant Conversations | Multi-turn dialogue | Improved conversational ability |
| GSM8K / MetaMath | Math reasoning | Better mathematical problem-solving |
| CodeAlpaca / Code-Feedback | Programming tasks | Improved code generation |
| FLAN Collection | Diverse NLP tasks | Broader task coverage |
| UltraChat | Long-form dialogue | Better handling of extended conversations |

#### Domain-Specific Fine-Tuning

- **Curriculum learning**: Start training on general instructions, then gradually shift to more specialized or difficult examples.
- **Task-specific adapters**: Train separate LoRA adapters for different domains (math, code, creative writing) and select the appropriate adapter at inference time.

#### Data Alignment

- **Match UI format**: If the final application expects specific output formats (e.g., step-by-step reasoning, JSON responses), construct or filter training examples that demonstrate these formats.
- **User feedback loop**: In production, log anonymized user interactions (if permitted) to create a fine-tuning set that reflects real usage patterns.

---

## Conclusion

Our fine-tuning pipeline demonstrates measurable improvements over the base model, with ROUGE scores increasing by 12–26% on a held-out test set. The model-centric and data-centric strategies outlined above provide clear directions for further performance gains. The most promising next steps would be:

1. **Hyperparameter sweep** on learning rate and number of epochs
2. **Increase LoRA rank** to allow more expressive updates
3. **Mix in domain-specific datasets** (e.g., math reasoning or code) to improve performance on specialized tasks