nanoTharoor
A Qwen2.5-3B-Instruct model fine-tuned on the rhetorical style of Shashi Tharoor — Indian politician, author, and former UN Under-Secretary-General, known for his elevated vocabulary, historical analogies, and aphoristic closures.
LoRA implemented from scratch in pure PyTorch — no PEFT during training. The project also includes Fisher Information Matrix analysis to measure whether LoRA implicitly protects the base model from catastrophic forgetting.
Key Finding
Mean Fisher-LoRA Pearson r = 0.002 — essentially zero. The LoRA updates are orthogonal to the parameters the base model relies on for general language capability. LoRA provides implicit forgetting protection by construction for persona fine-tuning (Outcome A).
Training Results
| Metric | Value |
|---|---|
| Train loss (epoch 4) | 2.2096 |
| Eval loss (epoch 4) | 2.3917 |
| Mean effective rank | 11.872 / 16 |
| Fisher-LoRA Pearson r | 0.002 |
| Parameters trained | 7.37M / 3.06B (0.24%) |
| Epochs | 4 |
| Dataset | 269 Tharoor Q&A pairs |
Architecture
- Base: Qwen/Qwen2.5-3B-Instruct
- LoRA rank r=16, alpha=32, dropout=0.05
- Target modules: q_proj, k_proj, v_proj, o_proj (all 36 layers → 144 adapters)
- LR: 2e-4 with cosine warmup
Usage
# Note: adapter.pt uses our custom format — load with lora.py from the repo
# Standard PEFT-compatible release coming soon
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-3B-Instruct",
torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
# Load adapter: see lora.py in this repo
Analysis plots
See the analysis/ folder for SVD evolution, effective rank heatmap, Fisher-LoRA overlap, and loss curves.