whats2000/lora-llama-pathoqa-checkpoints

LoRA-Llama PathoQA — all trained adapters (ablation study)

LoRA / DoRA adapters for meta-llama/Llama-3.2-1B-Instruct fine-tuned on PathoQA (4-option medical pathology MCQA). This repo holds every adapter from a 10-experiment ablation study (base model fixed at 1B; the study is about method). Full lab journal and code: see the course submission package (report.ipynb + experiments/NOTES.md).

Metric = Kaggle hw-1-question-answering test accuracy (public == private).

Best pipeline — 0.7988 (5-member option-likelihood ensemble)

Uniform-average the per-option probabilities of these 5 adapters (all rank 256, effective-batch 192):

role	path
zero-shot r256	experiments/exp06_zeroshot_recipe/e06c/saved_models
zero-shot r192	experiments/exp07_eff192_ensemble/e07a/saved_models
few-shot 2-shot	experiments/exp08_fewshot_diversity/e08b/saved_models
few-shot 4-shot	experiments/exp07_eff192_ensemble/e07b/saved_models
few-shot 8-shot	experiments/exp08_fewshot_diversity/e08a/saved_models

Progression: baseline 0.7700 → rank=256 0.7766 → +ensemble 0.7811 → +few-shot 0.7888 → @eff192 0.7922 → +multi-shot 0.7988.

Load an adapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from huggingface_hub import snapshot_download

base_id = "meta-llama/Llama-3.2-1B-Instruct"
local = snapshot_download("whats2000/lora-llama-pathoqa-checkpoints",
                          allow_patterns="experiments/exp06_zeroshot_recipe/e06c/saved_models/*")
adapter = f"{local}/experiments/exp06_zeroshot_recipe/e06c/saved_models"

tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, adapter).eval()

All adapters (organised by experiment)

• exp01_lora_rank/e01a..e — LoRA rank sweep r∈{8,16,32,64,128} (zero-shot)
• exp02_rank_scaling/e02a..e — rank push r∈{128,192,256,384,512}
• exp04_strategy_diverse/e04a — few-shot @ eff768
• exp05_dora_variant/e05a..c — DoRA r∈{128,192,256}
• exp06_zeroshot_recipe/e06a..f — LR / effective-batch / loss recipe
• exp07_eff192_ensemble/e07a,b — zero/few-shot retrained @ eff-batch 192
• exp08_fewshot_diversity/e08a,b — few-shot 8-shot / 2-shot @ eff192
• exp09_more_fewshot/e09a..c — few-shot 1/3/6-shot @ eff192

Each dir is a standard PEFT adapter (adapter_config.json + adapter_model.safetensors) plus its training_history.json and loss/accuracy curves.