distractor-lora-clean-only-llama1b

LoRA adapter weights (not merged) for distractor robustness training.

LoRA + Clean only (ans-only NLL)

Training Details

• Base model: meta-llama/Llama-3.2-1B
• Method: LoRA rank-16 adapters
• Training data: HotPotQA + MuSiQue contrastive pairs (19K train samples)
• Training steps: 594 (1 epoch), effective batch size 32

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "sandywong/distractor-lora-clean-only-llama1b")

# Or merge into base for inference:
model = model.merge_and_unload()

Part of

Distractor Robustness Training for Multi-hop QA project.