prajwalg1997/phi4mini-medical-lora2

Phi-4-mini Instruct — Medical LoRA (ctx=1024)

Base model: microsoft/Phi-4-mini-instruct
Adapter type: LoRA (QLoRA, 4-bit training)
Domain: Medical instruction + reasoning
Created: 2025-08-21 06:23:42

Training Summary

• Context length: 1024
• Epochs: 2
• Learning rate: 0.00021726991983621354
• LoRA r / alpha / dropout: 32 / 84 / 0.05
• Grad accumulation: 16
• Eval loss (100 examples, ctx=1024): 1.4389

Dataset

• FreedomIntelligence/medical-o1-reasoning-SFT (config: en)
• ~500 examples (80/20 split) used for quick HPO & final demo run

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

base_id = "microsoft/Phi-4-mini-instruct"
adapter_id = "prajwalg1997/phi4mini-medical-lora2"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16)

base = AutoModelForCausalLM.from_pretrained(
    base_id, device_map="auto", trust_remote_code=True,
    quantization_config=bnb, attn_implementation="sdpa"
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

Notes

• Trained with transformers==4.49.0, accelerate==1.3.0, bitsandbytes==0.47.0
• Generation tip: temperature=0.2, do_sample=False, max_new_tokens≈256–512