|
|
--- |
|
|
license: mit |
|
|
base_model: microsoft/Phi-3-mini-4k-instruct |
|
|
tags: |
|
|
- phi3 |
|
|
- lora |
|
|
- alpaca |
|
|
- instruction-tuning |
|
|
- fine-tuned |
|
|
datasets: |
|
|
- tatsu-lab/alpaca |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Phi-3 Mini 4K Instruct - Alpaca LoRA Fine-tuned |
|
|
|
|
|
This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) using LoRA (Low-Rank Adaptation) on the [Alpaca dataset](https://huggingface.co/datasets/tatsu-lab/alpaca). |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base Model**: microsoft/Phi-3-mini-4k-instruct |
|
|
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation) |
|
|
- **Dataset**: tatsu-lab/alpaca (52,002 instruction-following examples) |
|
|
- **Training Duration**: ~1.24 hours |
|
|
- **Final Training Loss**: 1.0445 |
|
|
- **Average Training Loss**: 1.0311 |
|
|
|
|
|
## Training Configuration |
|
|
|
|
|
- **LoRA Rank**: 16 |
|
|
- **LoRA Alpha**: 32 |
|
|
- **LoRA Dropout**: 0.05 |
|
|
- **Target Modules**: qkv_proj, o_proj, gate_proj, up_proj, down_proj |
|
|
- **Learning Rate**: 1e-5 |
|
|
- **Batch Size**: 2 (with gradient accumulation steps: 8) |
|
|
- **Epochs**: 1 |
|
|
- **Precision**: bfloat16 |
|
|
- **Gradient Checkpointing**: Enabled |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
from peft import PeftModel |
|
|
|
|
|
# Load tokenizer |
|
|
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct", trust_remote_code=True) |
|
|
|
|
|
# Load base model |
|
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
|
"microsoft/Phi-3-mini-4k-instruct", |
|
|
torch_dtype=torch.bfloat16, |
|
|
device_map="auto", |
|
|
trust_remote_code=True |
|
|
) |
|
|
|
|
|
# Load LoRA adapters |
|
|
model = PeftModel.from_pretrained(base_model, "johnlam90/phi3-mini-4k-instruct-alpaca-lora") |
|
|
model.eval() |
|
|
|
|
|
# Format prompt |
|
|
prompt = "Give three tips for staying healthy." |
|
|
formatted_prompt = f'''### Instruction: |
|
|
{prompt} |
|
|
|
|
|
### Response: |
|
|
''' |
|
|
|
|
|
# Generate |
|
|
inputs = tokenizer(formatted_prompt, return_tensors="pt") |
|
|
with torch.no_grad(): |
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=200, |
|
|
do_sample=False, |
|
|
eos_token_id=tokenizer.eos_token_id, |
|
|
pad_token_id=tokenizer.eos_token_id |
|
|
) |
|
|
|
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(response.split("### Response:")[1].strip()) |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
The model has been tested with comprehensive safety measures including: |
|
|
- ✅ NaN clamp protection for stable generation |
|
|
- ✅ Proper bfloat16 precision handling |
|
|
- ✅ Consistent and coherent responses across multiple test prompts |
|
|
- ✅ No numerical instabilities during training or inference |
|
|
|
|
|
## Training Details |
|
|
|
|
|
This model was fine-tuned with careful attention to: |
|
|
1. **Data Formatting**: Proper Alpaca instruction/input/output structure |
|
|
2. **Numerical Stability**: Using bfloat16 precision and conservative hyperparameters |
|
|
3. **Memory Efficiency**: Gradient checkpointing and optimized batch sizes |
|
|
4. **Safety Measures**: NaN protection and proper token handling |
|
|
|
|
|
## License |
|
|
|
|
|
This model is released under the MIT license, following the base model's licensing terms. |
|
|
|
