phi3-mini-yoda-v1

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on the dvgodoy/yoda_sentences dataset.

Model Description

• Base Model: microsoft/Phi-3-mini-4k-instruct
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with 4-bit quantization
• Task: Text Generation
• Language: English
• Dataset: dvgodoy/yoda_sentences

Training Configuration

• LoRA Rank: 4
• LoRA Alpha: 16
• Target Modules: qkv_proj, o_proj
• Quantization: 4-bit (NF4)
• Compute Type: bfloat16
• Learning Rate: 2e-4
• Batch Size: 1 (with gradient accumulation steps: 4)
• Optimizer: paged_adamw_8bit
• Max Length: 64 tokens
• Epochs: 1

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "gilbaes/phi3-mini-yoda-v1"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

prompt = "Translate to Yoda speak: I am learning to use the Force."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

This model was fine-tuned using:

• Framework: Hugging Face Transformers + TRL
• PEFT Method: LoRA (Low-Rank Adaptation)
• Quantization: BitsAndBytes 4-bit
• Gradient Checkpointing: Enabled
• Flash Attention 2: Enabled for efficiency

Limitations and Bias

This model inherits the limitations and biases from the base Phi-3 model and the training dataset. It's designed for educational purposes and may not be suitable for production use without further evaluation.

Citation

@misc{phi3-mini-yoda-v1},
  author = {gilbaes},
  title = {phi3-mini-yoda-v1},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/gilbaes/phi3-mini-yoda-v1}
}