vlachner/mistral-medquad-r32

Mistral 7B MedQuAD (LoRA r=32)

Training configuration

• Epochs: 1
• Batch size: 1
• Gradient accumulation: 4
• Learning rate: 2e-4
• Optimizer: paged_adamw_8bit
• Quantization: 4-bit (bitsandbytes)

Training Summary

Step	Training Loss	Validation Loss	Mean Token Accuracy
20	1.1077	1.0624	0.7329
100	0.7190	0.8991	0.7721
200	0.9566	0.8750	0.7758
300	0.9045	0.8575	0.7783
400	1.0345	0.8418	0.7806
500	0.9336	0.8355	0.7817

Final metrics

• Final Training Loss: 0.90
• Final Validation Loss: 0.84
• Final Mean Token Accuracy: 0.78
• Epochs: 1
• Total Steps: 500

Model fine-tuned on the MedQuAD dataset for medical QA using PEFT + QLoRA with rank = 32.