phi-3-mini-LoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.7121

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.94	0.0953	100	1.8918
1.8113	0.1907	200	1.7877
1.7578	0.2860	300	1.7701
1.756	0.3813	400	1.7637
1.7632	0.4766	500	1.7582
1.7477	0.5720	600	1.7542
1.7605	0.6673	700	1.7508
1.7312	0.7626	800	1.7482
1.7315	0.8580	900	1.7439
1.7148	0.9533	1000	1.7414
1.7263	1.0486	1100	1.7385
1.7184	1.1439	1200	1.7361
1.7187	1.2393	1300	1.7336
1.7231	1.3346	1400	1.7313
1.7433	1.4299	1500	1.7290
1.6962	1.5253	1600	1.7268
1.7136	1.6206	1700	1.7253
1.6969	1.7159	1800	1.7236
1.7028	1.8112	1900	1.7217
1.7066	1.9066	2000	1.7200
1.7123	2.0019	2100	1.7191
1.7005	2.0972	2200	1.7178
1.7052	2.1926	2300	1.7168
1.6946	2.2879	2400	1.7160
1.6728	2.3832	2500	1.7150
1.7033	2.4786	2600	1.7144
1.6893	2.5739	2700	1.7136
1.7206	2.6692	2800	1.7129
1.6747	2.7645	2900	1.7126
1.6981	2.8599	3000	1.7123
1.6928	2.9552	3100	1.7121

Framework versions

• PEFT 0.12.0
• Transformers 4.43.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1