--- base_model: microsoft/Phi-3-mini-4k-instruct library_name: peft license: mit tags: - trl - sft - generated_from_trainer model-index: - name: phi-3-mini-LoRA results: [] --- # phi-3-mini-LoRA This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.1247 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.001 - train_batch_size: 1 - eval_batch_size: 1 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0.1 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.6204 | 0.0301 | 100 | 0.3928 | | 0.3832 | 0.0602 | 200 | 0.2888 | | 0.3208 | 0.0902 | 300 | 0.2991 | | 0.3085 | 0.1203 | 400 | 0.2653 | | 0.2631 | 0.1504 | 500 | 0.2453 | | 0.2549 | 0.1805 | 600 | 0.2396 | | 0.2831 | 0.2105 | 700 | 0.2547 | | 0.2796 | 0.2406 | 800 | 0.3548 | | 0.3194 | 0.2707 | 900 | 0.2517 | | 0.2783 | 0.3008 | 1000 | 0.2480 | | 0.248 | 0.3308 | 1100 | 0.2452 | | 0.2804 | 0.3609 | 1200 | 0.2439 | | 0.2702 | 0.3910 | 1300 | 0.2413 | | 0.2561 | 0.4211 | 1400 | 0.2324 | | 0.2796 | 0.4511 | 1500 | 0.2468 | | 0.2971 | 0.4812 | 1600 | 0.2308 | | 0.2683 | 0.5113 | 1700 | 0.2255 | | 0.2568 | 0.5414 | 1800 | 0.2350 | | 0.2527 | 0.5714 | 1900 | 0.2240 | | 0.2215 | 0.6015 | 2000 | 0.2234 | | 0.2476 | 0.6316 | 2100 | 0.2085 | | 0.2294 | 0.6617 | 2200 | 0.2134 | | 0.2073 | 0.6917 | 2300 | 0.2150 | | 0.2113 | 0.7218 | 2400 | 0.2047 | | 0.2269 | 0.7519 | 2500 | 0.2045 | | 0.207 | 0.7820 | 2600 | 0.2012 | | 0.236 | 0.8120 | 2700 | 0.1997 | | 0.2292 | 0.8421 | 2800 | 0.1896 | | 0.2189 | 0.8722 | 2900 | 0.1901 | | 0.1981 | 0.9023 | 3000 | 0.1864 | | 0.1933 | 0.9323 | 3100 | 0.1913 | | 0.2002 | 0.9624 | 3200 | 0.1833 | | 0.2001 | 0.9925 | 3300 | 0.1852 | | 0.1653 | 1.0226 | 3400 | 0.1828 | | 0.1758 | 1.0526 | 3500 | 0.1815 | | 0.186 | 1.0827 | 3600 | 0.1855 | | 0.1816 | 1.1128 | 3700 | 0.1877 | | 0.1658 | 1.1429 | 3800 | 0.1785 | | 0.1752 | 1.1729 | 3900 | 0.1757 | | 0.1784 | 1.2030 | 4000 | 0.1751 | | 0.1582 | 1.2331 | 4100 | 0.1713 | | 0.1706 | 1.2632 | 4200 | 0.1726 | | 0.162 | 1.2932 | 4300 | 0.1712 | | 0.1749 | 1.3233 | 4400 | 0.1712 | | 0.177 | 1.3534 | 4500 | 0.1710 | | 0.1767 | 1.3835 | 4600 | 0.1705 | | 0.1629 | 1.4135 | 4700 | 0.1692 | | 0.147 | 1.4436 | 4800 | 0.1643 | | 0.1675 | 1.4737 | 4900 | 0.1639 | | 0.1635 | 1.5038 | 5000 | 0.1627 | | 0.1532 | 1.5338 | 5100 | 0.1625 | | 0.149 | 1.5639 | 5200 | 0.1646 | | 0.1543 | 1.5940 | 5300 | 0.1612 | | 0.159 | 1.6241 | 5400 | 0.1585 | | 0.1498 | 1.6541 | 5500 | 0.1577 | | 0.149 | 1.6842 | 5600 | 0.1569 | | 0.1621 | 1.7143 | 5700 | 0.1543 | | 0.1608 | 1.7444 | 5800 | 0.1531 | | 0.1392 | 1.7744 | 5900 | 0.1526 | | 0.157 | 1.8045 | 6000 | 0.1516 | | 0.1521 | 1.8346 | 6100 | 0.1495 | | 0.1428 | 1.8647 | 6200 | 0.1490 | | 0.1353 | 1.8947 | 6300 | 0.1471 | | 0.1494 | 1.9248 | 6400 | 0.1456 | | 0.1346 | 1.9549 | 6500 | 0.1451 | | 0.1434 | 1.9850 | 6600 | 0.1447 | | 0.1194 | 2.0150 | 6700 | 0.1463 | | 0.0978 | 2.0451 | 6800 | 0.1458 | | 0.1034 | 2.0752 | 6900 | 0.1471 | | 0.0999 | 2.1053 | 7000 | 0.1433 | | 0.1021 | 2.1353 | 7100 | 0.1407 | | 0.1044 | 2.1654 | 7200 | 0.1417 | | 0.1096 | 2.1955 | 7300 | 0.1415 | | 0.0868 | 2.2256 | 7400 | 0.1402 | | 0.0916 | 2.2556 | 7500 | 0.1396 | | 0.1036 | 2.2857 | 7600 | 0.1406 | | 0.106 | 2.3158 | 7700 | 0.1377 | | 0.1051 | 2.3459 | 7800 | 0.1365 | | 0.1089 | 2.3759 | 7900 | 0.1358 | | 0.1018 | 2.4060 | 8000 | 0.1354 | | 0.0981 | 2.4361 | 8100 | 0.1354 | | 0.0987 | 2.4662 | 8200 | 0.1353 | | 0.0977 | 2.4962 | 8300 | 0.1349 | | 0.099 | 2.5263 | 8400 | 0.1329 | | 0.0947 | 2.5564 | 8500 | 0.1330 | | 0.0913 | 2.5865 | 8600 | 0.1326 | | 0.0915 | 2.6165 | 8700 | 0.1310 | | 0.0882 | 2.6466 | 8800 | 0.1330 | | 0.0956 | 2.6767 | 8900 | 0.1302 | | 0.0934 | 2.7068 | 9000 | 0.1291 | | 0.0861 | 2.7368 | 9100 | 0.1277 | | 0.0911 | 2.7669 | 9200 | 0.1264 | | 0.0884 | 2.7970 | 9300 | 0.1265 | | 0.0928 | 2.8271 | 9400 | 0.1270 | | 0.0876 | 2.8571 | 9500 | 0.1274 | | 0.0879 | 2.8872 | 9600 | 0.1258 | | 0.0895 | 2.9173 | 9700 | 0.1255 | | 0.0833 | 2.9474 | 9800 | 0.1250 | | 0.0871 | 2.9774 | 9900 | 0.1247 | ### Framework versions - PEFT 0.12.0 - Transformers 4.43.3 - Pytorch 2.4.0+cu118 - Datasets 2.20.0 - Tokenizers 0.19.1