|
|
--- |
|
|
library_name: peft |
|
|
license: llama3 |
|
|
base_model: meta-llama/Meta-Llama-3-8B-Instruct |
|
|
tags: |
|
|
- llama-factory |
|
|
- ia3 |
|
|
- generated_from_trainer |
|
|
model-index: |
|
|
- name: train_multirc_1745950263 |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
|
|
# train_multirc_1745950263 |
|
|
|
|
|
This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on the multirc dataset. |
|
|
It achieves the following results on the evaluation set: |
|
|
- Loss: 0.1402 |
|
|
- Num Input Tokens Seen: 75778784 |
|
|
|
|
|
## Model description |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Training and evaluation data |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
### Training hyperparameters |
|
|
|
|
|
The following hyperparameters were used during training: |
|
|
- learning_rate: 5e-05 |
|
|
- train_batch_size: 2 |
|
|
- eval_batch_size: 2 |
|
|
- seed: 123 |
|
|
- gradient_accumulation_steps: 2 |
|
|
- total_train_batch_size: 4 |
|
|
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments |
|
|
- lr_scheduler_type: cosine |
|
|
- training_steps: 40000 |
|
|
|
|
|
### Training results |
|
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |
|
|
|:-------------:|:------:|:-----:|:---------------:|:-----------------:| |
|
|
| 0.1201 | 0.0326 | 200 | 0.3065 | 378944 | |
|
|
| 0.275 | 0.0653 | 400 | 0.2538 | 758192 | |
|
|
| 0.3329 | 0.0979 | 600 | 0.2204 | 1141408 | |
|
|
| 0.3121 | 0.1305 | 800 | 0.2062 | 1518336 | |
|
|
| 0.0258 | 0.1631 | 1000 | 0.1979 | 1901264 | |
|
|
| 0.3025 | 0.1958 | 1200 | 0.1973 | 2279552 | |
|
|
| 0.2009 | 0.2284 | 1400 | 0.1862 | 2668256 | |
|
|
| 0.1066 | 0.2610 | 1600 | 0.1832 | 3047328 | |
|
|
| 0.0954 | 0.2937 | 1800 | 0.1835 | 3429984 | |
|
|
| 0.1731 | 0.3263 | 2000 | 0.1794 | 3814576 | |
|
|
| 0.192 | 0.3589 | 2200 | 0.1803 | 4190352 | |
|
|
| 0.1108 | 0.3915 | 2400 | 0.1727 | 4567440 | |
|
|
| 0.486 | 0.4242 | 2600 | 0.1717 | 4944384 | |
|
|
| 0.2059 | 0.4568 | 2800 | 0.1689 | 5325216 | |
|
|
| 0.1585 | 0.4894 | 3000 | 0.1660 | 5698896 | |
|
|
| 0.3348 | 0.5221 | 3200 | 0.1693 | 6074432 | |
|
|
| 0.0563 | 0.5547 | 3400 | 0.1644 | 6454208 | |
|
|
| 0.0621 | 0.5873 | 3600 | 0.1668 | 6831056 | |
|
|
| 0.099 | 0.6200 | 3800 | 0.1655 | 7209536 | |
|
|
| 0.2068 | 0.6526 | 4000 | 0.1644 | 7593024 | |
|
|
| 0.0542 | 0.6852 | 4200 | 0.1623 | 7977072 | |
|
|
| 0.2057 | 0.7178 | 4400 | 0.1649 | 8353296 | |
|
|
| 0.1223 | 0.7505 | 4600 | 0.1606 | 8733232 | |
|
|
| 0.2777 | 0.7831 | 4800 | 0.1593 | 9113632 | |
|
|
| 0.0492 | 0.8157 | 5000 | 0.1642 | 9487952 | |
|
|
| 0.1335 | 0.8484 | 5200 | 0.1601 | 9861104 | |
|
|
| 0.1896 | 0.8810 | 5400 | 0.1587 | 10239088 | |
|
|
| 0.2386 | 0.9136 | 5600 | 0.1569 | 10619840 | |
|
|
| 0.2734 | 0.9462 | 5800 | 0.1564 | 10994720 | |
|
|
| 0.1958 | 0.9789 | 6000 | 0.1585 | 11376976 | |
|
|
| 0.103 | 1.0114 | 6200 | 0.1570 | 11758656 | |
|
|
| 0.2747 | 1.0440 | 6400 | 0.1573 | 12144016 | |
|
|
| 0.1405 | 1.0767 | 6600 | 0.1613 | 12531776 | |
|
|
| 0.2157 | 1.1093 | 6800 | 0.1581 | 12905136 | |
|
|
| 0.1939 | 1.1419 | 7000 | 0.1559 | 13278096 | |
|
|
| 0.2038 | 1.1746 | 7200 | 0.1634 | 13651520 | |
|
|
| 0.0898 | 1.2072 | 7400 | 0.1580 | 14034784 | |
|
|
| 0.2486 | 1.2398 | 7600 | 0.1530 | 14415120 | |
|
|
| 0.1112 | 1.2725 | 7800 | 0.1541 | 14794784 | |
|
|
| 0.0631 | 1.3051 | 8000 | 0.1565 | 15176240 | |
|
|
| 0.1669 | 1.3377 | 8200 | 0.1532 | 15548080 | |
|
|
| 0.0363 | 1.3703 | 8400 | 0.1529 | 15926832 | |
|
|
| 0.1787 | 1.4030 | 8600 | 0.1536 | 16305344 | |
|
|
| 0.0654 | 1.4356 | 8800 | 0.1537 | 16686528 | |
|
|
| 0.1872 | 1.4682 | 9000 | 0.1504 | 17073648 | |
|
|
| 0.0465 | 1.5009 | 9200 | 0.1515 | 17457952 | |
|
|
| 0.2969 | 1.5335 | 9400 | 0.1535 | 17831104 | |
|
|
| 0.1203 | 1.5661 | 9600 | 0.1644 | 18215168 | |
|
|
| 0.1355 | 1.5987 | 9800 | 0.1581 | 18592816 | |
|
|
| 0.1982 | 1.6314 | 10000 | 0.1482 | 18972864 | |
|
|
| 0.0933 | 1.6640 | 10200 | 0.1500 | 19350160 | |
|
|
| 0.2432 | 1.6966 | 10400 | 0.1490 | 19735024 | |
|
|
| 0.238 | 1.7293 | 10600 | 0.1493 | 20108768 | |
|
|
| 0.276 | 1.7619 | 10800 | 0.1471 | 20489424 | |
|
|
| 0.1526 | 1.7945 | 11000 | 0.1509 | 20870832 | |
|
|
| 0.1532 | 1.8271 | 11200 | 0.1501 | 21240960 | |
|
|
| 0.1297 | 1.8598 | 11400 | 0.1472 | 21615744 | |
|
|
| 0.2225 | 1.8924 | 11600 | 0.1479 | 21991984 | |
|
|
| 0.0627 | 1.9250 | 11800 | 0.1524 | 22366624 | |
|
|
| 0.1337 | 1.9577 | 12000 | 0.1514 | 22746000 | |
|
|
| 0.1459 | 1.9903 | 12200 | 0.1455 | 23122688 | |
|
|
| 0.092 | 2.0228 | 12400 | 0.1473 | 23494112 | |
|
|
| 0.2195 | 2.0555 | 12600 | 0.1492 | 23876160 | |
|
|
| 0.1655 | 2.0881 | 12800 | 0.1500 | 24261904 | |
|
|
| 0.122 | 2.1207 | 13000 | 0.1510 | 24643776 | |
|
|
| 0.0303 | 2.1534 | 13200 | 0.1468 | 25020496 | |
|
|
| 0.1037 | 2.1860 | 13400 | 0.1452 | 25391072 | |
|
|
| 0.0869 | 2.2186 | 13600 | 0.1498 | 25762416 | |
|
|
| 0.1484 | 2.2512 | 13800 | 0.1475 | 26139456 | |
|
|
| 0.0902 | 2.2839 | 14000 | 0.1471 | 26511344 | |
|
|
| 0.097 | 2.3165 | 14200 | 0.1510 | 26891616 | |
|
|
| 0.176 | 2.3491 | 14400 | 0.1493 | 27274960 | |
|
|
| 0.1557 | 2.3818 | 14600 | 0.1454 | 27652224 | |
|
|
| 0.2243 | 2.4144 | 14800 | 0.1457 | 28033168 | |
|
|
| 0.2481 | 2.4470 | 15000 | 0.1455 | 28414784 | |
|
|
| 0.2391 | 2.4796 | 15200 | 0.1512 | 28787168 | |
|
|
| 0.1592 | 2.5123 | 15400 | 0.1452 | 29164512 | |
|
|
| 0.0712 | 2.5449 | 15600 | 0.1458 | 29545056 | |
|
|
| 0.0906 | 2.5775 | 15800 | 0.1472 | 29922176 | |
|
|
| 0.0529 | 2.6102 | 16000 | 0.1474 | 30304336 | |
|
|
| 0.0921 | 2.6428 | 16200 | 0.1486 | 30688608 | |
|
|
| 0.2052 | 2.6754 | 16400 | 0.1460 | 31067744 | |
|
|
| 0.104 | 2.7081 | 16600 | 0.1479 | 31455328 | |
|
|
| 0.0636 | 2.7407 | 16800 | 0.1453 | 31833136 | |
|
|
| 0.1624 | 2.7733 | 17000 | 0.1447 | 32213296 | |
|
|
| 0.1023 | 2.8059 | 17200 | 0.1474 | 32588128 | |
|
|
| 0.2427 | 2.8386 | 17400 | 0.1442 | 32971552 | |
|
|
| 0.0961 | 2.8712 | 17600 | 0.1430 | 33356064 | |
|
|
| 0.1 | 2.9038 | 17800 | 0.1443 | 33739984 | |
|
|
| 0.0681 | 2.9365 | 18000 | 0.1442 | 34121824 | |
|
|
| 0.0941 | 2.9691 | 18200 | 0.1482 | 34498368 | |
|
|
| 0.0614 | 3.0016 | 18400 | 0.1465 | 34866272 | |
|
|
| 0.0636 | 3.0343 | 18600 | 0.1459 | 35258768 | |
|
|
| 0.1946 | 3.0669 | 18800 | 0.1436 | 35644416 | |
|
|
| 0.1265 | 3.0995 | 19000 | 0.1451 | 36017808 | |
|
|
| 0.0472 | 3.1321 | 19200 | 0.1423 | 36393536 | |
|
|
| 0.3222 | 3.1648 | 19400 | 0.1427 | 36770432 | |
|
|
| 0.055 | 3.1974 | 19600 | 0.1422 | 37152448 | |
|
|
| 0.1345 | 3.2300 | 19800 | 0.1473 | 37532496 | |
|
|
| 0.2611 | 3.2627 | 20000 | 0.1419 | 37910480 | |
|
|
| 0.2644 | 3.2953 | 20200 | 0.1422 | 38286080 | |
|
|
| 0.1328 | 3.3279 | 20400 | 0.1427 | 38664512 | |
|
|
| 0.0519 | 3.3606 | 20600 | 0.1426 | 39053472 | |
|
|
| 0.2293 | 3.3932 | 20800 | 0.1426 | 39432032 | |
|
|
| 0.216 | 3.4258 | 21000 | 0.1428 | 39812704 | |
|
|
| 0.0973 | 3.4584 | 21200 | 0.1435 | 40191088 | |
|
|
| 0.1368 | 3.4911 | 21400 | 0.1416 | 40567216 | |
|
|
| 0.0711 | 3.5237 | 21600 | 0.1451 | 40947696 | |
|
|
| 0.041 | 3.5563 | 21800 | 0.1432 | 41330624 | |
|
|
| 0.1808 | 3.5890 | 22000 | 0.1447 | 41708800 | |
|
|
| 0.0172 | 3.6216 | 22200 | 0.1433 | 42087824 | |
|
|
| 0.1509 | 3.6542 | 22400 | 0.1472 | 42461936 | |
|
|
| 0.1676 | 3.6868 | 22600 | 0.1443 | 42843696 | |
|
|
| 0.1843 | 3.7195 | 22800 | 0.1430 | 43221120 | |
|
|
| 0.1459 | 3.7521 | 23000 | 0.1420 | 43597776 | |
|
|
| 0.0701 | 3.7847 | 23200 | 0.1425 | 43979312 | |
|
|
| 0.1427 | 3.8174 | 23400 | 0.1432 | 44354480 | |
|
|
| 0.1453 | 3.8500 | 23600 | 0.1440 | 44727696 | |
|
|
| 0.2241 | 3.8826 | 23800 | 0.1465 | 45108608 | |
|
|
| 0.1722 | 3.9152 | 24000 | 0.1463 | 45482928 | |
|
|
| 0.1048 | 3.9479 | 24200 | 0.1429 | 45861584 | |
|
|
| 0.2229 | 3.9805 | 24400 | 0.1408 | 46243072 | |
|
|
| 0.1949 | 4.0131 | 24600 | 0.1428 | 46619680 | |
|
|
| 0.1122 | 4.0457 | 24800 | 0.1409 | 47007360 | |
|
|
| 0.1824 | 4.0783 | 25000 | 0.1419 | 47391600 | |
|
|
| 0.0632 | 4.1109 | 25200 | 0.1415 | 47768320 | |
|
|
| 0.1647 | 4.1436 | 25400 | 0.1448 | 48143424 | |
|
|
| 0.0713 | 4.1762 | 25600 | 0.1425 | 48524368 | |
|
|
| 0.1584 | 4.2088 | 25800 | 0.1409 | 48899856 | |
|
|
| 0.0332 | 4.2415 | 26000 | 0.1439 | 49280208 | |
|
|
| 0.1167 | 4.2741 | 26200 | 0.1445 | 49658080 | |
|
|
| 0.0985 | 4.3067 | 26400 | 0.1429 | 50034848 | |
|
|
| 0.1474 | 4.3393 | 26600 | 0.1418 | 50413376 | |
|
|
| 0.1732 | 4.3720 | 26800 | 0.1421 | 50793248 | |
|
|
| 0.1472 | 4.4046 | 27000 | 0.1414 | 51170976 | |
|
|
| 0.1362 | 4.4372 | 27200 | 0.1409 | 51559504 | |
|
|
| 0.0327 | 4.4699 | 27400 | 0.1415 | 51928704 | |
|
|
| 0.1415 | 4.5025 | 27600 | 0.1411 | 52297776 | |
|
|
| 0.0763 | 4.5351 | 27800 | 0.1419 | 52669472 | |
|
|
| 0.0933 | 4.5677 | 28000 | 0.1428 | 53045856 | |
|
|
| 0.2031 | 4.6004 | 28200 | 0.1423 | 53429232 | |
|
|
| 0.1147 | 4.6330 | 28400 | 0.1427 | 53810560 | |
|
|
| 0.2305 | 4.6656 | 28600 | 0.1429 | 54191536 | |
|
|
| 0.1353 | 4.6983 | 28800 | 0.1415 | 54572176 | |
|
|
| 0.0633 | 4.7309 | 29000 | 0.1413 | 54952896 | |
|
|
| 0.1857 | 4.7635 | 29200 | 0.1424 | 55327776 | |
|
|
| 0.0332 | 4.7961 | 29400 | 0.1412 | 55708896 | |
|
|
| 0.0454 | 4.8288 | 29600 | 0.1423 | 56085712 | |
|
|
| 0.0823 | 4.8614 | 29800 | 0.1421 | 56467376 | |
|
|
| 0.0687 | 4.8940 | 30000 | 0.1422 | 56841328 | |
|
|
| 0.0976 | 4.9267 | 30200 | 0.1432 | 57227184 | |
|
|
| 0.0957 | 4.9593 | 30400 | 0.1426 | 57605632 | |
|
|
| 0.1694 | 4.9919 | 30600 | 0.1418 | 57987472 | |
|
|
| 0.0476 | 5.0245 | 30800 | 0.1421 | 58367056 | |
|
|
| 0.1684 | 5.0571 | 31000 | 0.1426 | 58746720 | |
|
|
| 0.0495 | 5.0897 | 31200 | 0.1413 | 59124272 | |
|
|
| 0.0825 | 5.1224 | 31400 | 0.1407 | 59504688 | |
|
|
| 0.1022 | 5.1550 | 31600 | 0.1402 | 59875840 | |
|
|
| 0.107 | 5.1876 | 31800 | 0.1410 | 60247360 | |
|
|
| 0.0413 | 5.2202 | 32000 | 0.1409 | 60622464 | |
|
|
| 0.0824 | 5.2529 | 32200 | 0.1405 | 61006768 | |
|
|
| 0.0257 | 5.2855 | 32400 | 0.1407 | 61386992 | |
|
|
| 0.314 | 5.3181 | 32600 | 0.1407 | 61770000 | |
|
|
| 0.1222 | 5.3508 | 32800 | 0.1403 | 62154640 | |
|
|
| 0.1365 | 5.3834 | 33000 | 0.1407 | 62541664 | |
|
|
| 0.0645 | 5.4160 | 33200 | 0.1419 | 62912976 | |
|
|
| 0.181 | 5.4486 | 33400 | 0.1417 | 63289520 | |
|
|
| 0.3059 | 5.4813 | 33600 | 0.1418 | 63668416 | |
|
|
| 0.0325 | 5.5139 | 33800 | 0.1410 | 64043792 | |
|
|
| 0.0339 | 5.5465 | 34000 | 0.1409 | 64433840 | |
|
|
| 0.1333 | 5.5792 | 34200 | 0.1413 | 64808624 | |
|
|
| 0.0653 | 5.6118 | 34400 | 0.1410 | 65182704 | |
|
|
| 0.0782 | 5.6444 | 34600 | 0.1412 | 65562192 | |
|
|
| 0.1815 | 5.6771 | 34800 | 0.1420 | 65940816 | |
|
|
| 0.1105 | 5.7097 | 35000 | 0.1421 | 66326768 | |
|
|
| 0.1614 | 5.7423 | 35200 | 0.1419 | 66705744 | |
|
|
| 0.2069 | 5.7749 | 35400 | 0.1422 | 67084928 | |
|
|
| 0.0295 | 5.8076 | 35600 | 0.1419 | 67462064 | |
|
|
| 0.1712 | 5.8402 | 35800 | 0.1421 | 67846112 | |
|
|
| 0.0931 | 5.8728 | 36000 | 0.1422 | 68221552 | |
|
|
| 0.1396 | 5.9055 | 36200 | 0.1418 | 68606416 | |
|
|
| 0.1184 | 5.9381 | 36400 | 0.1427 | 68980176 | |
|
|
| 0.0667 | 5.9707 | 36600 | 0.1425 | 69349984 | |
|
|
| 0.0768 | 6.0033 | 36800 | 0.1419 | 69729984 | |
|
|
| 0.1498 | 6.0359 | 37000 | 0.1418 | 70107936 | |
|
|
| 0.1819 | 6.0685 | 37200 | 0.1420 | 70487856 | |
|
|
| 0.0429 | 6.1012 | 37400 | 0.1422 | 70865792 | |
|
|
| 0.0431 | 6.1338 | 37600 | 0.1421 | 71244784 | |
|
|
| 0.2317 | 6.1664 | 37800 | 0.1420 | 71630704 | |
|
|
| 0.0573 | 6.1990 | 38000 | 0.1415 | 72002688 | |
|
|
| 0.3545 | 6.2317 | 38200 | 0.1421 | 72385776 | |
|
|
| 0.1921 | 6.2643 | 38400 | 0.1422 | 72773152 | |
|
|
| 0.1193 | 6.2969 | 38600 | 0.1421 | 73149584 | |
|
|
| 0.187 | 6.3296 | 38800 | 0.1419 | 73519536 | |
|
|
| 0.2218 | 6.3622 | 39000 | 0.1415 | 73902896 | |
|
|
| 0.1392 | 6.3948 | 39200 | 0.1423 | 74278960 | |
|
|
| 0.2032 | 6.4274 | 39400 | 0.1425 | 74655728 | |
|
|
| 0.0652 | 6.4601 | 39600 | 0.1425 | 75025808 | |
|
|
| 0.0902 | 6.4927 | 39800 | 0.1425 | 75402576 | |
|
|
| 0.0873 | 6.5253 | 40000 | 0.1425 | 75778784 | |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.15.2.dev0 |
|
|
- Transformers 4.51.3 |
|
|
- Pytorch 2.6.0+cu124 |
|
|
- Datasets 3.5.0 |
|
|
- Tokenizers 0.21.1 |
