train_rte_1744902660

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the rte dataset. It achieves the following results on the evaluation set:

• Loss: 0.0774
• Num Input Tokens Seen: 98761256

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• training_steps: 40000

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.0584	1.4207	200	0.0774	496688
0.0039	2.8414	400	0.0834	991488
0.0003	4.2567	600	0.1363	1481464
0.0004	5.6774	800	0.1855	1979088
0.0001	7.0927	1000	0.2183	2468504
0.0	8.5134	1200	0.1933	2963120
0.0001	9.9340	1400	0.2315	3459048
0.0	11.3494	1600	0.2587	3951104
0.0	12.7701	1800	0.2743	4445432
0.0	14.1854	2000	0.2844	4938824
0.0	15.6061	2200	0.2960	5433720
0.0	17.0214	2400	0.3036	5925896
0.0	18.4421	2600	0.3125	6422360
0.0	19.8627	2800	0.3172	6914152
0.0	21.2781	3000	0.3272	7403976
0.0	22.6988	3200	0.3299	7902520
0.0	24.1141	3400	0.3362	8394080
0.0	25.5348	3600	0.3423	8884224
0.0	26.9554	3800	0.3420	9382368
0.0	28.3708	4000	0.3501	9872768
0.0	29.7914	4200	0.3545	10366000
0.0	31.2068	4400	0.3608	10867488
0.0	32.6275	4600	0.3596	11358568
0.0	34.0428	4800	0.3623	11852320
0.0	35.4635	5000	0.3695	12343880
0.0	36.8841	5200	0.3708	12837040
0.0	38.2995	5400	0.3777	13329368
0.0	39.7201	5600	0.3780	13828784
0.0	41.1355	5800	0.3850	14315304
0.0	42.5561	6000	0.3871	14806592
0.0	43.9768	6200	0.3893	15305208
0.0	45.3922	6400	0.3951	15791608
0.0	46.8128	6600	0.3974	16292464
0.0	48.2282	6800	0.3966	16781768
0.0	49.6488	7000	0.4015	17278560
0.0	51.0642	7200	0.4038	17769384
0.0	52.4848	7400	0.3994	18262680
0.0	53.9055	7600	0.4118	18763936
0.0	55.3209	7800	0.4110	19258096
0.0	56.7415	8000	0.4104	19753648
0.0	58.1569	8200	0.4171	20244128
0.0	59.5775	8400	0.4128	20739208
0.0	60.9982	8600	0.4171	21236872
0.0	62.4135	8800	0.4167	21726944
0.0	63.8342	9000	0.4200	22223288
0.0	65.2496	9200	0.4206	22716672
0.0	66.6702	9400	0.4227	23209088
0.0	68.0856	9600	0.4196	23701520
0.0	69.5062	9800	0.4218	24197944
0.0	70.9269	10000	0.4202	24694272
0.0	72.3422	10200	0.4217	25191256
0.0	73.7629	10400	0.4249	25688288
0.0	75.1783	10600	0.4238	26177720
0.0	76.5989	10800	0.4300	26675248
0.0	78.0143	11000	0.4231	27168496
0.0	79.4349	11200	0.4251	27664360
0.0	80.8556	11400	0.4301	28161984
0.0	82.2709	11600	0.4217	28655448
0.0	83.6916	11800	0.4288	29151808
0.0	85.1070	12000	0.4339	29642952
0.0	86.5276	12200	0.4287	30140536
0.0	87.9483	12400	0.4317	30639808
0.0	89.3636	12600	0.4309	31135048
0.0	90.7843	12800	0.4307	31630256
0.0	92.1996	13000	0.4296	32121256
0.0	93.6203	13200	0.4304	32618184
0.0	95.0357	13400	0.4312	33115432
0.0	96.4563	13600	0.4353	33609472
0.0	97.8770	13800	0.4407	34098712
0.0	99.2923	14000	0.4400	34590368
0.0	100.7130	14200	0.4391	35081248
0.0	102.1283	14400	0.4460	35571464
0.0	103.5490	14600	0.4449	36063824
0.0	104.9697	14800	0.4521	36557944
0.0	106.3850	15000	0.4538	37048560
0.0	107.8057	15200	0.4597	37543928
0.0	109.2210	15400	0.4631	38035968
0.0	110.6417	15600	0.4684	38526000
0.0	112.0570	15800	0.4645	39021440
0.0	113.4777	16000	0.4721	39519712
0.0	114.8984	16200	0.4756	40014440
0.0	116.3137	16400	0.4805	40509368
0.0	117.7344	16600	0.4793	41001000
0.0	119.1497	16800	0.4868	41492672
0.0	120.5704	17000	0.4840	41991984
0.0	121.9911	17200	0.4771	42486736
0.0	123.4064	17400	0.4898	42979888
0.0	124.8271	17600	0.4915	43473920
0.0	126.2424	17800	0.4914	43963728
0.0	127.6631	18000	0.4932	44457208
0.0	129.0784	18200	0.4886	44952664
0.0	130.4991	18400	0.4925	45446704
0.0	131.9198	18600	0.4959	45936552
0.0	133.3351	18800	0.5014	46426240
0.0	134.7558	19000	0.4893	46921256
0.0	136.1711	19200	0.4897	47412080
0.0	137.5918	19400	0.4879	47911024
0.0	139.0071	19600	0.4870	48404752
0.0	140.4278	19800	0.4864	48901416
0.0	141.8485	20000	0.4915	49400736
0.0	143.2638	20200	0.4930	49895752
0.0	144.6845	20400	0.4899	50380736
0.0	146.0998	20600	0.4898	50871288
0.0	147.5205	20800	0.4821	51360328
0.0	148.9412	21000	0.4874	51853696
0.0	150.3565	21200	0.4815	52348712
0.0	151.7772	21400	0.4884	52842992
0.0	153.1925	21600	0.4864	53335368
0.0	154.6132	21800	0.4901	53831240
0.0	156.0285	22000	0.4912	54320840
0.0	157.4492	22200	0.4865	54818304
0.0	158.8699	22400	0.4858	55310560
0.0	160.2852	22600	0.4850	55805192
0.0	161.7059	22800	0.4839	56294240
0.0	163.1212	23000	0.4884	56785216
0.0	164.5419	23200	0.4807	57277112
0.0	165.9626	23400	0.4874	57768960
0.0	167.3779	23600	0.4851	58259216
0.0	168.7986	23800	0.4885	58754552
0.0	170.2139	24000	0.4892	59250304
0.0	171.6346	24200	0.4885	59743752
0.0	173.0499	24400	0.4832	60240920
0.0	174.4706	24600	0.4898	60738488
0.0	175.8913	24800	0.4862	61232632
0.0	177.3066	25000	0.4878	61726896
0.0	178.7273	25200	0.4899	62220440
0.0	180.1426	25400	0.4852	62713544
0.0	181.5633	25600	0.4852	63208560
0.0	182.9840	25800	0.4849	63703320
0.0	184.3993	26000	0.4857	64195280
0.0	185.8200	26200	0.4884	64693448
0.0	187.2353	26400	0.4897	65180864
0.0	188.6560	26600	0.4893	65680024
0.0	190.0713	26800	0.4870	66173368
0.0	191.4920	27000	0.4905	66664968
0.0	192.9127	27200	0.4854	67157528
0.0	194.3280	27400	0.4863	67657848
0.0	195.7487	27600	0.4899	68154280
0.0	197.1640	27800	0.4914	68648760
0.0	198.5847	28000	0.4902	69145424
0.0	200.0	28200	0.4871	69634592
0.0	201.4207	28400	0.4901	70126824
0.0	202.8414	28600	0.4939	70621048
0.0	204.2567	28800	0.4908	71112744
0.0	205.6774	29000	0.4958	71609328
0.0	207.0927	29200	0.4913	72096488
0.0	208.5134	29400	0.4966	72590600
0.0	209.9340	29600	0.4942	73085400
0.0	211.3494	29800	0.4975	73578704
0.0	212.7701	30000	0.4928	74071832
0.0	214.1854	30200	0.4931	74558088
0.0	215.6061	30400	0.4973	75054720
0.0	217.0214	30600	0.4974	75550968
0.0	218.4421	30800	0.4968	76052048
0.0	219.8627	31000	0.4932	76544760
0.0	221.2781	31200	0.4976	77039312
0.0	222.6988	31400	0.4983	77536608
0.0	224.1141	31600	0.4965	78029096
0.0	225.5348	31800	0.4944	78521640
0.0	226.9554	32000	0.4984	79014704
0.0	228.3708	32200	0.5011	79509056
0.0	229.7914	32400	0.4929	80004760
0.0	231.2068	32600	0.4949	80498576
0.0	232.6275	32800	0.5013	80992160
0.0	234.0428	33000	0.4981	81484216
0.0	235.4635	33200	0.5018	81981536
0.0	236.8841	33400	0.5018	82469112
0.0	238.2995	33600	0.5000	82967264
0.0	239.7201	33800	0.4991	83460632
0.0	241.1355	34000	0.5007	83946936
0.0	242.5561	34200	0.4985	84438976
0.0	243.9768	34400	0.4987	84936992
0.0	245.3922	34600	0.5009	85424648
0.0	246.8128	34800	0.5009	85921552
0.0	248.2282	35000	0.5021	86414392
0.0	249.6488	35200	0.4961	86904424
0.0	251.0642	35400	0.5033	87399560
0.0	252.4848	35600	0.4995	87900568
0.0	253.9055	35800	0.5003	88391952
0.0	255.3209	36000	0.5009	88887288
0.0	256.7415	36200	0.5007	89375944
0.0	258.1569	36400	0.5006	89868176
0.0	259.5775	36600	0.5023	90365056
0.0	260.9982	36800	0.4998	90855096
0.0	262.4135	37000	0.4999	91348504
0.0	263.8342	37200	0.4994	91843280
0.0	265.2496	37400	0.5007	92339160
0.0	266.6702	37600	0.5034	92834936
0.0	268.0856	37800	0.5040	93329096
0.0	269.5062	38000	0.4936	93825960
0.0	270.9269	38200	0.5021	94316976
0.0	272.3422	38400	0.4993	94808456
0.0	273.7629	38600	0.4997	95304384
0.0	275.1783	38800	0.4965	95796256
0.0	276.5989	39000	0.4983	96293992
0.0	278.0143	39200	0.4999	96783960
0.0	279.4349	39400	0.4987	97275176
0.0	280.8556	39600	0.5033	97769584
0.0	282.2709	39800	0.5014	98266712
0.0	283.6916	40000	0.5009	98761256

Framework versions

• PEFT 0.15.1
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1