train_rte_1744902662

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the rte dataset. It achieves the following results on the evaluation set:

• Loss: 0.0684
• Num Input Tokens Seen: 107274480

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• training_steps: 40000

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.0778	1.4207	200	0.1123	540280
0.0786	2.8414	400	0.0933	1077480
0.0634	4.2567	600	0.0881	1609584
0.0893	5.6774	800	0.0856	2150192
0.0545	7.0927	1000	0.0823	2681640
0.0455	8.5134	1200	0.0804	3218528
0.0652	9.9340	1400	0.0779	3757240
0.095	11.3494	1600	0.0765	4292384
0.0727	12.7701	1800	0.0748	4828992
0.048	14.1854	2000	0.0737	5364048
0.0597	15.6061	2200	0.0725	5901512
0.0434	17.0214	2400	0.0719	6435768
0.0823	18.4421	2600	0.0712	6974976
0.0433	19.8627	2800	0.0706	7509488
0.0495	21.2781	3000	0.0709	8041736
0.0644	22.6988	3200	0.0687	8583128
0.0558	24.1141	3400	0.0701	9117488
0.0524	25.5348	3600	0.0692	9649136
0.0403	26.9554	3800	0.0690	10191288
0.0445	28.3708	4000	0.0684	10724032
0.0364	29.7914	4200	0.0687	11259816
0.0463	31.2068	4400	0.0690	11805200
0.0228	32.6275	4600	0.0694	12337832
0.0278	34.0428	4800	0.0694	12874672
0.0366	35.4635	5000	0.0688	13408200
0.0565	36.8841	5200	0.0701	13943952
0.0289	38.2995	5400	0.0687	14478600
0.0526	39.7201	5600	0.0715	15021728
0.0445	41.1355	5800	0.0716	15548872
0.0241	42.5561	6000	0.0714	16082664
0.0275	43.9768	6200	0.0719	16624832
0.0402	45.3922	6400	0.0738	17152040
0.0521	46.8128	6600	0.0733	17696104
0.029	48.2282	6800	0.0738	18228312
0.0214	49.6488	7000	0.0743	18767376
0.0272	51.0642	7200	0.0757	19300560
0.0221	52.4848	7400	0.0773	19837208
0.011	53.9055	7600	0.0767	20381384
0.0281	55.3209	7800	0.0797	20917960
0.0248	56.7415	8000	0.0790	21456616
0.0318	58.1569	8200	0.0802	21988808
0.0348	59.5775	8400	0.0843	22526872
0.031	60.9982	8600	0.0832	23067872
0.0297	62.4135	8800	0.0843	23599328
0.0176	63.8342	9000	0.0853	24138832
0.0229	65.2496	9200	0.0887	24675016
0.0163	66.6702	9400	0.0918	25209352
0.0148	68.0856	9600	0.0938	25745352
0.0029	69.5062	9800	0.0975	26284824
0.0062	70.9269	10000	0.0991	26824264
0.003	72.3422	10200	0.1025	27363992
0.0116	73.7629	10400	0.1026	27904360
0.0082	75.1783	10600	0.1085	28436064
0.0261	76.5989	10800	0.1116	28976440
0.0054	78.0143	11000	0.1136	29511840
0.0029	79.4349	11200	0.1186	30049440
0.006	80.8556	11400	0.1239	30590008
0.0127	82.2709	11600	0.1293	31127008
0.0051	83.6916	11800	0.1319	31665584
0.0067	85.1070	12000	0.1387	32199088
0.0039	86.5276	12200	0.1411	32739240
0.0069	87.9483	12400	0.1456	33281296
0.0042	89.3636	12600	0.1521	33819016
0.0029	90.7843	12800	0.1591	34356400
0.0017	92.1996	13000	0.1638	34889896
0.0051	93.6203	13200	0.1682	35429768
0.001	95.0357	13400	0.1739	35969976
0.0013	96.4563	13600	0.1820	36505712
0.0007	97.8770	13800	0.1836	37036976
0.0004	99.2923	14000	0.1941	37570400
0.0004	100.7130	14200	0.1960	38103616
0.0004	102.1283	14400	0.1996	38636544
0.0004	103.5490	14600	0.2114	39171560
0.0014	104.9697	14800	0.2186	39706992
0.0002	106.3850	15000	0.2263	40239280
0.0006	107.8057	15200	0.2279	40778072
0.0002	109.2210	15400	0.2329	41312720
0.0001	110.6417	15600	0.2427	41845224
0.0001	112.0570	15800	0.2472	42384256
0.0001	113.4777	16000	0.2527	42925008
0.0001	114.8984	16200	0.2539	43462528
0.0003	116.3137	16400	0.2598	43999968
0.0001	117.7344	16600	0.2656	44533664
0.0001	119.1497	16800	0.2654	45067976
0.0001	120.5704	17000	0.2706	45610752
0.0001	121.9911	17200	0.2695	46147416
0.0001	123.4064	17400	0.2831	46682792
0.0001	124.8271	17600	0.2867	47218688
0.0001	126.2424	17800	0.2923	47751176
0.0	127.6631	18000	0.2893	48286872
0.0001	129.0784	18200	0.3002	48824840
0.0001	130.4991	18400	0.2981	49361064
0.0001	131.9198	18600	0.3053	49893616
0.0	133.3351	18800	0.3089	50425120
0.0	134.7558	19000	0.3148	50963088
0.0	136.1711	19200	0.3139	51496048
0.0	137.5918	19400	0.3163	52038608
0.0	139.0071	19600	0.3232	52575544
0.0	140.4278	19800	0.3225	53114912
0.0	141.8485	20000	0.3298	53657368
0.0	143.2638	20200	0.3290	54195776
0.0	144.6845	20400	0.3311	54722232
0.0	146.0998	20600	0.3375	55255168
0.0	147.5205	20800	0.3408	55786616
0.0	148.9412	21000	0.3495	56322200
0.0	150.3565	21200	0.3486	56860136
0.0	151.7772	21400	0.3489	57396560
0.0	153.1925	21600	0.3573	57930904
0.0	154.6132	21800	0.3561	58469832
0.0	156.0285	22000	0.3613	59001744
0.0	157.4492	22200	0.3674	59542632
0.0	158.8699	22400	0.3639	60077280
0.0	160.2852	22600	0.3684	60614824
0.0	161.7059	22800	0.3740	61145384
0.0	163.1212	23000	0.3782	61678824
0.0	164.5419	23200	0.3845	62213064
0.0	165.9626	23400	0.3833	62746840
0.0	167.3779	23600	0.3817	63279640
0.0	168.7986	23800	0.3894	63817648
0.0	170.2139	24000	0.3891	64355456
0.0	171.6346	24200	0.3878	64891336
0.0	173.0499	24400	0.3959	65431304
0.0	174.4706	24600	0.3989	65971176
0.0	175.8913	24800	0.3945	66508200
0.0	177.3066	25000	0.4059	67044512
0.0	178.7273	25200	0.4071	67581248
0.0	180.1426	25400	0.4062	68116280
0.0	181.5633	25600	0.4176	68654016
0.0	182.9840	25800	0.4113	69191168
0.0	184.3993	26000	0.4147	69725736
0.0	185.8200	26200	0.4204	70266432
0.0	187.2353	26400	0.4238	70795080
0.0	188.6560	26600	0.4208	71337664
0.0	190.0713	26800	0.4231	71873944
0.0	191.4920	27000	0.4281	72406760
0.0	192.9127	27200	0.4262	72941856
0.0	194.3280	27400	0.4358	73486320
0.0	195.7487	27600	0.4344	74024784
0.0	197.1640	27800	0.4328	74562272
0.0	198.5847	28000	0.4311	75101016
0.0	200.0	28200	0.4385	75632576
0.0	201.4207	28400	0.4355	76166696
0.0	202.8414	28600	0.4365	76703192
0.0	204.2567	28800	0.4444	77237304
0.0	205.6774	29000	0.4429	77775808
0.0	207.0927	29200	0.4520	78304552
0.0	208.5134	29400	0.4476	78842312
0.0	209.9340	29600	0.4482	79379384
0.0	211.3494	29800	0.4616	79916200
0.0	212.7701	30000	0.4521	80450848
0.0	214.1854	30200	0.4568	80978696
0.0	215.6061	30400	0.4638	81517864
0.0	217.0214	30600	0.4560	82057360
0.0	218.4421	30800	0.4525	82601680
0.0	219.8627	31000	0.4652	83137640
0.0	221.2781	31200	0.4654	83674536
0.0	222.6988	31400	0.4640	84215064
0.0	224.1141	31600	0.4630	84750440
0.0	225.5348	31800	0.4612	85284976
0.0	226.9554	32000	0.4595	85820408
0.0	228.3708	32200	0.4604	86358288
0.0	229.7914	32400	0.4609	86896432
0.0	231.2068	32600	0.4675	87433496
0.0	232.6275	32800	0.4675	87969480
0.0	234.0428	33000	0.4732	88503984
0.0	235.4635	33200	0.4790	89043584
0.0	236.8841	33400	0.4747	89572896
0.0	238.2995	33600	0.4836	90114360
0.0	239.7201	33800	0.4766	90650032
0.0	241.1355	34000	0.4783	91178208
0.0	242.5561	34200	0.4764	91712168
0.0	243.9768	34400	0.4746	92253832
0.0	245.3922	34600	0.4764	92783304
0.0	246.8128	34800	0.4837	93323088
0.0	248.2282	35000	0.4820	93858448
0.0	249.6488	35200	0.4862	94391144
0.0	251.0642	35400	0.4764	94929608
0.0	252.4848	35600	0.4799	95474424
0.0	253.9055	35800	0.4801	96007792
0.0	255.3209	36000	0.4891	96546584
0.0	256.7415	36200	0.4852	97077888
0.0	258.1569	36400	0.4851	97612368
0.0	259.5775	36600	0.4912	98151616
0.0	260.9982	36800	0.4780	98684232
0.0	262.4135	37000	0.4813	99220560
0.0	263.8342	37200	0.4805	99758136
0.0	265.2496	37400	0.4862	100296152
0.0	266.6702	37600	0.4896	100836120
0.0	268.0856	37800	0.4815	101372264
0.0	269.5062	38000	0.4875	101912112
0.0	270.9269	38200	0.4872	102446016
0.0	272.3422	38400	0.4860	102980360
0.0	273.7629	38600	0.4883	103519296
0.0	275.1783	38800	0.4914	104053200
0.0	276.5989	39000	0.4837	104594720
0.0	278.0143	39200	0.4871	105126640
0.0	279.4349	39400	0.4871	105660640
0.0	280.8556	39600	0.4871	106198248
0.0	282.2709	39800	0.4871	106737720
0.0	283.6916	40000	0.4871	107274480

Framework versions

• PEFT 0.15.1
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1