train_rte_1744902665

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the rte dataset. It achieves the following results on the evaluation set:

• Loss: 0.0704
• Num Input Tokens Seen: 107274480

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• training_steps: 40000

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.0864	1.4207	200	0.1323	540280
0.079	2.8414	400	0.0969	1077480
0.0663	4.2567	600	0.0895	1609584
0.0931	5.6774	800	0.0865	2150192
0.059	7.0927	1000	0.0837	2681640
0.0487	8.5134	1200	0.0818	3218528
0.0693	9.9340	1400	0.0799	3757240
0.0978	11.3494	1600	0.0791	4292384
0.0802	12.7701	1800	0.0775	4828992
0.0499	14.1854	2000	0.0768	5364048
0.0613	15.6061	2200	0.0752	5901512
0.0475	17.0214	2400	0.0745	6435768
0.0849	18.4421	2600	0.0741	6974976
0.0483	19.8627	2800	0.0733	7509488
0.0533	21.2781	3000	0.0735	8041736
0.0662	22.6988	3200	0.0715	8583128
0.0585	24.1141	3400	0.0720	9117488
0.0536	25.5348	3600	0.0720	9649136
0.0489	26.9554	3800	0.0714	10191288
0.0498	28.3708	4000	0.0714	10724032
0.0432	29.7914	4200	0.0711	11259816
0.0535	31.2068	4400	0.0715	11805200
0.0312	32.6275	4600	0.0715	12337832
0.0349	34.0428	4800	0.0714	12874672
0.0412	35.4635	5000	0.0709	13408200
0.0597	36.8841	5200	0.0715	13943952
0.0342	38.2995	5400	0.0704	14478600
0.059	39.7201	5600	0.0704	15021728
0.0522	41.1355	5800	0.0709	15548872
0.0295	42.5561	6000	0.0710	16082664
0.0325	43.9768	6200	0.0711	16624832
0.044	45.3922	6400	0.0711	17152040
0.0588	46.8128	6600	0.0719	17696104
0.0341	48.2282	6800	0.0721	18228312
0.0292	49.6488	7000	0.0730	18767376
0.0316	51.0642	7200	0.0735	19300560
0.0283	52.4848	7400	0.0736	19837208
0.0167	53.9055	7600	0.0731	20381384
0.0312	55.3209	7800	0.0762	20917960
0.0274	56.7415	8000	0.0755	21456616
0.0414	58.1569	8200	0.0755	21988808
0.0384	59.5775	8400	0.0778	22526872
0.0395	60.9982	8600	0.0762	23067872
0.0354	62.4135	8800	0.0781	23599328
0.0255	63.8342	9000	0.0781	24138832
0.0292	65.2496	9200	0.0791	24675016
0.0233	66.6702	9400	0.0797	25209352
0.022	68.0856	9600	0.0810	25745352
0.0069	69.5062	9800	0.0829	26284824
0.0125	70.9269	10000	0.0825	26824264
0.0052	72.3422	10200	0.0870	27363992
0.0222	73.7629	10400	0.0839	27904360
0.0177	75.1783	10600	0.0872	28436064
0.0359	76.5989	10800	0.0884	28976440
0.022	78.0143	11000	0.0893	29511840
0.0055	79.4349	11200	0.0917	30049440
0.011	80.8556	11400	0.0915	30590008
0.0186	82.2709	11600	0.0956	31127008
0.0242	83.6916	11800	0.0971	31665584
0.0262	85.1070	12000	0.0980	32199088
0.0126	86.5276	12200	0.1010	32739240
0.0115	87.9483	12400	0.1037	33281296
0.0202	89.3636	12600	0.1061	33819016
0.0209	90.7843	12800	0.1083	34356400
0.0078	92.1996	13000	0.1106	34889896
0.0097	93.6203	13200	0.1133	35429768
0.0048	95.0357	13400	0.1138	35969976
0.0062	96.4563	13600	0.1164	36505712
0.0024	97.8770	13800	0.1196	37036976
0.0023	99.2923	14000	0.1213	37570400
0.0026	100.7130	14200	0.1236	38103616
0.003	102.1283	14400	0.1292	38636544
0.0026	103.5490	14600	0.1275	39171560
0.0083	104.9697	14800	0.1316	39706992
0.0014	106.3850	15000	0.1339	40239280
0.0084	107.8057	15200	0.1374	40778072
0.0061	109.2210	15400	0.1412	41312720
0.0024	110.6417	15600	0.1484	41845224
0.0029	112.0570	15800	0.1469	42384256
0.0014	113.4777	16000	0.1485	42925008
0.0015	114.8984	16200	0.1511	43462528
0.004	116.3137	16400	0.1549	43999968
0.0013	117.7344	16600	0.1557	44533664
0.0008	119.1497	16800	0.1616	45067976
0.0021	120.5704	17000	0.1608	45610752
0.0015	121.9911	17200	0.1639	46147416
0.0012	123.4064	17400	0.1689	46682792
0.0013	124.8271	17600	0.1701	47218688
0.0119	126.2424	17800	0.1766	47751176
0.0007	127.6631	18000	0.1814	48286872
0.0031	129.0784	18200	0.1835	48824840
0.0041	130.4991	18400	0.1855	49361064
0.0042	131.9198	18600	0.1927	49893616
0.0004	133.3351	18800	0.1908	50425120
0.0004	134.7558	19000	0.1944	50963088
0.0006	136.1711	19200	0.2051	51496048
0.0003	137.5918	19400	0.2001	52038608
0.0007	139.0071	19600	0.2065	52575544
0.0003	140.4278	19800	0.2146	53114912
0.0003	141.8485	20000	0.2164	53657368
0.0022	143.2638	20200	0.2204	54195776
0.0002	144.6845	20400	0.2224	54722232
0.0006	146.0998	20600	0.2283	55255168
0.0006	147.5205	20800	0.2333	55786616
0.0004	148.9412	21000	0.2350	56322200
0.0003	150.3565	21200	0.2438	56860136
0.0002	151.7772	21400	0.2434	57396560
0.0001	153.1925	21600	0.2479	57930904
0.0001	154.6132	21800	0.2529	58469832
0.0001	156.0285	22000	0.2553	59001744
0.0001	157.4492	22200	0.2570	59542632
0.001	158.8699	22400	0.2659	60077280
0.0003	160.2852	22600	0.2696	60614824
0.0002	161.7059	22800	0.2692	61145384
0.0002	163.1212	23000	0.2708	61678824
0.0001	164.5419	23200	0.2757	62213064
0.0001	165.9626	23400	0.2784	62746840
0.0	167.3779	23600	0.2879	63279640
0.0001	168.7986	23800	0.2873	63817648
0.0	170.2139	24000	0.2914	64355456
0.0	171.6346	24200	0.2951	64891336
0.0	173.0499	24400	0.2955	65431304
0.0	174.4706	24600	0.2949	65971176
0.0001	175.8913	24800	0.3027	66508200
0.0001	177.3066	25000	0.3048	67044512
0.0	178.7273	25200	0.3058	67581248
0.0	180.1426	25400	0.3092	68116280
0.0	181.5633	25600	0.3119	68654016
0.0	182.9840	25800	0.3177	69191168
0.0	184.3993	26000	0.3174	69725736
0.0001	185.8200	26200	0.3201	70266432
0.0	187.2353	26400	0.3265	70795080
0.0	188.6560	26600	0.3255	71337664
0.0	190.0713	26800	0.3332	71873944
0.0001	191.4920	27000	0.3330	72406760
0.0	192.9127	27200	0.3379	72941856
0.0	194.3280	27400	0.3333	73486320
0.0	195.7487	27600	0.3379	74024784
0.0	197.1640	27800	0.3372	74562272
0.0	198.5847	28000	0.3413	75101016
0.0	200.0	28200	0.3469	75632576
0.0	201.4207	28400	0.3473	76166696
0.0	202.8414	28600	0.3513	76703192
0.0	204.2567	28800	0.3588	77237304
0.0	205.6774	29000	0.3607	77775808
0.0	207.0927	29200	0.3624	78304552
0.0	208.5134	29400	0.3567	78842312
0.0	209.9340	29600	0.3637	79379384
0.0	211.3494	29800	0.3648	79916200
0.0	212.7701	30000	0.3697	80450848
0.0	214.1854	30200	0.3757	80978696
0.0	215.6061	30400	0.3725	81517864
0.0	217.0214	30600	0.3748	82057360
0.0	218.4421	30800	0.3792	82601680
0.0	219.8627	31000	0.3769	83137640
0.0	221.2781	31200	0.3801	83674536
0.0	222.6988	31400	0.3842	84215064
0.0	224.1141	31600	0.3857	84750440
0.0	225.5348	31800	0.3825	85284976
0.0	226.9554	32000	0.3818	85820408
0.0	228.3708	32200	0.3894	86358288
0.0	229.7914	32400	0.3895	86896432
0.0	231.2068	32600	0.3825	87433496
0.0	232.6275	32800	0.3906	87969480
0.0	234.0428	33000	0.3918	88503984
0.0	235.4635	33200	0.3934	89043584
0.0	236.8841	33400	0.4044	89572896
0.0	238.2995	33600	0.3927	90114360
0.0	239.7201	33800	0.4034	90650032
0.0	241.1355	34000	0.4063	91178208
0.0	242.5561	34200	0.4017	91712168
0.0	243.9768	34400	0.4046	92253832
0.0	245.3922	34600	0.4086	92783304
0.0	246.8128	34800	0.4016	93323088
0.0	248.2282	35000	0.4019	93858448
0.0	249.6488	35200	0.4071	94391144
0.0	251.0642	35400	0.3990	94929608
0.0	252.4848	35600	0.4011	95474424
0.0	253.9055	35800	0.4070	96007792
0.0	255.3209	36000	0.3991	96546584
0.0	256.7415	36200	0.4101	97077888
0.0	258.1569	36400	0.3991	97612368
0.0	259.5775	36600	0.4082	98151616
0.0	260.9982	36800	0.4057	98684232
0.0	262.4135	37000	0.4145	99220560
0.0	263.8342	37200	0.4050	99758136
0.0	265.2496	37400	0.4118	100296152
0.0	266.6702	37600	0.4149	100836120
0.0	268.0856	37800	0.4066	101372264
0.0	269.5062	38000	0.4120	101912112
0.0	270.9269	38200	0.4087	102446016
0.0	272.3422	38400	0.4136	102980360
0.0	273.7629	38600	0.4182	103519296
0.0	275.1783	38800	0.4100	104053200
0.0	276.5989	39000	0.4106	104594720
0.0	278.0143	39200	0.4107	105126640
0.0	279.4349	39400	0.4083	105660640
0.0	280.8556	39600	0.4118	106198248
0.0	282.2709	39800	0.4026	106737720
0.0	283.6916	40000	0.4115	107274480

Framework versions

• PEFT 0.15.1
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1