train_rte_1744902664

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the rte dataset. It achieves the following results on the evaluation set:

• Loss: 0.0629
• Num Input Tokens Seen: 107274480

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• training_steps: 40000

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.0491	1.4207	200	0.0629	540280
0.005	2.8414	400	0.0702	1077480
0.0	4.2567	600	0.1548	1609584
0.0003	5.6774	800	0.1438	2150192
0.0	7.0927	1000	0.1761	2681640
0.0	8.5134	1200	0.2480	3218528
0.0	9.9340	1400	0.2674	3757240
0.0	11.3494	1600	0.2769	4292384
0.0	12.7701	1800	0.2819	4828992
0.0	14.1854	2000	0.2893	5364048
0.0	15.6061	2200	0.2929	5901512
0.0	17.0214	2400	0.2999	6435768
0.0	18.4421	2600	0.3084	6974976
0.0	19.8627	2800	0.3087	7509488
0.0	21.2781	3000	0.3147	8041736
0.0	22.6988	3200	0.3204	8583128
0.0	24.1141	3400	0.3240	9117488
0.0	25.5348	3600	0.3288	9649136
0.0	26.9554	3800	0.3316	10191288
0.0	28.3708	4000	0.3384	10724032
0.0	29.7914	4200	0.3431	11259816
0.0	31.2068	4400	0.3427	11805200
0.0	32.6275	4600	0.3491	12337832
0.0	34.0428	4800	0.3498	12874672
0.0	35.4635	5000	0.3507	13408200
0.0	36.8841	5200	0.3563	13943952
0.0	38.2995	5400	0.3576	14478600
0.0	39.7201	5600	0.3635	15021728
0.0	41.1355	5800	0.3641	15548872
0.0	42.5561	6000	0.3667	16082664
0.0	43.9768	6200	0.3691	16624832
0.0	45.3922	6400	0.3713	17152040
0.0	46.8128	6600	0.3752	17696104
0.0	48.2282	6800	0.3728	18228312
0.0	49.6488	7000	0.3765	18767376
0.0	51.0642	7200	0.3789	19300560
0.0	52.4848	7400	0.3763	19837208
0.0	53.9055	7600	0.3784	20381384
0.0	55.3209	7800	0.3799	20917960
0.0	56.7415	8000	0.3831	21456616
0.0	58.1569	8200	0.3817	21988808
0.0	59.5775	8400	0.3794	22526872
0.0	60.9982	8600	0.3789	23067872
0.0	62.4135	8800	0.3758	23599328
0.0	63.8342	9000	0.3784	24138832
0.0	65.2496	9200	0.3757	24675016
0.0	66.6702	9400	0.3758	25209352
0.0	68.0856	9600	0.3737	25745352
0.0	69.5062	9800	0.3698	26284824
0.0	70.9269	10000	0.3729	26824264
0.0	72.3422	10200	0.3695	27363992
0.0	73.7629	10400	0.3671	27904360
0.0	75.1783	10600	0.3651	28436064
0.0	76.5989	10800	0.3671	28976440
0.0	78.0143	11000	0.3629	29511840
0.0	79.4349	11200	0.3618	30049440
0.0	80.8556	11400	0.3605	30590008
0.0	82.2709	11600	0.3593	31127008
0.0	83.6916	11800	0.3576	31665584
0.0	85.1070	12000	0.3566	32199088
0.0	86.5276	12200	0.3567	32739240
0.0	87.9483	12400	0.3572	33281296
0.0	89.3636	12600	0.3584	33819016
0.0	90.7843	12800	0.3589	34356400
0.0	92.1996	13000	0.3587	34889896
0.0	93.6203	13200	0.3606	35429768
0.0	95.0357	13400	0.3597	35969976
0.0	96.4563	13600	0.3621	36505712
0.0	97.8770	13800	0.3600	37036976
0.0	99.2923	14000	0.3638	37570400
0.0	100.7130	14200	0.3658	38103616
0.0	102.1283	14400	0.3670	38636544
0.0	103.5490	14600	0.3708	39171560
0.0	104.9697	14800	0.3750	39706992
0.0	106.3850	15000	0.3711	40239280
0.0	107.8057	15200	0.3677	40778072
0.0	109.2210	15400	0.3723	41312720
0.0	110.6417	15600	0.3729	41845224
0.0	112.0570	15800	0.3721	42384256
0.0	113.4777	16000	0.3730	42925008
0.0	114.8984	16200	0.3745	43462528
0.0	116.3137	16400	0.3759	43999968
0.0	117.7344	16600	0.3769	44533664
0.0	119.1497	16800	0.3747	45067976
0.0	120.5704	17000	0.3742	45610752
0.0	121.9911	17200	0.3737	46147416
0.0	123.4064	17400	0.3732	46682792
0.0	124.8271	17600	0.3719	47218688
0.0	126.2424	17800	0.3741	47751176
0.0	127.6631	18000	0.3715	48286872
0.0	129.0784	18200	0.3650	48824840
0.0	130.4991	18400	0.3678	49361064
0.0	131.9198	18600	0.3643	49893616
0.0	133.3351	18800	0.3651	50425120
0.0	134.7558	19000	0.3632	50963088
0.0	136.1711	19200	0.3625	51496048
0.0	137.5918	19400	0.3600	52038608
0.0	139.0071	19600	0.3624	52575544
0.0	140.4278	19800	0.3606	53114912
0.0	141.8485	20000	0.3608	53657368
0.0	143.2638	20200	0.3562	54195776
0.0	144.6845	20400	0.3570	54722232
0.0	146.0998	20600	0.3533	55255168
0.0	147.5205	20800	0.3561	55786616
0.0	148.9412	21000	0.3573	56322200
0.0	150.3565	21200	0.3531	56860136
0.0	151.7772	21400	0.3527	57396560
0.0	153.1925	21600	0.3501	57930904
0.0	154.6132	21800	0.3533	58469832
0.0	156.0285	22000	0.3507	59001744
0.0	157.4492	22200	0.3516	59542632
0.0	158.8699	22400	0.3509	60077280
0.0	160.2852	22600	0.3489	60614824
0.0	161.7059	22800	0.3503	61145384
0.0	163.1212	23000	0.3426	61678824
0.0	164.5419	23200	0.3506	62213064
0.0	165.9626	23400	0.3466	62746840
0.0	167.3779	23600	0.3469	63279640
0.0	168.7986	23800	0.3495	63817648
0.0	170.2139	24000	0.3469	64355456
0.0	171.6346	24200	0.3439	64891336
0.0	173.0499	24400	0.3463	65431304
0.0	174.4706	24600	0.3455	65971176
0.0	175.8913	24800	0.3466	66508200
0.0	177.3066	25000	0.3473	67044512
0.0	178.7273	25200	0.3446	67581248
0.0	180.1426	25400	0.3438	68116280
0.0	181.5633	25600	0.3425	68654016
0.0	182.9840	25800	0.3481	69191168
0.0	184.3993	26000	0.3445	69725736
0.0	185.8200	26200	0.3436	70266432
0.0	187.2353	26400	0.3460	70795080
0.0	188.6560	26600	0.3451	71337664
0.0	190.0713	26800	0.3443	71873944
0.0	191.4920	27000	0.3494	72406760
0.0	192.9127	27200	0.3485	72941856
0.0	194.3280	27400	0.3499	73486320
0.0	195.7487	27600	0.3473	74024784
0.0	197.1640	27800	0.3461	74562272
0.0	198.5847	28000	0.3526	75101016
0.0	200.0	28200	0.3558	75632576
0.0	201.4207	28400	0.3512	76166696
0.0	202.8414	28600	0.3511	76703192
0.0	204.2567	28800	0.3513	77237304
0.0	205.6774	29000	0.3518	77775808
0.0	207.0927	29200	0.3507	78304552
0.0	208.5134	29400	0.3526	78842312
0.0	209.9340	29600	0.3501	79379384
0.0	211.3494	29800	0.3551	79916200
0.0	212.7701	30000	0.3518	80450848
0.0	214.1854	30200	0.3548	80978696
0.0	215.6061	30400	0.3560	81517864
0.0	217.0214	30600	0.3535	82057360
0.0	218.4421	30800	0.3534	82601680
0.0	219.8627	31000	0.3578	83137640
0.0	221.2781	31200	0.3527	83674536
0.0	222.6988	31400	0.3554	84215064
0.0	224.1141	31600	0.3615	84750440
0.0	225.5348	31800	0.3553	85284976
0.0	226.9554	32000	0.3554	85820408
0.0	228.3708	32200	0.3597	86358288
0.0	229.7914	32400	0.3616	86896432
0.0	231.2068	32600	0.3626	87433496
0.0	232.6275	32800	0.3620	87969480
0.0	234.0428	33000	0.3595	88503984
0.0	235.4635	33200	0.3627	89043584
0.0	236.8841	33400	0.3633	89572896
0.0	238.2995	33600	0.3604	90114360
0.0	239.7201	33800	0.3580	90650032
0.0	241.1355	34000	0.3675	91178208
0.0	242.5561	34200	0.3640	91712168
0.0	243.9768	34400	0.3642	92253832
0.0	245.3922	34600	0.3655	92783304
0.0	246.8128	34800	0.3619	93323088
0.0	248.2282	35000	0.3633	93858448
0.0	249.6488	35200	0.3628	94391144
0.0	251.0642	35400	0.3638	94929608
0.0	252.4848	35600	0.3602	95474424
0.0	253.9055	35800	0.3663	96007792
0.0	255.3209	36000	0.3675	96546584
0.0	256.7415	36200	0.3670	97077888
0.0	258.1569	36400	0.3660	97612368
0.0	259.5775	36600	0.3692	98151616
0.0	260.9982	36800	0.3672	98684232
0.0	262.4135	37000	0.3655	99220560
0.0	263.8342	37200	0.3661	99758136
0.0	265.2496	37400	0.3704	100296152
0.0	266.6702	37600	0.3708	100836120
0.0	268.0856	37800	0.3688	101372264
0.0	269.5062	38000	0.3680	101912112
0.0	270.9269	38200	0.3662	102446016
0.0	272.3422	38400	0.3695	102980360
0.0	273.7629	38600	0.3647	103519296
0.0	275.1783	38800	0.3698	104053200
0.0	276.5989	39000	0.3675	104594720
0.0	278.0143	39200	0.3663	105126640
0.0	279.4349	39400	0.3677	105660640
0.0	280.8556	39600	0.3691	106198248
0.0	282.2709	39800	0.3651	106737720
0.0	283.6916	40000	0.3661	107274480

Framework versions

• PEFT 0.15.1
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1