train_cb_1745950317

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the cb dataset. It achieves the following results on the evaluation set:

• Loss: 0.3400
• Num Input Tokens Seen: 23078128

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 123
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• training_steps: 40000

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.4422	3.5133	200	0.4803	116248
0.417	7.0177	400	0.3748	232144
0.0922	10.5310	600	0.3435	346496
0.2681	14.0354	800	0.3400	462696
0.0903	17.5487	1000	0.3539	578728
0.0956	21.0531	1200	0.3442	692976
0.0069	24.5664	1400	0.3744	809080
0.0357	28.0708	1600	0.3671	924048
0.0052	31.5841	1800	0.3766	1040096
0.0061	35.0885	2000	0.4047	1155784
0.0027	38.6018	2200	0.4160	1271880
0.0018	42.1062	2400	0.4212	1386392
0.0034	45.6195	2600	0.4522	1502448
0.0016	49.1239	2800	0.4501	1616928
0.0009	52.6372	3000	0.4757	1732240
0.0016	56.1416	3200	0.4546	1847880
0.0009	59.6549	3400	0.4640	1963376
0.0002	63.1593	3600	0.4879	2078344
0.0005	66.6726	3800	0.4897	2193696
0.0004	70.1770	4000	0.5064	2309024
0.0007	73.6903	4200	0.5030	2425544
0.0006	77.1947	4400	0.5300	2539944
0.0002	80.7080	4600	0.5184	2655720
0.0002	84.2124	4800	0.5348	2771904
0.0002	87.7257	5000	0.5421	2887856
0.0002	91.2301	5200	0.5307	3003888
0.0003	94.7434	5400	0.5369	3118800
0.0002	98.2478	5600	0.5511	3234376
0.0002	101.7611	5800	0.5583	3350608
0.0002	105.2655	6000	0.5621	3466256
0.0003	108.7788	6200	0.5678	3582008
0.0001	112.2832	6400	0.5698	3696904
0.0002	115.7965	6600	0.5619	3812728
0.0001	119.3009	6800	0.5994	3927256
0.0001	122.8142	7000	0.6037	4043128
0.0002	126.3186	7200	0.5946	4158920
0.0001	129.8319	7400	0.6031	4274536
0.0001	133.3363	7600	0.6267	4389864
0.0001	136.8496	7800	0.6281	4505192
0.0001	140.3540	8000	0.6321	4620656
0.0001	143.8673	8200	0.6333	4736960
0.0001	147.3717	8400	0.6338	4850688
0.0	150.8850	8600	0.6369	4965800
0.0	154.3894	8800	0.6387	5082848
0.0001	157.9027	9000	0.6416	5197896
0.0001	161.4071	9200	0.6640	5312976
0.0	164.9204	9400	0.6399	5428816
0.0	168.4248	9600	0.6467	5542632
0.0	171.9381	9800	0.6772	5660064
0.0	175.4425	10000	0.6818	5775432
0.0	178.9558	10200	0.6635	5891480
0.0	182.4602	10400	0.6624	6006016
0.0	185.9735	10600	0.6576	6121200
0.0	189.4779	10800	0.6816	6236696
0.0	192.9912	11000	0.6779	6352152
0.0	196.4956	11200	0.6700	6467792
0.0	200.0	11400	0.6890	6581880
0.0	203.5133	11600	0.6884	6697328
0.0	207.0177	11800	0.7128	6811792
0.0	210.5310	12000	0.6922	6928248
0.0	214.0354	12200	0.7340	7043832
0.0	217.5487	12400	0.7419	7157984
0.0	221.0531	12600	0.7471	7274032
0.0	224.5664	12800	0.7260	7390136
0.0	228.0708	13000	0.7219	7505120
0.0	231.5841	13200	0.7331	7619616
0.0	235.0885	13400	0.7320	7736064
0.0	238.6018	13600	0.7455	7850792
0.0	242.1062	13800	0.7547	7965808
0.0	245.6195	14000	0.7392	8081552
0.0	249.1239	14200	0.7261	8197208
0.0	252.6372	14400	0.7496	8312272
0.0	256.1416	14600	0.7355	8426888
0.0	259.6549	14800	0.7620	8542448
0.0	263.1593	15000	0.7750	8658448
0.0	266.6726	15200	0.7526	8773608
0.0	270.1770	15400	0.7705	8887928
0.0	273.6903	15600	0.7543	9004600
0.0	277.1947	15800	0.7446	9119624
0.0	280.7080	16000	0.7641	9233904
0.0	284.2124	16200	0.7727	9351032
0.0	287.7257	16400	0.7616	9465944
0.0	291.2301	16600	0.7777	9581568
0.0	294.7434	16800	0.7768	9696576
0.0	298.2478	17000	0.7894	9811496
0.0	301.7611	17200	0.8158	9926600
0.0	305.2655	17400	0.7808	10042072
0.0	308.7788	17600	0.7879	10156616
0.0	312.2832	17800	0.7923	10272688
0.0	315.7965	18000	0.8144	10386824
0.0	319.3009	18200	0.7970	10502040
0.0	322.8142	18400	0.7929	10617608
0.0	326.3186	18600	0.8050	10731768
0.0	329.8319	18800	0.7777	10848480
0.0	333.3363	19000	0.8078	10963328
0.0	336.8496	19200	0.7839	11078712
0.0	340.3540	19400	0.8035	11193832
0.0	343.8673	19600	0.8068	11309368
0.0	347.3717	19800	0.8348	11424912
0.0	350.8850	20000	0.7916	11539864
0.0	354.3894	20200	0.8303	11654632
0.0	357.9027	20400	0.8251	11771008
0.0	361.4071	20600	0.8041	11886608
0.0	364.9204	20800	0.8056	12002608
0.0	368.4248	21000	0.8178	12117448
0.0	371.9381	21200	0.8268	12233152
0.0	375.4425	21400	0.8350	12346784
0.0	378.9558	21600	0.8515	12463336
0.0	382.4602	21800	0.8257	12578616
0.0	385.9735	22000	0.8088	12693160
0.0	389.4779	22200	0.8599	12808696
0.0	392.9912	22400	0.9067	12924056
0.0	396.4956	22600	0.8353	13039656
0.0	400.0	22800	0.8269	13154552
0.0	403.5133	23000	0.8564	13269320
0.0	407.0177	23200	0.8489	13385512
0.0	410.5310	23400	0.8479	13501208
0.0	414.0354	23600	0.8427	13617048
0.0	417.5487	23800	0.8487	13733448
0.0	421.0531	24000	0.8331	13848288
0.0	424.5664	24200	0.8755	13963536
0.0	428.0708	24400	0.8666	14080024
0.0	431.5841	24600	0.8540	14194520
0.0	435.0885	24800	0.8528	14310080
0.0	438.6018	25000	0.8280	14427448
0.0	442.1062	25200	0.8015	14542448
0.0	445.6195	25400	0.8213	14657640
0.0	449.1239	25600	0.8155	14772328
0.0	452.6372	25800	0.8089	14888712
0.0	456.1416	26000	0.7789	15002944
0.0	459.6549	26200	0.8078	15118544
0.0	463.1593	26400	0.7963	15234184
0.0	466.6726	26600	0.8154	15349544
0.0	470.1770	26800	0.8179	15465448
0.0	473.6903	27000	0.8571	15581752
0.0	477.1947	27200	0.8176	15696720
0.0	480.7080	27400	0.8097	15812864
0.0	484.2124	27600	0.8679	15928512
0.0	487.7257	27800	0.8476	16043264
0.0	491.2301	28000	0.8072	16158992
0.0	494.7434	28200	0.8343	16274040
0.0	498.2478	28400	0.8759	16389944
0.0	501.7611	28600	0.8398	16506208
0.0	505.2655	28800	0.8493	16621272
0.0	508.7788	29000	0.8266	16737072
0.0	512.2832	29200	0.8220	16852312
0.0	515.7965	29400	0.7940	16967744
0.0	519.3009	29600	0.8374	17083368
0.0	522.8142	29800	0.8460	17197984
0.0	526.3186	30000	0.8089	17314032
0.0	529.8319	30200	0.8425	17428904
0.0	533.3363	30400	0.8380	17543048
0.0	536.8496	30600	0.8113	17659880
0.0	540.3540	30800	0.8418	17773728
0.0	543.8673	31000	0.7708	17889344
0.0	547.3717	31200	0.8254	18005392
0.0	550.8850	31400	0.8248	18120296
0.0	554.3894	31600	0.8140	18235552
0.0	557.9027	31800	0.8168	18352024
0.0	561.4071	32000	0.8280	18466080
0.0	564.9204	32200	0.8156	18581584
0.0	568.4248	32400	0.7841	18697408
0.0	571.9381	32600	0.7724	18811608
0.0	575.4425	32800	0.8385	18927640
0.0	578.9558	33000	0.7809	19043672
0.0	582.4602	33200	0.7646	19157776
0.0	585.9735	33400	0.8207	19272744
0.0	589.4779	33600	0.8416	19388520
0.0	592.9912	33800	0.7581	19504472
0.0	596.4956	34000	0.8201	19618408
0.0	600.0	34200	0.8070	19734128
0.0	603.5133	34400	0.7923	19849608
0.0	607.0177	34600	0.8245	19964704
0.0	610.5310	34800	0.8121	20080968
0.0	614.0354	35000	0.8001	20195624
0.0	617.5487	35200	0.8197	20311640
0.0	621.0531	35400	0.8002	20426832
0.0	624.5664	35600	0.7819	20541816
0.0	628.0708	35800	0.7758	20656416
0.0	631.5841	36000	0.7611	20771136
0.0	635.0885	36200	0.7788	20886272
0.0	638.6018	36400	0.8212	21001560
0.0	642.1062	36600	0.8321	21115320
0.0	645.6195	36800	0.8022	21230216
0.0	649.1239	37000	0.7443	21344656
0.0	652.6372	37200	0.8040	21461664
0.0	656.1416	37400	0.7712	21576216
0.0	659.6549	37600	0.8044	21692088
0.0	663.1593	37800	0.7838	21807184
0.0	666.6726	38000	0.7712	21923192
0.0	670.1770	38200	0.7939	22037928
0.0	673.6903	38400	0.7669	22153968
0.0	677.1947	38600	0.7145	22269648
0.0	680.7080	38800	0.7588	22385640
0.0	684.2124	39000	0.7613	22502040
0.0	687.7257	39200	0.7583	22616408
0.0	691.2301	39400	0.7583	22732496
0.0	694.7434	39600	0.7583	22846704
0.0	698.2478	39800	0.7583	22962016
0.0	701.7611	40000	0.7583	23078128

Framework versions

• PEFT 0.15.2.dev0
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1