tiny_bert_bc_km_5_v1

This model is a fine-tuned version of on the Hartunka/processed_book_corpus-km-5 dataset. It achieves the following results on the evaluation set:

• Loss: 1.5993
• Accuracy: 0.6797

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 96
• eval_batch_size: 96
• seed: 10
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 10000
• num_epochs: 25

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
5.868	0.4215	10000	5.6390	0.1587
5.757	0.8431	20000	5.5611	0.1631
5.6406	1.2646	30000	5.4171	0.1782
3.3457	1.6861	40000	2.9074	0.4865
2.8544	2.1077	50000	2.4903	0.5423
2.5821	2.5292	60000	2.2188	0.5786
2.4361	2.9507	70000	2.0899	0.5988
2.3431	3.3723	80000	2.0070	0.6118
2.2752	3.7938	90000	1.9472	0.6216
2.2181	4.2153	100000	1.9012	0.6284
2.1874	4.6369	110000	1.8619	0.6334
2.1438	5.0584	120000	1.8210	0.6391
2.117	5.4799	130000	1.7991	0.6428
2.1013	5.9014	140000	1.7722	0.6463
2.0674	6.3230	150000	1.7556	0.6491
2.0597	6.7445	160000	1.7411	0.6516
2.0254	7.1660	170000	1.7298	0.6539
2.0177	7.5876	180000	1.7182	0.6553
2.0039	8.0091	190000	1.7304	0.6571
1.9926	8.4306	200000	1.6948	0.6593
1.9789	8.8522	210000	1.6922	0.6603
1.9682	9.2737	220000	1.6792	0.6616
1.9548	9.6952	230000	1.6715	0.6629
1.9364	10.1168	240000	1.6794	0.6642
1.9411	10.5383	250000	1.6578	0.6653
1.9335	10.9598	260000	1.6482	0.6665
1.9191	11.3814	270000	1.6511	0.6670
1.9194	11.8029	280000	1.6416	0.6685
1.9004	12.2244	290000	1.6558	0.6690
1.8987	12.6460	300000	1.6351	0.6699
1.8838	13.0675	310000	1.6363	0.6708
1.8851	13.4890	320000	1.6329	0.6713
1.8829	13.9106	330000	1.6368	0.6720
1.8755	14.3321	340000	1.6218	0.6730
1.8662	14.7536	350000	1.6259	0.6733
1.8555	15.1751	360000	1.6301	0.6739
1.8581	15.5967	370000	1.6150	0.6748
1.8464	16.0182	380000	1.6229	0.6753
1.8441	16.4397	390000	1.6136	0.6757
1.8378	16.8613	400000	1.6066	0.6761
1.8364	17.2828	410000	1.6130	0.6770
1.8323	17.7043	420000	1.6023	0.6776
1.8194	18.1259	430000	1.6100	0.6777
1.8169	18.5474	440000	1.6132	0.6779
1.8174	18.9689	450000	1.6051	0.6789
1.8105	19.3905	460000	1.6220	0.6791
1.8115	19.8120	470000	1.5991	0.6795
1.8015	20.2335	480000	1.6119	0.6801
1.7975	20.6551	490000	1.6027	0.6804
1.7918	21.0766	500000	1.6094	0.6808
1.7896	21.4981	510000	1.6055	0.6812
1.7885	21.9197	520000	1.6038	0.6814
1.7882	22.3412	530000	1.6068	0.6816
1.7826	22.7627	540000	1.6105	0.6819
1.7816	23.1843	550000	1.6100	0.6822
1.7788	23.6058	560000	1.6041	0.6828
1.772	24.0273	570000	1.6118	0.6827
1.7736	24.4488	580000	1.6020	0.6835
1.7698	24.8704	590000	1.6105	0.6830

Framework versions

• Transformers 4.40.0
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.19.1