tapt_base_LR-2e-05

This model is a fine-tuned version of bioformers/bioformer-16L on the Mardiyyah/TAPT_data_V2_split dataset. It achieves the following results on the evaluation set:

• Loss: 1.9182

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 3407
• gradient_accumulation_steps: 16
• total_train_batch_size: 1024
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.1046	1	2.2216
No log	0.2092	2	2.1888
No log	0.3137	3	2.1733
No log	0.4183	4	2.1297
No log	0.5229	5	2.1922
No log	0.6275	6	2.1468
No log	0.7320	7	2.1433
No log	0.8366	8	2.0927
No log	0.9412	9	2.1243
2.4847	1.1046	10	2.1267
2.4847	1.2092	11	2.0820
2.4847	1.3137	12	2.0738
2.4847	1.4183	13	2.0427
2.4847	1.5229	14	2.0568
2.4847	1.6275	15	2.0835
2.4847	1.7320	16	2.0789
2.4847	1.8366	17	2.0674
2.4847	1.9412	18	2.0401
2.4101	2.1046	19	2.0421
2.4101	2.2092	20	2.0762
2.4101	2.3137	21	2.0065
2.4101	2.4183	22	2.0763
2.4101	2.5229	23	2.0424
2.4101	2.6275	24	2.0310
2.4101	2.7320	25	2.0874
2.4101	2.8366	26	2.0235
2.4101	2.9412	27	2.0597
2.3677	3.1046	28	1.9865
2.3677	3.2092	29	2.0295
2.3677	3.3137	30	2.0296
2.3677	3.4183	31	2.0019
2.3677	3.5229	32	1.9696
2.3677	3.6275	33	2.0265
2.3677	3.7320	34	2.0107
2.3677	3.8366	35	2.0344
2.3677	3.9412	36	2.0281
2.2639	4.1046	37	2.0171
2.2639	4.2092	38	2.0344
2.2639	4.3137	39	1.9914
2.2639	4.4183	40	1.9856
2.2639	4.5229	41	2.0357
2.2639	4.6275	42	2.0289
2.2639	4.7320	43	1.9714
2.2639	4.8366	44	1.9895
2.2639	4.9412	45	1.9905
2.2037	5.1046	46	1.9589
2.2037	5.2092	47	1.9865
2.2037	5.3137	48	2.0114
2.2037	5.4183	49	2.0008
2.2037	5.5229	50	1.9578
2.2037	5.6275	51	2.0294
2.2037	5.7320	52	1.9585
2.2037	5.8366	53	1.9783
2.2037	5.9412	54	1.9880
2.16	6.1046	55	2.0060
2.16	6.2092	56	1.9558
2.16	6.3137	57	1.9664
2.16	6.4183	58	1.9201
2.16	6.5229	59	1.9816
2.16	6.6275	60	1.9682
2.16	6.7320	61	1.9605
2.16	6.8366	62	1.9233
2.16	6.9412	63	1.9687
2.1108	7.1046	64	1.9987
2.1108	7.2092	65	2.0023
2.1108	7.3137	66	1.9627
2.1108	7.4183	67	2.0215
2.1108	7.5229	68	1.9613
2.1108	7.6275	69	2.0261
2.1108	7.7320	70	1.9626
2.1108	7.8366	71	2.0007
2.1108	7.9412	72	1.9404
2.0949	8.1046	73	1.9943
2.0949	8.2092	74	2.0443
2.0949	8.3137	75	1.9909
2.0949	8.4183	76	1.9790
2.0949	8.5229	77	1.9505
2.0949	8.6275	78	1.9477
2.0949	8.7320	79	2.0272
2.0949	8.8366	80	1.9549
2.0949	8.9412	81	1.9641
2.0617	9.1046	82	1.9859
2.0617	9.2092	83	1.9376
2.0617	9.3137	84	1.9699
2.0617	9.4183	85	1.9334
2.0617	9.5229	86	1.9708
2.0617	9.6275	87	1.9700
2.0617	9.7320	88	1.9634
2.0617	9.8366	89	1.9220
2.0617	9.9412	90	1.9669
2.0509	10.1046	91	1.9568
2.0509	10.2092	92	1.9699
2.0509	10.3137	93	2.0316
2.0509	10.4183	94	1.9130
2.0509	10.5229	95	1.9707
2.0509	10.6275	96	1.9624
2.0509	10.7320	97	1.9516
2.0509	10.8366	98	1.9508
2.0509	10.9412	99	1.9166
1.9835	11.1046	100	1.9469
1.9835	11.2092	101	1.9620
1.9835	11.3137	102	1.9470
1.9835	11.4183	103	1.9458
1.9835	11.5229	104	1.9585
1.9835	11.6275	105	1.9451
1.9835	11.7320	106	1.9203
1.9835	11.8366	107	1.9323
1.9835	11.9412	108	1.9641
1.9719	12.1046	109	1.9262
1.9719	12.2092	110	1.9800
1.9719	12.3137	111	1.9422
1.9719	12.4183	112	1.9286
1.9719	12.5229	113	1.9934
1.9719	12.6275	114	1.9704
1.9719	12.7320	115	1.9390
1.9719	12.8366	116	1.9161
1.9719	12.9412	117	1.9483
1.9663	13.1046	118	1.9584
1.9663	13.2092	119	1.9642
1.9663	13.3137	120	1.9447
1.9663	13.4183	121	2.0014
1.9663	13.5229	122	1.8806
1.9663	13.6275	123	1.9487
1.9663	13.7320	124	1.9181
1.9663	13.8366	125	1.9238
1.9663	13.9412	126	1.9514
1.9785	14.1046	127	1.9426
1.9785	14.2092	128	1.9766
1.9785	14.3137	129	1.9118
1.9785	14.4183	130	1.9367
1.9785	14.5229	131	1.9372
1.9785	14.6275	132	1.9232
1.9785	14.7320	133	1.9999
1.9785	14.8366	134	1.9355
1.9785	14.9412	135	1.9657
1.9329	15.1046	136	1.9451
1.9329	15.2092	137	1.9597
1.9329	15.3137	138	1.9180
1.9329	15.4183	139	1.9344
1.9329	15.5229	140	1.9772
1.9329	15.6275	141	1.9797
1.9329	15.7320	142	1.9061
1.9329	15.8366	143	1.8886
1.9329	15.9412	144	1.9685
1.9144	16.1046	145	1.9798
1.9144	16.2092	146	1.9588
1.9144	16.3137	147	1.9274
1.9144	16.4183	148	1.9590
1.9144	16.5229	149	1.9553
1.9144	16.6275	150	1.9143
1.9144	16.7320	151	1.9269
1.9144	16.8366	152	1.9654
1.9144	16.9412	153	1.9789
1.9103	17.1046	154	1.9569
1.9103	17.2092	155	1.9652
1.9103	17.3137	156	1.9810
1.9103	17.4183	157	1.9285
1.9103	17.5229	158	1.9378
1.9103	17.6275	159	1.9520
1.9103	17.7320	160	1.9782
1.9103	17.8366	161	1.9681
1.9103	17.9412	162	1.8926
1.887	18.1046	163	1.9334
1.887	18.2092	164	1.9252
1.887	18.3137	165	1.9399
1.887	18.4183	166	1.9518
1.887	18.5229	167	1.9924
1.887	18.6275	168	1.9054
1.887	18.7320	169	1.9480
1.887	18.8366	170	1.9308
1.887	18.9412	171	1.9343
1.8644	19.1046	172	1.9861
1.8644	19.2092	173	1.9453
1.8644	19.3137	174	1.8999
1.8644	19.4183	175	1.9309
1.8644	19.5229	176	1.9544
1.8644	19.6275	177	1.9436
1.8644	19.7320	178	1.9165
1.8644	19.8366	179	1.9696
1.8644	19.9412	180	1.9248
1.8687	20.1046	181	1.9517
1.8687	20.2092	182	1.9042
1.8687	20.3137	183	1.9925
1.8687	20.4183	184	1.8843
1.8687	20.5229	185	1.9794
1.8687	20.6275	186	1.9789
1.8687	20.7320	187	1.9192
1.8687	20.8366	188	1.9174
1.8687	20.9412	189	1.9568
1.8361	21.1046	190	1.9128
1.8361	21.2092	191	1.9429
1.8361	21.3137	192	1.9558
1.8361	21.4183	193	1.9128
1.8361	21.5229	194	1.9589
1.8361	21.6275	195	1.9745
1.8361	21.7320	196	1.9994
1.8361	21.8366	197	1.9594
1.8361	21.9412	198	1.9064
1.8461	22.1046	199	1.9475
1.8461	22.2092	200	1.9638
1.8461	22.3137	201	1.9351
1.8461	22.4183	202	1.9184
1.8461	22.5229	203	1.9657
1.8461	22.6275	204	1.9109
1.8461	22.7320	205	1.9320
1.8461	22.8366	206	1.9680
1.8461	22.9412	207	1.9629
1.8246	23.1046	208	1.9430
1.8246	23.2092	209	1.9262
1.8246	23.3137	210	1.9615
1.8246	23.4183	211	1.9560
1.8246	23.5229	212	1.9661
1.8246	23.6275	213	1.9781
1.8246	23.7320	214	1.9806
1.8246	23.8366	215	1.9735
1.8246	23.9412	216	1.9583
1.8181	24.1046	217	1.9555
1.8181	24.2092	218	1.9165
1.8181	24.3137	219	1.9638
1.8181	24.4183	220	2.0008
1.8181	24.5229	221	1.9247
1.8181	24.6275	222	1.9720
1.8181	24.7320	223	2.0084
1.8181	24.8366	224	1.9424
1.8181	24.9412	225	1.9111
1.797	25.1046	226	1.9787
1.797	25.2092	227	1.9613
1.797	25.3137	228	1.8806
1.797	25.4183	229	1.9231
1.797	25.5229	230	1.9022
1.797	25.6275	231	1.9683
1.797	25.7320	232	1.9825
1.797	25.8366	233	1.9629
1.797	25.9412	234	1.9116
1.7749	26.1046	235	1.9700
1.7749	26.2092	236	1.9812
1.7749	26.3137	237	1.9249
1.7749	26.4183	238	1.9684
1.7749	26.5229	239	1.9605
1.7749	26.6275	240	1.8918
1.7749	26.7320	241	1.9443
1.7749	26.8366	242	1.9148
1.7749	26.9412	243	1.8974
1.8022	27.1046	244	1.9712
1.8022	27.2092	245	1.9719
1.8022	27.3137	246	1.9540
1.8022	27.4183	247	1.8907
1.8022	27.5229	248	1.9908
1.8022	27.6275	249	1.9274
1.8022	27.7320	250	1.9234
1.8022	27.8366	251	1.9581
1.8022	27.9412	252	1.9409
1.7879	28.1046	253	1.8716
1.7879	28.2092	254	1.9945
1.7879	28.3137	255	1.8658
1.7879	28.4183	256	1.9468
1.7879	28.5229	257	1.9457
1.7879	28.6275	258	1.9555
1.7879	28.7320	259	1.9545
1.7879	28.8366	260	1.9226
1.7879	28.9412	261	1.9331
1.8019	29.1046	262	1.9786
1.8019	29.2092	263	1.9768
1.8019	29.3137	264	1.9601
1.8019	29.4183	265	1.9172
1.8019	29.5229	266	1.9222
1.8019	29.6275	267	1.9184
1.8019	29.7320	268	1.8822
1.8019	29.8366	269	1.9162
1.8019	29.9412	270	1.9770
1.7614	30.1046	271	1.9033
1.7614	30.2092	272	1.9455
1.7614	30.3137	273	1.9107
1.7614	30.4183	274	1.9813
1.7614	30.5229	275	1.9427
1.7614	30.6275	276	1.9499
1.7614	30.7320	277	1.9612
1.7614	30.8366	278	1.9451
1.7614	30.9412	279	1.9133
1.7619	31.1046	280	1.9205
1.7619	31.2092	281	1.9468
1.7619	31.3137	282	1.9465
1.7619	31.4183	283	1.8833
1.7619	31.5229	284	1.9414
1.7619	31.6275	285	1.9486
1.7619	31.7320	286	1.9185
1.7619	31.8366	287	1.9519
1.7619	31.9412	288	1.9386
1.7713	32.1046	289	1.8967
1.7713	32.2092	290	1.9650
1.7713	32.3137	291	1.9199
1.7713	32.4183	292	1.9147
1.7713	32.5229	293	1.9160
1.7713	32.6275	294	1.9954
1.7713	32.7320	295	1.9218
1.7713	32.8366	296	1.9787
1.7713	32.9412	297	1.9362
1.7635	33.1046	298	1.9281
1.7635	33.2092	299	1.9211
1.7635	33.3137	300	1.9681
1.7635	33.4183	301	1.9094
1.7635	33.5229	302	1.9846
1.7635	33.6275	303	1.9461
1.7635	33.7320	304	1.8948
1.7635	33.8366	305	1.9371
1.7635	33.9412	306	2.0065
1.7394	34.1046	307	1.9282
1.7394	34.2092	308	1.9413
1.7394	34.3137	309	1.9884
1.7394	34.4183	310	1.9174
1.7394	34.5229	311	1.9595
1.7394	34.6275	312	1.9424
1.7394	34.7320	313	1.9495
1.7394	34.8366	314	1.9161
1.7394	34.9412	315	1.9632
1.7434	35.1046	316	1.9130
1.7434	35.2092	317	1.9850
1.7434	35.3137	318	1.9291
1.7434	35.4183	319	1.9300
1.7434	35.5229	320	1.9400
1.7434	35.6275	321	1.9736
1.7434	35.7320	322	1.9033
1.7434	35.8366	323	1.9249
1.7434	35.9412	324	1.9796
1.7578	36.1046	325	1.9596
1.7578	36.2092	326	1.9294
1.7578	36.3137	327	1.9572
1.7578	36.4183	328	1.9537
1.7578	36.5229	329	1.9745
1.7578	36.6275	330	1.9568
1.7578	36.7320	331	1.9689
1.7578	36.8366	332	1.9140
1.7578	36.9412	333	1.9298
1.7497	37.1046	334	1.9699
1.7497	37.2092	335	1.9077
1.7497	37.3137	336	1.9559
1.7497	37.4183	337	1.9622
1.7497	37.5229	338	1.9239
1.7497	37.6275	339	1.9739
1.7497	37.7320	340	1.9366
1.7497	37.8366	341	1.9858
1.7497	37.9412	342	1.9603
1.7378	38.1046	343	1.9392
1.7378	38.2092	344	1.9555
1.7378	38.3137	345	1.9803
1.7378	38.4183	346	1.9502
1.7378	38.5229	347	1.9591
1.7378	38.6275	348	1.9583
1.7378	38.7320	349	1.9507
1.7378	38.8366	350	1.9411
1.7378	38.9412	351	1.9221
1.7324	39.1046	352	1.9468
1.7324	39.2092	353	1.9370
1.7324	39.3137	354	1.9278
1.7324	39.4183	355	1.9604
1.7324	39.5229	356	1.9376
1.7324	39.6275	357	1.9473
1.7324	39.7320	358	1.9490
1.7324	39.8366	359	1.9134
1.7324	39.9412	360	1.9323
1.7195	40.1046	361	1.9119
1.7195	40.2092	362	1.9394
1.7195	40.3137	363	1.9960
1.7195	40.4183	364	1.9789
1.7195	40.5229	365	1.9750
1.7195	40.6275	366	1.9399
1.7195	40.7320	367	1.9516
1.7195	40.8366	368	1.9410
1.7195	40.9412	369	1.9318
1.7043	41.1046	370	1.9890
1.7043	41.2092	371	1.9841
1.7043	41.3137	372	1.9188
1.7043	41.4183	373	1.9615
1.7043	41.5229	374	1.9061
1.7043	41.6275	375	1.9100
1.7043	41.7320	376	1.9422
1.7043	41.8366	377	1.9640
1.7043	41.9412	378	1.9177
1.7169	42.1046	379	1.9164
1.7169	42.2092	380	1.9375
1.7169	42.3137	381	1.9526
1.7169	42.4183	382	1.9388
1.7169	42.5229	383	1.9379
1.7169	42.6275	384	1.9315
1.7169	42.7320	385	1.9418
1.7169	42.8366	386	1.9460
1.7169	42.9412	387	1.9130
1.7315	43.1046	388	1.9540
1.7315	43.2092	389	1.9513
1.7315	43.3137	390	2.0079
1.7315	43.4183	391	1.9754
1.7315	43.5229	392	1.9725
1.7315	43.6275	393	1.9601
1.7315	43.7320	394	1.9266
1.7315	43.8366	395	1.9546
1.7315	43.9412	396	1.9186
1.7095	44.1046	397	1.9545
1.7095	44.2092	398	2.0219
1.7095	44.3137	399	1.9499
1.7095	44.4183	400	1.9414
1.7095	44.5229	401	1.9617
1.7095	44.6275	402	1.9940
1.7095	44.7320	403	1.9617
1.7095	44.8366	404	1.9692
1.7095	44.9412	405	1.9219
1.7071	45.1046	406	1.9611
1.7071	45.2092	407	1.9779
1.7071	45.3137	408	1.9238
1.7071	45.4183	409	1.9090
1.7071	45.5229	410	1.9342
1.7071	45.6275	411	1.9936
1.7071	45.7320	412	1.8978
1.7071	45.8366	413	1.9208
1.7071	45.9412	414	1.9177
1.7116	46.1046	415	1.9579
1.7116	46.2092	416	1.9422
1.7116	46.3137	417	1.9287
1.7116	46.4183	418	1.9445
1.7116	46.5229	419	1.9237
1.7116	46.6275	420	1.9270
1.7116	46.7320	421	1.9493
1.7116	46.8366	422	1.9743
1.7116	46.9412	423	1.9578
1.733	47.1046	424	1.9019
1.733	47.2092	425	1.9443
1.733	47.3137	426	1.9662
1.733	47.4183	427	1.9728
1.733	47.5229	428	1.9234
1.733	47.6275	429	1.9166
1.733	47.7320	430	1.9413
1.733	47.8366	431	1.8856
1.733	47.9412	432	1.9527
1.7065	48.1046	433	1.9524
1.7065	48.2092	434	1.9683
1.7065	48.3137	435	1.9489
1.7065	48.4183	436	1.9564
1.7065	48.5229	437	1.9767
1.7065	48.6275	438	1.9059
1.7065	48.7320	439	1.9629
1.7065	48.8366	440	1.9380
1.7065	48.9412	441	1.9695
1.6997	49.1046	442	1.9250
1.6997	49.2092	443	1.9392
1.6997	49.3137	444	1.9523
1.6997	49.4183	445	1.9459
1.6997	49.5229	446	1.9365
1.6997	49.6275	447	1.9134
1.6997	49.7320	448	1.9434
1.6997	49.8366	449	1.9772
1.6997	49.9412	450	1.9776

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0