CeLLaTe-tapt-bioformer16l-tokenizer-adapted_v2

This model is a fine-tuned version of Mardiyyah/CeLLaTe-bioformer16l-tokenizer-adapted_v2 on the Mardiyyah/TAPT_CeLLaTe2.0 dataset. It achieves the following results on the evaluation set:

• Loss: 2.1972
• Accuracy: 0.6595
• Perplexity: 8.9995

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Perplexity
No log	1.0	15	10.0400	0.0001	22925.8849
No log	2.0	30	8.3449	0.0151	4208.6382
No log	3.0	45	7.4704	0.0487	1755.2841
No log	4.0	60	6.6988	0.1537	811.4223
No log	5.0	75	5.7744	0.2423	321.9416
No log	6.0	90	5.0991	0.2929	163.8712
7.2448	7.0	105	4.6161	0.3352	101.1012
7.2448	8.0	120	4.3294	0.3684	75.9018
7.2448	9.0	135	4.0365	0.4165	56.6304
7.2448	10.0	150	3.7739	0.4547	43.5498
7.2448	11.0	165	3.6480	0.4708	38.3989
7.2448	12.0	180	3.5158	0.4929	33.6412
7.2448	13.0	195	3.3748	0.5207	29.2174
3.7694	14.0	210	3.3173	0.5312	27.5857
3.7694	15.0	225	3.1718	0.5520	23.8504
3.7694	16.0	240	3.0936	0.5672	22.0569
3.7694	17.0	255	2.9628	0.5847	19.3523
3.7694	18.0	270	2.9632	0.5905	19.3593
3.7694	19.0	285	2.8408	0.6028	17.1289
2.9131	20.0	300	2.8172	0.6016	16.7297
2.9131	21.0	315	2.7721	0.6076	15.9929
2.9131	22.0	330	2.6897	0.6212	14.7273
2.9131	23.0	345	2.7109	0.6143	15.0433
2.9131	24.0	360	2.6392	0.6273	14.0017
2.9131	25.0	375	2.5636	0.6340	12.9831
2.9131	26.0	390	2.6081	0.6283	13.5731
2.5444	27.0	405	2.5520	0.6382	12.8327
2.5444	28.0	420	2.5556	0.6311	12.8793
2.5444	29.0	435	2.4450	0.6420	11.5305
2.5444	30.0	450	2.4491	0.6444	11.5782
2.5444	31.0	465	2.4579	0.6372	11.6799
2.5444	32.0	480	2.4132	0.6445	11.1694
2.5444	33.0	495	2.4018	0.6413	11.0430
2.3127	34.0	510	2.4056	0.6424	11.0851
2.3127	35.0	525	2.3329	0.6547	10.3082
2.3127	36.0	540	2.3531	0.6487	10.5180
2.3127	37.0	555	2.3558	0.6461	10.5470
2.3127	38.0	570	2.3010	0.6507	9.9838
2.3127	39.0	585	2.3393	0.6503	10.3739
2.1667	40.0	600	2.3152	0.6504	10.1271
2.1667	41.0	615	2.2805	0.6537	9.7815
2.1667	42.0	630	2.3036	0.6549	10.0099
2.1667	43.0	645	2.2601	0.6543	9.5840
2.1667	44.0	660	2.2526	0.6575	9.5124
2.1667	45.0	675	2.2620	0.6560	9.6025
2.1667	46.0	690	2.2442	0.6563	9.4332
2.0694	47.0	705	2.2202	0.6594	9.2095
2.0694	48.0	720	2.2120	0.6605	9.1335
2.0694	49.0	735	2.1944	0.6616	8.9749
2.0694	50.0	750	2.1647	0.6646	8.7118
2.0694	51.0	765	2.2410	0.6562	9.4024
2.0694	52.0	780	2.1533	0.6645	8.6136
2.0694	53.0	795	2.1766	0.6628	8.8162
1.9781	54.0	810	2.1730	0.6596	8.7850
1.9781	55.0	825	2.1170	0.6672	8.3063
1.9781	56.0	840	2.1476	0.6655	8.5644
1.9781	57.0	855	2.1318	0.6683	8.4299
1.9781	58.0	870	2.1554	0.6620	8.6312
1.9781	59.0	885	2.1558	0.6613	8.6352
1.9185	60.0	900	2.1578	0.6582	8.6522

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0