bert-mini-mlm-finetuned-imdb

This model is a fine-tuned version of google/bert_uncased_L-4_H-256_A-4 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.6935

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 200

Training results

Training Loss	Epoch	Step	Validation Loss
3.2058	0.64	500	2.9411
3.1048	1.28	1000	2.9042
3.0631	1.92	1500	2.8780
3.0197	2.56	2000	2.8667
3.0071	3.2	2500	2.8503
2.9886	3.84	3000	2.8319
2.9577	4.48	3500	2.8127
2.9498	5.12	4000	2.8080
2.9301	5.75	4500	2.7894
2.9229	6.39	5000	2.7912
2.9027	7.03	5500	2.7874
2.8961	7.67	6000	2.7785
2.8869	8.31	6500	2.7619
2.8793	8.95	7000	2.7607
2.8729	9.59	7500	2.7581
2.8523	10.23	8000	2.7593
2.8525	10.87	8500	2.7433
2.8403	11.51	9000	2.7505
2.8318	12.15	9500	2.7444
2.8314	12.79	10000	2.7352
2.8136	13.43	10500	2.7334
2.8161	14.07	11000	2.7280
2.7955	14.71	11500	2.7342
2.7951	15.35	12000	2.7237
2.7878	15.98	12500	2.7171
2.7816	16.62	13000	2.7160
2.7805	17.26	13500	2.7120
2.7776	17.9	14000	2.7078
2.7661	18.54	14500	2.7086
2.7678	19.18	15000	2.7017
2.7613	19.82	15500	2.7015
2.7516	20.46	16000	2.6958
2.7529	21.1	16500	2.6909
2.7422	21.74	17000	2.6966
2.738	22.38	17500	2.7034
2.7303	23.02	18000	2.6935

Framework versions

• Transformers 4.25.1
• Pytorch 1.12.1+cu113
• Datasets 2.7.1
• Tokenizers 0.13.2