distilbert-base-uncased-finetuned-squad

This model is a fine-tuned version of distilbert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 5.4253

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
4.635	1.0	1384	4.5790
4.4544	2.0	2768	4.4523
4.2617	3.0	4152	4.3310
4.1031	4.0	5536	4.2816
3.9956	5.0	6920	4.2679
3.8676	6.0	8304	4.2379
3.7723	7.0	9688	4.3650
3.6775	8.0	11072	4.2837
3.5571	9.0	12456	4.3691
3.469	10.0	13840	4.3981
3.3777	11.0	15224	4.4369
3.2817	12.0	16608	4.5183
3.1812	13.0	17992	4.6001
3.0944	14.0	19376	4.6602
3.0193	15.0	20760	4.7157
2.9398	16.0	22144	4.7307
2.8544	17.0	23528	4.8100
2.7735	18.0	24912	4.9065
2.7244	19.0	26296	4.9776
2.6561	20.0	27680	5.0650
2.5995	21.0	29064	5.1428
2.5324	22.0	30448	5.1684
2.4836	23.0	31832	5.1686
2.4398	24.0	33216	5.2283
2.4037	25.0	34600	5.2688
2.3645	26.0	35984	5.3086
2.3382	27.0	37368	5.3551
2.3117	28.0	38752	5.4321
2.2725	29.0	40136	5.4103
2.2693	30.0	41520	5.4253

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1