FYP_qa_final

This model is a fine-tuned version of deepset/deberta-v3-base-squad2 on an MRQA dataset. It achieves the following results on the evaluation set:

• Loss: 2.7493

Model description

This model is trained for performing extractive question-answering tasks for academic essays.

Intended uses & limitations

More information needed

Training and evaluation data

The dataset used for training is listed below according to training sequences:

1. MRQA(train split)
2. UCLNLP/adversarial_qa
3. mbartolo/synQA
4. MRQA(test split)*This model

Training procedure

The training approach uses the fine-tuning approach of transfer learning on the pre-trained model to perform NLP QA tasks. Each time a model was trained with one dataset only and saved as the PTMs for the next training. This model is the last model that trained with MRQA(test split).

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.8084	0.48	300	3.1468
2.5707	0.96	600	2.9035
2.5187	1.44	900	2.7175
2.4463	1.91	1200	2.7497
2.4328	2.39	1500	2.7229
2.3839	2.87	1800	2.7493

Framework versions

• Transformers 4.39.3
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2