Update README.md

README.md

@@ -11,28 +11,93 @@ model-index:
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
+# Important Note:
+I created the `combined` metric (55% F1 score + 45% exact match score) to retrieve the best result. Here is the setting in the `TrainingArguments`:
+```
+  load_best_model_at_end=True,
+  metric_for_best_model='combined',
+  greater_is_better=True,
+```
+
 # DSPFirst-Finetuning-5
 
-This model is a fine-tuned version of [ahotrod/electra_large_discriminator_squad2_512](https://huggingface.co/ahotrod/electra_large_discriminator_squad2_512) on an unknown dataset.
+This model is a fine-tuned version of [ahotrod/electra_large_discriminator_squad2_512](https://huggingface.co/ahotrod/electra_large_discriminator_squad2_512) on a generated Questions and Answers dataset from the DSPFirst textbook based on the SQuAD 2.0 format.<br />
 It achieves the following results on the evaluation set:
 - Loss: 0.8529
 - Exact: 66.3117
 - F1: 73.4039
 - Combined: 70.2124
 
-## Model description
-
-More information needed
+## More accurate metrics:
+
+### Before fine-tuning:
+
+```
+ 'HasAns_exact': 54.71817606079797,
+ 'HasAns_f1': 61.08672724332754,
+ 'HasAns_total': 1579,
+ 'NoAns_exact': 88.78048780487805,
+ 'NoAns_f1': 88.78048780487805,
+ 'NoAns_total': 205,
+ 'best_exact': 58.63228699551569,
+ 'best_exact_thresh': 0.0,
+ 'best_f1': 64.26902596256402,
+ 'best_f1_thresh': 0.0,
+ 'exact': 58.63228699551569,
+ 'f1': 64.26902596256404,
+ 'total': 1784
+```
+
+### After fine-tuning:
+
+```
+ 'HasAns_exact': 67.57441418619379,
+ 'HasAns_f1': 75.92137683558988,
+ 'HasAns_total': 1579,
+ 'NoAns_exact': 63.41463414634146,
+ 'NoAns_f1': 63.41463414634146,
+ 'NoAns_total': 205,
+ 'best_exact': 67.0964125560538,
+ 'best_exact_thresh': 0.0,
+ 'best_f1': 74.48422310728503,
+ 'best_f1_thresh': 0.0,
+ 'exact': 67.0964125560538,
+ 'f1': 74.48422310728503,
+ 'total': 1784
+```
+
+# Dataset
+A visualization of the dataset can be found [here](https://github.gatech.edu/pages/VIP-ITS/textbook_SQuAD_explore/explore/textbookv1.0/textbook/).<br />
+The split between train and test is 70% and 30% respectively.
+```
+DatasetDict({
+    train: Dataset({
+        features: ['id', 'title', 'context', 'question', 'answers'],
+        num_rows: 4160
+    })
+    test: Dataset({
+        features: ['id', 'title', 'context', 'question', 'answers'],
+        num_rows: 1784
+    })
+})
+```
 
 ## Intended uses & limitations
 
-More information needed
+This model is fine-tuned to answer questions from the DSPFirst textbook. I'm not really sure what I am doing so you should review before using it.<br />
+Also, you should improve the Dataset either by using a **better generated questions and answers model** (currently using https://github.com/patil-suraj/question_generation) or perform **data augmentation** to increase dataset size.
 
 ## Training and evaluation data
 
-More information needed
+- `batch_size` of 6 results in 14.03 GB VRAM
+- Utilizes `gradient_accumulation_steps` to get total batch size to 514 (batch size should be at least 256)
+- 4.52 GB RAM
+- 30% of the total questions is dedicated for evaluating.
 
 ## Training procedure
+- The model was trained from [Google Colab](https://colab.research.google.com/drive/1dJXNstk2NSenwzdtl9xA8AqjP4LL-Ks_?usp=sharing)
+- Utilizes Tesla P100 16GB, took 6.3 hours to train
+- `load_best_model_at_end` is enabled in TrainingArguments
 
 ### Training hyperparameters