Updated to Roberta-base from distilbert

README.md

@@ -14,29 +14,30 @@ widget:
 
 ---
 Text Multi-Label Sequence Classification model used to decode if passages contain a misfortunate event, a cause for misfortune, and/or an action to mollify or prevent some misfortune.
-8293 passages were used for Training and split into 5 folds (~6634 for the train set, ~1659 for the validation set over 5 folds). 
+8293 passages were used for Training and split into 5 folds (~6634 for the train set, ~1659 for the validation set over 5 folds). However, as previous comparisons between folds have shown no difference in accuracy or overfitting, to save on computation, we only trained on the 1st fold. 
 
 <br><b>Parameters</b>:
-<br>Transformer: distilbert-base-uncased
-<br>Tokenizer: distilbert-base-uncased
+<br>Transformer: roberta-base
+<br>Tokenizer: roberta-base
+<br>Head: multi_label_classification
 <br>learning rate: 2e-05 
 <br>weight decay: .01
 <br>Dropout: .1
 <br>Batch Size: 8
 <br>Epochs: 15
 <br>Metric for best model: F1 micro
-<br><br>Using epoch 13, the current F1 micro score of 2074 passages not used for training is .637. individual class f1 scores are shown below. Note that at this moment, some labels have been excluded as they are not relevant for the final use of the model.
+<br><br>Using epoch 15, the current F1 micro score of 2074 passages not used for training is .664 which is an improvement compared to the identical model with distilbert which achieved F1 micro of .637. Individual class f1 scores are shown below. Note that some labels have been excluded as they are not relevant for the final use of the model.
 <ul>
   <li>EVENT:  -
     <ul>
       <li>
-        Illness:  .866
+        Illness:  .876
       </li>
       <li>
-        Accident:  .41
+        Accident:  .458
       </li>
       <li>
-        Other:  .583
+        Other:  .588
       </li>
     </ul>
   </li>
@@ -46,16 +47,16 @@ Text Multi-Label Sequence Classification model used to decode if passages contai
         Just Happens:  -
       </li>
       <li>
-        Material Physical:  .431
+        Material Physical:  .476
       </li>
       <li>
-        Spirits and Gods:  .667
+        Spirits and Gods:  .728
       </li>
       <li>
-        Witchcraft and Sorcery:  .615
+        Witchcraft and Sorcery:  .651
       </li>
       <li>
-        Rule Violation Taboo:  .555
+        Rule Violation Taboo:  .517
       </li>
       <li>
         Jealous Evil Eye:  -
@@ -65,19 +66,19 @@ Text Multi-Label Sequence Classification model used to decode if passages contai
   <li>ACTION:  -
     <ul>
       <li>
-        Physical Material:  .635
+        Physical Material:  .672
       </li>
       <li>
-        Technical Specialist:  .357
+        Technical Specialist:  .5
       </li>
       <li>
-        Divination:  .303
+        Divination:  .406
       </li>
       <li>
-        Shaman Medium Healer:  .549
+        Shaman Medium Healer:  .582
       </li>
       <li>
-        Priest High Religion:  .34
+        Priest High Religion:  .375
       </li>
       <li>
         Other:  -