Update README.md
Browse files
README.md
CHANGED
|
@@ -29,3 +29,22 @@ Other training configuration:
|
|
| 29 |
| ADAM_BETA2 | 0.98 |
|
| 30 |
| ADAM_WEIGHT_DECAY | 0.01 |
|
| 31 |
| PEAK_LR | 1e-3 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
| ADAM_BETA2 | 0.98 |
|
| 30 |
| ADAM_WEIGHT_DECAY | 0.01 |
|
| 31 |
| PEAK_LR | 1e-3 |
|
| 32 |
+
|
| 33 |
+
Evaluation Results on GLUE after pretraining (30k steps):
|
| 34 |
+
|
| 35 |
+
| Task | Score |
|
| 36 |
+
|----------------------|-----------|
|
| 37 |
+
| CoLA-MattCorr | 20.0 |
|
| 38 |
+
| MRPC-F1 | 83.6 |
|
| 39 |
+
| STS-B-Pearson | 82.7 |
|
| 40 |
+
| SST-2-Acc | 89.7 |
|
| 41 |
+
| QQP-F1 | 89.0 |
|
| 42 |
+
| MNLI-m | 79.7 |
|
| 43 |
+
| MNLI-mm | 80.1 |
|
| 44 |
+
| QNLI-Acc | 89.3 |
|
| 45 |
+
| WNLI-Acc | 31.0 |
|
| 46 |
+
| RTE-Acc | 57.4 |
|
| 47 |
+
|
| 48 |
+
Evaluation Scores Curve (AVG of scores) during pretraining:
|
| 49 |
+
|
| 50 |
+
|