File size: 1,025 Bytes
99233b6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | Hyperparameters for GLUE:
- Learning rate: 5e-5
- Batch size: 64
- Max epochs: 10
- Patience: 10 (for CoLA, MRPC, RTE, BoolQ, MultiRC, and WSC), 100 (for MNLI, QQP, QNLI, and SST-2)
- Random seed: 12
- Weight decay: 0.1
- Warmup ratio: 0.1
- Learning rate scheduler: cosine
- Eval strategy: epoch (for CoLA, MRPC, RTE, BoolQ, MultiRC, and WSC), steps (for MNLI, QQP, QNLI, and SST-2)
- Eval every: 1 (for CoLA, MRPC, RTE, BoolQ, MultiRC, and WSC), 200 (for SST-2 and QNLI), 500 (for MNLI and QQP)
Hyperparameters for MSGS:
- Learning rate: 5e-5 (for CR, SC, RP, MV_RTP, and SC_LC), 1.5e-5 (for LC), 1e-5 (for SC_RP), 8e-6 (for MV_LC), 5e-6 (for MV), 5e-7 (CR_LC)
- Batch size: 32
- Max epochs: 10 (for CR, SC, RP, MV_RTP, SC_LC, SC_RP, MV, and CR_LC), 3 (for LC), 5 (for MV_LC)
- Patience: 10 (for CR, SC, RP, MV_RTP, SC_LC, SC_RP, MV, and CR_LC), 3 (for LC), 5 (for MV_LC)
- Random seed: 12
- Weight decay: 0.1
- Warmup ratio: 0.1
- Learning rate scheduler: cosine
- Eval strategy: epoch
- Eval every: 1 |