Update README.md
Browse files
README.md
CHANGED
|
@@ -46,9 +46,12 @@ Training was done on the training splits of
|
|
| 46 |
|
| 47 |
## Training procedure
|
| 48 |
|
| 49 |
-
The procedure for training was
|
| 50 |
-
by only grabbing the last logit form doing a feedforward on the whole prompt (question with choices) and we do cross entropy loss on this last logit with the 4 options to choose 4 from
|
| 51 |
-
(so we don't do cross entyropy on the whole vocabulary we only do it on the tokens of the letters of the 4 options (A, B, C and D))
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
### Training hyperparameters
|
| 54 |
|
|
|
|
| 46 |
|
| 47 |
## Training procedure
|
| 48 |
|
| 49 |
+
The procedure for training was done with example of any amount of choices, for each batch size we padd the options to the biggest amount of options and example has in that batch,
|
| 50 |
+
and from there we do the training by only grabbing the last logit form doing a feedforward on the whole prompt (question with choices) and we do cross entropy loss on this last logit with the 4 options to choose 4 from
|
| 51 |
+
(so we don't do cross entyropy on the whole vocabulary we only do it on the tokens of the letters of the 4 options (A, B, C and D)).
|
| 52 |
+
|
| 53 |
+
We also template all the training examples with 7 random templates as to make the model robuts to different types of ways one could ask an MCQA question, using different prompts can
|
| 54 |
+
make the results vary a lot
|
| 55 |
|
| 56 |
### Training hyperparameters
|
| 57 |
|