File size: 7,526 Bytes
7dea30b 501c926 7dea30b 3392877 7dea30b 3392877 7dea30b c6eaad0 23eb29c 7dea30b 501c926 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 |
---
library_name: transformers
license: apache-2.0
base_model: unsloth/Qwen3-0.6B-Base
tags:
- unsloth
- generated_from_trainer
model-index:
- name: Qwen3-0.6B-MNLP_M2_mcqa_model
results: []
datasets:
- andresnowak/MNLP_MCQA_dataset
- andresnowak/MNLP_M2_mcqa_dataset
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# Qwen3-0.6B-MNLP_M2_mcqa_model
This model is a fine-tuned version of [unsloth/Qwen3-0.6B-Base](https://huggingface.co/unsloth/Qwen3-0.6B-Base) on an unknown dataset.
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
Training was done on the training splits of
- MEDMCQA
- MMLU
- Sciq
- Ai2 Arc
- Math_qa
- ScienceQa
- Openbookqa
## Training procedure
The procedure for training was to only leave the question that have only 4 choices to chose from, and from there we do the training
by only grabbing the last logit form doing a feedforward on the whole prompt (question with choices) and we do cross entropy loss on this last logit with the 4 options to choose 4 from
(so we don't do cross entyropy on the whole vocabulary we only do it on the tokens of the letters of the 4 options (A, B, C and D))
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 32
- total_train_batch_size: 64
- optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.04
- num_epochs: 2
## Evaluation Results
The model was evaluated on a suite of Multiple Choice Question Answering (MCQA) benchmarks (on its validation and test sets repsectively for each one),
and NLP4education is only the approximated 1000 question and answers given to use.
**Important Note on MCQA Evals Benchmark:**
**The performance on these benchmarks is as follows**:
### First evaluation: The tests where done with this prompt (type 5):
```
This question assesses challenging STEM problems as found on graduate standardized tests. Carefully evaluate the options and select the correct answer.
---
[Insert Question Here]
---
[Insert Choices Here, e.g.:
A. Option 1
B. Option 2
C. Option 3
D. Option 4]
---
Your response should include the letter and the exact text of the correct choice.
Example: B. Entropy increases.
Answer:
```
And the teseting was done on ``` [Letter]. [Text answer]```
| Benchmark | Accuracy (Acc) | Normalized Accuracy (Acc Norm) |
| :----------------- | :------------- | :----------------------------- |
| ARC Challenge | 66.28% | 64.92% |
| ARC Easy | 84.22% | 81.33% |
| GPQA | 38.84% | 36.61% |
| Math QA | 25.03% | 24.67% |
| MCQA Evals | 43.51% | 40.91% |
| MMLU | 52.17% | 52.17% |
| MMLU Pro | 16.45% | 15.04% |
| MuSR | 53.17% | 52.25% |
| NLP4Education | 44.45% | 42.65% |
| **Overall** | **47.12%** | **45.62%** |
### Second evaluation: (type 0)
```
The following are multiple choice questions (with answers) about knowledge and skills in advanced master-level STEM courses.
---
*[Insert Question Here]*
---
*[Insert Choices Here, e.g.:*
*A. Option 1*
*B. Option 2*
*C. Option 3*
*D. Option 4]*
---
Answer:
```
And the teseting was done on ``` [Letter]. [Text answer]```
| Benchmark | Accuracy (Acc) | Normalized Accuracy (Acc Norm) |
| :----------------- | :------------- | :----------------------------- |
| ARC Challenge | 69.95% | 65.33% |
| ARC Easy | 84.45% | 78.51% |
| GPQA | 31.92% | 28.57% |
| Math QA | 27.02% | 26.88% |
| MCQA Evals | 43.90% | 35.32% |
| MMLU | 52.17% | 52.17% |
| MMLU Pro | 15.04% | 13.27% |
| MuSR | 53.17% | 52.25% |
| NLP4Education | 49.14% | 42.85% |
| **Overall** | **47.42%** | **43.91%** |
### Third evaluation: (type 2)
```
This is part of an assessment on graduate-level science, technology, engineering, and mathematics (STEM) concepts. Each question is multiple-choice and requires a single correct answer.
---
*[Insert Question Here]*
---
*[Insert Choices Here, e.g.:*
*A. Option 1*
*B. Option 2*
*C. Option 3*
*D. Option 4]*
---
For grading purposes, respond with: [LETTER]. [VERBATIM TEXT]
Example: D. Planck constant
Your Response:
```
And the teseting was done on ``` [Letter]. [Text answer]```
| Benchmark | Accuracy (Acc) | Normalized Accuracy (Acc Norm) |
| :----------------- | :------------- | :----------------------------- |
| ARC Challenge | 55.34% | 55.34% |
| ARC Easy | 74.00% | 74.00% |
| GPQA | 29.69% | 29.69% |
| Math QA | 22.35% | 22.35% |
| MCQA Evals | 37.92% | 37.92% |
| MMLU | 52.14% | 52.14% |
| MMLU Pro | 12.98% | 12.98% |
| MuSR | 53.04% | 53.04% |
| NLP4Education | 36.36% | 36.36% |
| **Overall** | **41.53%** | **41.53%** |
### First evaluation: (type 0)
```
The following are multiple choice questions (with answers) about knowledge and skills in advanced master-level STEM courses.
---
*[Insert Question Here]*
---
*[Insert Choices Here, e.g.:*
*A. Option 1*
*B. Option 2*
*C. Option 3*
*D. Option 4]*
---
Answer:
```
And the teseting was done on ``` [Letter]```
| Benchmark | Accuracy (Acc) | Normalized Accuracy (Acc Norm) |
| :----------------- | :------------- | :----------------------------- |
| ARC Challenge | 70.63% | 70.63% |
| ARC Easy | 85.13% | 85.13% |
| GPQA | 25.45% | 25.45% |
| Math QA | 27.35% | 27.35% |
| MCQA Evals | 45.97% | 45.97% |
| MMLU | 52.14% | 52.14% |
| MMLU Pro | 14.97% | 14.97% |
| MuSR | 53.04% | 53.04% |
| NLP4Education | 50.86% | 50.86% |
| **Overall** | **47.28%** | **47.28%** |
### Framework versions
- Transformers 4.51.3
- Pytorch 2.5.1+cu121
- Datasets 3.6.0
- Tokenizers 0.21.0 |