Card: MNLI accuracy on the full validation_matched split
Browse files
README.md
CHANGED
|
@@ -384,6 +384,10 @@ Per-task accuracy / ECE on the held-out datasets of the rebuilt set, v8 against
|
|
| 384 |
| **macro / micro** | 0.703 / 0.711 | 0.706 / 0.711 |
|
| 385 |
| macro over the subsets not trained on | 0.658 | 0.645 |
|
| 386 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 387 |
**Calibration of v11 on our own sets** (10-bin top-label ECE on stored temperature-1 logits, read at the served temperatures;
|
| 388 |
heldout_jb: held-out generated families, test_teacher2: held-out document questions, guard: knowledge guard, cal_human:
|
| 389 |
human-labelled validation rows):
|
|
|
|
| 384 |
| **macro / micro** | 0.703 / 0.711 | 0.706 / 0.711 |
|
| 385 |
| macro over the subsets not trained on | 0.658 | 0.645 |
|
| 386 |
|
| 387 |
+
On all 9,815 MNLI validation_matched items, decider-2b v11 (revision `533964d`) scores **0.864** (entailment 0.844, neutral
|
| 388 |
+
0.853, contradiction 0.894), level with the multinli subset above. An independent run on 120 other validation_matched items
|
| 389 |
+
measured 0.792 ([discussion #4](https://huggingface.co/Mapika/decider-2b/discussions/4)).
|
| 390 |
+
|
| 391 |
**Calibration of v11 on our own sets** (10-bin top-label ECE on stored temperature-1 logits, read at the served temperatures;
|
| 392 |
heldout_jb: held-out generated families, test_teacher2: held-out document questions, guard: knowledge guard, cal_human:
|
| 393 |
human-labelled validation rows):
|