Mapika commited on
Commit
ff99089
·
verified ·
1 Parent(s): 533964d

Card: MNLI accuracy on the full validation_matched split

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -384,6 +384,10 @@ Per-task accuracy / ECE on the held-out datasets of the rebuilt set, v8 against
384
  | **macro / micro** | 0.703 / 0.711 | 0.706 / 0.711 |
385
  | macro over the subsets not trained on | 0.658 | 0.645 |
386
 
 
 
 
 
387
  **Calibration of v11 on our own sets** (10-bin top-label ECE on stored temperature-1 logits, read at the served temperatures;
388
  heldout_jb: held-out generated families, test_teacher2: held-out document questions, guard: knowledge guard, cal_human:
389
  human-labelled validation rows):
 
384
  | **macro / micro** | 0.703 / 0.711 | 0.706 / 0.711 |
385
  | macro over the subsets not trained on | 0.658 | 0.645 |
386
 
387
+ On all 9,815 MNLI validation_matched items, decider-2b v11 (revision `533964d`) scores **0.864** (entailment 0.844, neutral
388
+ 0.853, contradiction 0.894), level with the multinli subset above. An independent run on 120 other validation_matched items
389
+ measured 0.792 ([discussion #4](https://huggingface.co/Mapika/decider-2b/discussions/4)).
390
+
391
  **Calibration of v11 on our own sets** (10-bin top-label ECE on stored temperature-1 logits, read at the served temperatures;
392
  heldout_jb: held-out generated families, test_teacher2: held-out document questions, guard: knowledge guard, cal_human:
393
  human-labelled validation rows):