Model card: Decision Index 0.3 public suite scores vs Jev and v1
Browse files
README.md
CHANGED
|
@@ -79,16 +79,20 @@ prediction, is in [Maincode/matilda-jev-decision-index](https://huggingface.co/d
|
|
| 79 |
under `runs/matilda-jev-v1.5/`. [`scores.json`](scores.json) in this repository is the earlier 0.2.1 run
|
| 80 |
(62.76), kept for reference.
|
| 81 |
|
| 82 |
-
| Metric |
|
| 83 |
-
|---|---:|
|
| 84 |
-
| Decision Index | 63.47 |
|
| 85 |
-
| Raw | 72.62 |
|
| 86 |
-
| Breadth | 62.45 |
|
| 87 |
-
| Knowledge & Reasoning | 49.83 |
|
| 88 |
-
| Language Understanding | 69.91 |
|
| 89 |
-
| Retrieval & Classification | 65.96 |
|
| 90 |
-
| Tools & Automation | 80.84 |
|
| 91 |
-
| Arts & Human Taste | 45.41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
Area scores are chance-corrected skill multiplied by 100. These results are
|
| 94 |
diagnostic: earlier training in this model's lineage used benchmark-related
|
|
|
|
| 79 |
under `runs/matilda-jev-v1.5/`. [`scores.json`](scores.json) in this repository is the earlier 0.2.1 run
|
| 80 |
(62.76), kept for reference.
|
| 81 |
|
| 82 |
+
| Metric | Jev | MATILDA JEV v1 | **MATILDA JEV v1.5** |
|
| 83 |
+
|---|---:|---:|---:|
|
| 84 |
+
| Decision Index (public) | 57.96 | 60.16 | **63.47** |
|
| 85 |
+
| Raw | 68.55 | 70.04 | **72.62** |
|
| 86 |
+
| Breadth | 57.27 | 59.21 | **62.45** |
|
| 87 |
+
| Knowledge & Reasoning | **53.87** | 46.84 | 49.83 |
|
| 88 |
+
| Language Understanding | 59.23 | 64.80 | **69.91** |
|
| 89 |
+
| Retrieval & Classification | 55.42 | 61.07 | **65.96** |
|
| 90 |
+
| Tools & Automation | 75.09 | 79.07 | **80.84** |
|
| 91 |
+
| Arts & Human Taste | 39.08 | **46.22** | 45.41 |
|
| 92 |
+
|
| 93 |
+
Jev is the public-suite part of its Decision Index 0.3 board entry (per-benchmark public results, as published in the
|
| 94 |
+
kit's `tests/fixtures/board-0.3.json`), not its Full score. MATILDA JEV v1 is
|
| 95 |
+
[Maincode/matilda-jev-v1](https://huggingface.co/Maincode/matilda-jev-v1) (`runs/matilda-jev-v1.3/` in the same dataset).
|
| 96 |
|
| 97 |
Area scores are chance-corrected skill multiplied by 100. These results are
|
| 98 |
diagnostic: earlier training in this model's lineage used benchmark-related
|