| # eval/ | |
| Raw outputs so we can recompute the headline scores **without re-running the model**. | |
| ## Files to push | |
| | file | what it is | | |
| |---|---| | |
| | `per_item_test.jsonl` | one line per test item: model topic, Jev topic, confidence, show/hide for each | | |
| | `confusion.json` | 56×56 confusion matrix over the test split | | |
| ## Suggested schema for `per_item_test.jsonl` | |
| ```json | |
| {"id": "...", "model_topic": "...", "jev_topic": "...", "confidence": 0.71, "model_show_hide": "show", "jev_show_hide": "show", "human_topic": null} | |
| ``` | |
| ## Numbers to reproduce from this | |
| - top-1 topic accuracy vs Jev (report: 92/105 = 87.6%) | |
| - macro-F1 (report: 0.572) | |
| - show/hide agreement (report: 95/105 = 90.5%) | |
| - confidence split: of 24 errors, 19 < 0.5 confidence, only 1 ≥ 0.9 | |