eval/
Raw outputs so we can recompute the headline scores without re-running the model.
Files to push
| file | what it is |
|---|---|
per_item_test.jsonl |
one line per test item: model topic, Jev topic, confidence, show/hide for each |
confusion.json |
56×56 confusion matrix over the test split |
Suggested schema for per_item_test.jsonl
{"id": "...", "model_topic": "...", "jev_topic": "...", "confidence": 0.71, "model_show_hide": "show", "jev_show_hide": "show", "human_topic": null}
Numbers to reproduce from this
- top-1 topic accuracy vs Jev (report: 92/105 = 87.6%)
- macro-F1 (report: 0.572)
- show/hide agreement (report: 95/105 = 90.5%)
- confidence split: of 24 errors, 19 < 0.5 confidence, only 1 ≥ 0.9