Evaluation log: method and first back-test result (RG-50.030.0034)

Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE

docs/evaluation-log.md

@@ -0,0 +1,39 @@
+# Evaluation log
+
+Results of CuratorBot's subject-heading suggestions against the museum's own catalogue records, for slides, reports and publications. Add one row per run; never edit a past row (add a corrected one instead).
+
+## Method
+
+- **Interviews:** released USHMM oral histories with a public transcript (`collections.ushmm.org/oh_findingaids/<RG>_trs_en.pdf`) and a catalogue record. Transcripts are imported with `scripts/import_transcript_pdf.py`; the museum notes they have not been checked for spelling or accuracy.
+- **Reference:** the record's subject headings (`catalogue.json`), copied from the USHMM record via the EHRI portal.
+- **Vocabulary condition A:** the record's headings were added to the demo vocabulary before the run, so a miss reflects the model rather than a missing heading. Condition B (demo vocabulary only) shows vocabulary gaps.
+- **Scoring:** `curatorbot/backtest.py`. CuratorBot suggests at most 12 headings; only grounded, in-vocabulary suggestions count, ranked by confidence, top 10 scored.
+  - *Recall (exact):* catalogue headings matched exactly by a suggestion, divided by all catalogue headings.
+  - *Recall (main heading):* also counts a suggestion with the same main heading but a different subdivision.
+  - *Ceiling:* with a top-10 cut-off, the best possible recall is min(10, n) / n for n catalogue headings.
+  - *Precision@10 (exact):* exact matches among the 10 scored suggestions.
+- **Grounding:** every suggestion must quote the transcript verbatim; the validators reject any that do not.
+
+## Results
+
+| Date | Interview | Model | Vocab | Catalogue headings | Recall exact | Recall main heading | Ceiling | Precision@10 exact | Quotes verified / rejected |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 2026-10-26 | RG-50.030.0034, Regina Hamburger Bomba (1990) | openrouter: anthropic/claude-sonnet-5.5 (to confirm from `brief.json` `run.llm`) | A | 16 | 44% (7/16) | 56% (9/16) | 62.5% (10/16) | 70% (7/10) | 20 / 0 |
+
+## Readings for slides
+
+- **RG-50.030.0034:** 7 of CuratorBot's top 10 suggested headings matched the museum's catalogue exactly, and every suggestion was backed by a verified quote from the transcript. Two more had the right main heading with a different subdivision. (One interview; transcript unverified by the museum.)
+
+## Notes on misses
+
+Expected kinds of miss, to confirm per interview from the Catalogue comparison tab:
+
+- **Form headings** added by cataloguing convention rather than from the content, e.g. "Women--Personal narratives.", "Holocaust survivors--Interviews.".
+- **Narrow topical headings** for brief passages, e.g. "Ukraine--Armed Forces--Poland.".
+
+## Open methodological points
+
+- Replace the fixed top-10 cut-off with k = number of catalogue headings (R-precision), and report the ceiling alongside recall.
+- Report condition A and condition B for each interview.
+- Use the museum's authority export as the vocabulary once available; record its version here.
+- Grow the sample before claiming a general figure; report per-interview results and the mean.