Spaces:
Running
Evaluation log
Results of CuratorBot's subject-heading suggestions against the museum's own catalogue records, for slides, reports and publications. Add one row per run; never edit a past row's figures (add a corrected row or a note instead). Keep each run's backtest.json (it is uploaded with the brief to the private dataset).
Method
- Interviews: released USHMM oral histories with a public transcript (
collections.ushmm.org/oh_findingaids/<RG>_trs_en.pdf) and a catalogue record. Transcripts are imported withscripts/import_transcript_pdf.py; the museum notes they have not been checked for spelling or accuracy. - Reference: the record's subject headings (
catalogue.json), copied from the USHMM record via the EHRI portal. - Vocabulary condition A: the record's headings (all test interviews' records together,
scripts/build_eval_vocabulary.py) were added to the demo vocabulary before the run, so a miss reflects the model rather than a missing heading. An upper bound: every right answer is on a short list. - Condition B: the shared demo vocabulary only (67 headings, written for the prototype on 2026-10-08 and not checked against id.loc.gov). Shows the vocabulary gap. Partly in-sample: the 2026-10-27 revision added form headings taken from the test records and renamed two ghetto headings to their current form.
- Condition C: the headings of the rest of the USHMM oral history collection, harvested from the EHRI portal's EAD export (
scripts/harvest_ehri_headings.py), with the test interviews' own records left out, replacing the demo list (CURATORBOT_VOCABULARY). A realistic list, large and chosen without reference to the test interviews: a heading is available only if other interviews use it. The main condition until the museum's authority export is available. - Scoring:
curatorbot/backtest.py. CuratorBot suggests at most 12 headings; only grounded, in-vocabulary suggestions count, ranked by confidence, top 10 scored.- Recall (exact): catalogue headings matched exactly by a suggestion, divided by all catalogue headings.
- Recall (main heading): also counts a suggestion with the same main heading but a different subdivision.
- Ceiling: with a top-10 cut-off, the best possible recall is min(10, n) / n for n catalogue headings.
- Precision@10 (exact): exact matches among the 10 scored suggestions.
- Cost: OpenRouter list prices at the time of the run (Claude Sonnet 5.5: US$2 per million input tokens, US$10 per million output tokens).
- Grounding: every suggestion must quote the transcript verbatim; the validators reject any that do not. "Quotes verified / rejected" counted quote failures and out-of-vocabulary headings together until 2026-10-27; briefs now record them separately (
quotes_rejected,out_of_vocabulary). - Rescoring:
curatorbot revalidate <brief.json> --catalogue <catalogue.json>reapplies the current validators and back-test to a saved brief without a new model call. Rescored rows say so.
Results
Reference v1 (all rows below): the record's topical and geographic subject headings only. It omitted corporate-name headings such as "Auschwitz (Concentration camp)", so suggestions matching those were wrongly counted as extras. Reference v2 adds corporate names (personal names stay excluded, as CuratorBot does not suggest them yet); v2 runs are reported separately.
| Date | Interview | Model | Vocab | Catalogue headings | Recall exact | Recall main heading | Ceiling | Precision@10 exact | Quotes verified / rejected | Time, tokens, cost |
|---|---|---|---|---|---|---|---|---|---|---|
| 2026-10-26 | RG-50.030.0034, Regina Hamburger Bomba (1990), run 1 | openrouter: anthropic/claude-sonnet-5.5 (no fallback); spaCy en_core_web_sm | A | 16 | 44% (7/16) | 56% (9/16) | 62.5% (10/16) | 70% (7/10), inferred: backtest file not kept | 20 / 0 | 34 s; 16,776 in / 4,877 out tokens; about US$0.08 |
| 2026-10-26 | RG-50.030.0034, Regina Hamburger Bomba (1990), run 2 | openrouter: anthropic/claude-sonnet-5.5 (no fallback); spaCy en_core_web_trf | A | 16 | 44% (7/16) | 62.5% (10/16) | 62.5% (10/16) | 70% (7/10), recorded | 19 / 0 | 38 s; 16,776 in / 5,375 out tokens; about US$0.09 |
| 2026-10-26 | RG-50.030.0002, Rita Kerner Hilton (1994), run 1 | openrouter: anthropic/claude-sonnet-5.5 (no fallback); spaCy en_core_web_trf | A | 13 | 38% (5/13) | 38% (5/13) | 77% (10/13) | 50% (5/10), recorded | 20 / 0 | 46 s; 51,705 in / 5,985 out tokens; about US$0.16 |
Reference v2 (topical, geographic and corporate-name headings)
Vocabulary: demo list with superseded "Ghettos, Jewish--" forms changed to "Jewish ghettos--", plus all three records' v2 headings (113 headings). Output limit 16,000 tokens.
| Date | Interview | Model | Vocab | Catalogue headings | Recall exact | Recall main heading | Ceiling | Precision@10 exact | Quotes verified / rejected | Time, tokens, cost |
|---|---|---|---|---|---|---|---|---|---|---|
| 2026-10-27 | RG-50.030.0034, Regina Hamburger Bomba (1990), run 3 | openrouter: anthropic/claude-sonnet-5.5 (no fallback); spaCy en_core_web_trf | A | 20 | 30% (6/20) | 45% (9/20) | 50% (10/20) | 60% (6/10) | 21 / 0 | 35 s; 17,205 in / 4,891 out tokens; about US$0.08 |
| 2026-10-27 | RG-50.030.0002, Rita Kerner Hilton (1994), run 3 | openrouter: anthropic/claude-sonnet-5.5 (no fallback); spaCy en_core_web_trf | A | 17 | 41% (7/17) | 53% (9/17) | 59% (10/17) | 70% (7/10) | 22 / 0 | 47 s; 51,992 in / 6,685 out tokens; about US$0.17 |
| 2026-10-27 | RG-50.030.0001, David A. Kochalski (1994), run 1 | openrouter: anthropic/claude-sonnet-5.5 (fallback switched off); spaCy en_core_web_trf | A | 16 | 25% (4/16) | 38% (6/16) | 62.5% (10/16) | 40% (4/10) | 28 / 1 (the 1 is a heading stopped by a validator bug, not a quote) | 53 s; 31,795 in / 7,461 out tokens; about US$0.14 |
| 2026-10-27 | RG-50.030.0001, David A. Kochalski (1994), run 1 rescored (curatorbot revalidate, no new model call) |
as run 1 | A | 16 | 25% (4/16) | 38% (6/16) | 62.5% (10/16) | 40% (4/10) | 28 / 0 (0 quotes rejected, 0 outside the vocabulary) | none |
| Bomba and Hilton (pooled) | 37 | 35% (13/37) | 49% (18/37) | 54% (20/37) | 65% (13/20) | 43 / 0 | about US$0.25 | |||
| All three (pooled) | 53 | 32% (17/53) | 45% (24/53) | 57% (30/53) | 57% (17/30) | 71 / 0 | about US$0.39 in total |
Excluded run: RG-50.030.0002 run 2 (2026-10-27). Claude's answer exceeded the 6,000-token output limit then in force, so the fallback model (openai/gpt-4.1-mini) answered: recall 12% exact, 24% main heading. Of its 18 quoted items, 9 failed verification and were rejected by the validators, which shows the safeguard working on a weaker model. Not a measure of CuratorBot's configured model; the limit is now 16,000.
Protocol v3 (from 2026-10-27): changes tested against the v2 runs above
What changed, all driven by the misses above:
- Prompt: cover the whole life (before, during, liberation, after: DP camps, emigration, resettlement); pair a specific ghetto or camp heading with its town or country heading; leave generic form headings to the rule.
- Form headings by rule (
curatorbot/data/form_headings.json): Women-- or Men--Personal narratives. (from a grounded narrator-gender quote), Holocaust, Jewish (1939-1945)--Personal narratives., Holocaust survivors--Interviews. Marked "Added by rule" in the brief. - Vocabulary: bare chronological terms ("1939-1945" and others) removed; "Ghettos, Jewish--" forms changed to "Jewish ghettos--"; generic form headings added. Condition A headings now come from a separate file (
CURATORBOT_VOCAB_EXTRA, built byscripts/build_eval_vocabulary.py), so the shared list stays clean. - Scoring: the existing columns score the model's own suggestions (top 10, as before, so v2 and v3 compare directly). New: R-precision (exact matches in the top n of everything offered, n = catalogue headings) and precision and recall over all suggestions, both including the rule headings.
- Places: GeoNames lookup for towns the period gazetteer lacks, "near" anchors and country centres as marked approximations; coverage counted per brief.
Caveat to state with any v3 figure: the form-heading rules were derived from these same three records, so "with rules" figures on them are in-sample. Report model-only and with-rules figures separately, and confirm the rules on interviews not used to design them.
| Date | Interview | Model | Catalogue headings | Model only: exact (precision@10) | With rules: R-precision | With rules: all suggestions, precision / recall exact | Quotes verified / rejected / outside vocabulary | Journeys drawn (exact) / total | Time, tokens, cost |
|---|---|---|---|---|---|---|---|---|---|
| 2026-10-27 | RG-50.030.0034, Bomba, run 4 | openrouter: anthropic/claude-sonnet-5.5 (fallback off); spaCy trf | 20 | 25% (5/20), by main heading 45% (9/20); precision 50% (5/10) | 35% (7/20) | 47% (7/15) / 35% (7/20) | 21 / 0 / 0 | 5 (5) / 8 | 42 s; 17,528 in / 6,074 out; about US$0.10 |
| 2026-10-27 | RG-50.030.0002, Hilton, run 4 | as above | 17 | 35% (6/17), by main heading 47% (8/17); precision 60% (6/10) | 41% (7/17) | 47% (7/15) / 41% (7/17) | 20 / 0 / 0 | 7 (3) / 7 | 39 s; 52,315 in / 5,816 out; about US$0.16 |
| 2026-10-27 | RG-50.030.0001, Kochalski, run 2 | as above | 16 | 31% (5/16), by main heading 50% (8/16); precision 50% (5/10) | 50% (8/16) | 53% (8/15) / 50% (8/16) | 25 / 0 / 0 | 2 (0) / 12 | 50 s; 32,118 in / 7,367 out; about US$0.14 |
| All three (pooled) | 53 | 30% (16/53), by main heading 47% (25/53); precision 53% (16/30) | 42% (22/53) | 49% (22/45) / 42% (22/53) | 66 / 0 / 0 | 14 (8) / 27 | about US$0.40 in total |
Place mentions placed (gazetteer, GeoNames or approximate): 65 of 84 (Bomba), 126 of 171 (Hilton), 90 of 112 (Kochalski); 281 of 367 (77%) in all. Before v3, Bomba had no journey on the map and Hilton three.
What v3 shows
- The prompt changes did not move the model's own choices. Model-only exact matches: 16 of 53 (v3) against 17 of 53 (v2); precision 53% against 57%. Within the run-to-run variation already seen. The misses it targeted persist: "Łódź (Poland)" (Bomba, Hilton), emigration headings (Hilton, Kochalski), "Jews--Poland." (Kochalski). Some v2 matches were lost instead ("Diseases.", "Dentists." for Hilton), which suggests a slot limit: the model may suggest at most 12 headings, but the records hold 13 to 20 (mean 17.7). Next test: allow up to 20 and score by R-precision.
- Form headings by rule raise recall from 30% to 42% (22 of 53), at 49% precision over all suggestions. In-sample: the rules were designed from these records.
- Places improved most. Bomba: 5 of 8 journeys drawn (none before). Hilton: all 7 drawn, 4 of them approximate. Kochalski: 2 of 12, because the testimony itself rarely names his places. His journeys read "his home town (name not given in transcript)", "a forest where a German firm was cutting trees", "a farm in a village (not named)": the model reported the vagueness honestly instead of inventing names, and they are listed for a curator. Two names are transcribers' phonetic spellings, "Varka (ph)" and "Garbutka (ph)" (probably Warka and Garbatka, south of Warsaw), which the lookup cannot match yet.
- Validators: 66 quotes verified, none rejected, no heading outside the vocabulary.
Protocol v4 (from 2026-10-27): three follow-ups from v3
- Heading limit raised from 12 to 20 (
CURATORBOT_MAX_HEADINGS), with a note that records usually hold 13 to 20. Tests the slot-limit explanation for v3. New figure: model-only R-precision (top n of the model's own suggestions, n = catalogue headings), alongside precision@10. - Phonetic spellings: names a transcriber marked "(ph)" are matched approximately (common sound swaps, then close spellings among places in the collection's countries) and flagged uncertain with the spelling heard. GeoNames coverage widened to places of 500 people or more in those countries (Warka, Garbatka-Letnisko). Known weakness: close spelling can pick the wrong town ("Shedlets" gives Sedlec, CZ, not Siedlce); every such match is marked for a curator.
- Unnamed places ("a farm in a village", "his home town (name not given)") are counted separately from named places the lookup could not find, and labelled "Not named in the testimony".
| Date | Interview | Catalogue headings | Model suggestions | Model only: precision@10 / R-precision | With rules: R-precision / all-suggestion precision / recall | Quotes verified / rejected / outside vocabulary | Journeys drawn (exact) / total; not drawn: unnamed / not found | Time, tokens, cost |
|---|---|---|---|---|---|---|---|---|
| 2026-10-27 | RG-50.030.0034, Bomba, run 5 | 20 | 20 | 60% (6/10) / 60% (12/20) | 65% (13/20) / 61% (14/23) / 70% (14/20) | 30 / 0 / 0 | 8 (8) / 9; 0 / 1 | 53 s; 17,567 in / 8,308 out; about US$0.12 |
| 2026-10-27 | RG-50.030.0002, Hilton, run 5 | 17 | 20 | 70% (7/10) / 65% (11/17) | 59% (10/17) / 56% (13/23) / 76% (13/17) | 28 / 0 / 0 | 7 (3) / 7; 0 / 0 | 42 s; 52,354 in / 6,439 out; about US$0.17 |
| 2026-10-27 | RG-50.030.0001, Kochalski, run 3 | 16 | 19 | 60% (6/10) / 44% (7/16) | 50% (8/16) / 46% (10/22) / 62% (10/16) | 31 / 0 / 0 | 4 (2) / 11; 6 / 1 | 52 s; 32,157 in / 8,019 out; about US$0.14 |
| All three (pooled) | 53 | 59 | 63% (19/30) / 57% (30/53) | 58% (31/53) / 54% (37/68) / 70% (37/53) | 89 / 0 / 0 | 19 (13) / 27; 6 / 2 | about US$0.43 in total |
Derived from the rows above: the model's own 59 suggestions matched 32 catalogue headings exactly (60% recall, 54% precision); the 9 rule headings matched 5 more (Women--/Men--Personal narratives. in all three, Holocaust survivors--Interviews. for Bomba, Holocaust, Jewish (1939-1945)--Personal narratives. for Kochalski). Place mentions placed: 69, 130 and 91 of 84, 171 and 112 (290 of 367, 79%).
What v4 shows
- The slot limit was the constraint. With room for 20 suggestions instead of 12, recall over everything offered rose from 42% (22 of 53, v3) to 70% (37 of 53), and precision over everything offered rose too, from 49% to 54%: the extra suggestions were mostly right. The model's top-10 precision also rose (63%, against 53% and 57% before), though part of that may be run-to-run variation.
- Rule headings help recall but cost ranking. Ranked first (confidence 1.0), they displace correct model headings from the top n: with rules R-precision is 58% against 57% for the model alone. Rank rule headings after the model's, or show them separately, and score them separately.
- Places: Bomba 8 of 9 journeys drawn (5 of 8 in v3, none in v2). Kochalski 4 of 11; of the 7 not drawn, 6 involve places the testimony does not name, 1 a named place not found.
- Validators: 89 quotes verified, none rejected, no heading outside the vocabulary.
Caveat that matters most: all figures so far are vocabulary condition A: the catalogue's own headings were in the list the model chose from (49 reference headings among about 110). That measures whether the model picks the right headings from a list containing them, not how it would do against the museum's full authority file. Condition B (shared vocabulary only) and, later, the museum's authority export are needed before quoting a general figure.
Condition B: shared vocabulary only (2026-10-28)
Same settings as the v4 rows (protocol v4, Claude Sonnet 5.5, fallback off, spaCy trf, museum transcripts), with CURATORBOT_VOCAB_EXTRA unset: the model chose from the 67-heading demo list alone.
Ceiling. Only 11 of the 53 catalogue headings (21%) are in the shared list: Kochalski 4 of 16, Hilton 5 of 17, Bomba 2 of 20. Of the 11:
| Status in the shared list | Headings | Count |
|---|---|---|
| In the prototype list (2026-10-08), unchanged | United States--Emigration and immigration. (Kochalski, Hilton); Holocaust, Jewish (1939-1945)--Personal narratives. (Kochalski); Auschwitz (Concentration camp), Bergen-Belsen (Concentration camp) (Hilton) | 5 |
| In the prototype list in a superseded form, renamed 2026-10-27 | Jewish ghettos--Poland--Warsaw. (Kochalski); Jewish ghettos--Poland--Łódź. (Hilton) | 2 |
| Added 2026-10-27, taken from these records | Men--Personal narratives. (Kochalski); Women--Personal narratives. (Hilton, Bomba); Holocaust survivors--Interviews. (Bomba) | 4 |
Five of the 11 are form headings, supplied by rule; the model's own target is the other 6 (topical and place headings), none of them Bomba's.
| Interview | Catalogue headings / available | Model suggestions | Model only: exact, all suggestions (precision@10) | Model only: by main heading (top 10) | With rules: precision / recall exact | Quotes verified / rejected / outside vocabulary | Journeys drawn (exact) / total | Model time |
|---|---|---|---|---|---|---|---|---|
| RG-50.030.0034, Bomba, run 6 | 20 / 2 | 15 | 0 of 0 available (0%) | 30% (6/20) | 11% (2/18) / 10% (2/20) | 23 / 0 / 0 | 5 (5) / 7 | 45 s |
| RG-50.030.0002, Hilton, run 6 | 17 / 5 | 18 | 4 of 4 available (40%) | 47% (8/17) | 24% (5/21) / 29% (5/17) | 26 / 0 / 0 | 7 (3) / 7 | 36 s |
| RG-50.030.0001, Kochalski, run 4 | 16 / 4 | 15 | 2 of 2 available (10%) | 38% (6/16) | 22% (4/18) / 25% (4/16) | 24 / 0 / 0 | 3 (2) / 8 | 40 s |
| All three (pooled) | 53 / 11 | 48 | 6 of 6 available; precision over all model suggestions 12% (6/48) | 38% (20/53) | 19% (11/57) / 21% (11/53) | 73 / 0 / 0 | 15 (10) / 22 |
Matches are inferred from the counts: the model may not propose form headings (those come by rule), so its exact matches can only be among the 6 topical headings available, and the rules account for the other 5. Run figures are from the console output; briefs kept in private/runs/v4-B/.
What condition B shows
- Every catalogue heading that was available was found: 11 of 11 (6 by the model, 5 by rule). Recall equals the ceiling, so in this condition recall is set by the vocabulary, not the model.
- When the right heading is missing, the model offers the nearest one that is there. Precision falls from 54% (condition A, all suggestions with rules) to 19%, and main-heading recall stays at 38% (condition A, v3: 47%): many extras share the catalogue's main heading with a different subdivision. Bomba is the clearest case: no topical heading of hers is in the list, and the model still offered 15, all grounded in quotes. To review: whether these are defensible near-misses or stretches, and whether the model should offer fewer when the list fits badly.
- Grounding is unaffected: 73 quotes verified, none rejected, nothing outside the vocabulary.
- Places and journeys do not depend on the vocabulary and stay close to the A runs (the journey counts differ because each run's model writes its own journey list).
- Reading for the report: with a short generic list, CuratorBot's recall is capped by coverage (21% here), so the quality of the authority list matters more than the model. Condition C tests a realistic list.
Condition C: headings of the rest of the collection (2026-10-28)
Vocabulary: the subject, place, corporate-name and genre headings of the USHMM oral history collection on the EHRI portal (EAD 2002 export of us-005578-irn504457, downloaded 2026-10-28: 1,149 interviews), with the three test interviews' own records left out (matched by record number and title): 5,346 headings from 1,146 interviews, 1,786 of them used by two or more. It replaces the demo list (CURATORBOT_VOCABULARY); nothing was chosen with the test interviews in mind. Other settings as v4.
Ceiling: 40 of 53 catalogue headings (75%) remain available: Kochalski 14 of 16, Hilton 12 of 17, Bomba 14 of 20. The 13 that no other interview uses are local or unusual: Pabianice (3 headings), Radomsko, Częstochowa subdivisions (Weapons industry, Conscript labor, Abortion), the Warsaw orphanage headings, Ukraine--Armed Forces--Poland., Diseases., Dentists., and "Hugo Schneider Aktiengesellschaft." (our reference copies this as "Hugo Schneider Aktiengesellschaft."; the EHRI record reads "Hugo Schneider Aktiengesellschaft. (HASAG)": a copying difference, left as is in every condition so the conditions stay comparable).
| Interview | Catalogue headings / available | Model suggestions | Model only: precision@10 / R-precision (all suggestions) | Model only: by main heading (top 10) | With rules: precision / recall exact | Quotes verified / rejected / outside vocabulary | Journeys drawn (exact) / total | Time, tokens, cost |
|---|---|---|---|---|---|---|---|---|
| RG-50.030.0034, Bomba, run 7 | 20 / 14 | 19 | 50% (5/10) / 30% (6/20) | 45% (9/20) | 36% (8/22) / 40% (8/20) | 27 / 0 / 0 | 7 (6) / 7 | 52 s; 97,854 in / 7,808 out; about US$0.27 |
| RG-50.030.0002, Hilton, run 7 | 17 / 12 | 20 | 40% (4/10) / 24% (4/17) | 35% (6/17) | 22% (5/23) / 29% (5/17) | 28 / 0 / 0 | 7 (3) / 7 | 48 s; 132,641 in / 7,603 out; about US$0.34 |
| RG-50.030.0001, Kochalski, run 5 | 16 / 14 | 20 | 30% (3/10) / 19% (3/16) | 31% (5/16) | 22% (5/23) / 31% (5/16) | 30 / 0 / 0 | 2 (1) / 9 | 48 s; 112,444 in / 7,459 out; about US$0.30 |
| All three (pooled) | 53 / 40 | 59 | 40% (12/30) / 25% (13/53) | 38% (20/53) | 26% (18/68) / 34% (18/53) | 85 / 0 / 0 | 16 (10) / 23 | about US$0.91 in total |
Briefs kept in private/runs/v4-C/.
The three conditions side by side (pooled, three interviews)
| A: catalogue headings added (about 110) | B: demo list (67) | C: rest of the collection (5,346) | |
|---|---|---|---|
| Catalogue headings available | 53 of 53 | 11 of 53 (21%) | 40 of 53 (75%) |
| Model only: precision@10 | 63% | 17% (5 of 30) | 40% |
| With rules: precision, all suggestions | 54% | 19% | 26% |
| With rules: recall exact | 70% | 21% | 34% |
| Available headings found (with rules) | 37 of 53 (70%) | 11 of 11 (100%) | 18 of 40 (45%) |
| Quotes rejected | 0 of 89 | 0 of 73 | 0 of 85 |
| Model cost per interview | 12 to 17 US cents | about the same | 27 to 34 US cents |
What condition C shows
- With a realistic list, choosing becomes the bottleneck. Three quarters of the right headings were available, but CuratorBot found fewer than half of them (18 of 40). In A it found 70% of all headings from a short list; in B every available one. Among 5,346 options, the model more often picks a neighbour of the catalogue's heading: main-heading recall (38%) stays at the B level while exact recall is 34%.
- Precision of the top suggestions holds up better than recall: 40% of the model's top-10 suggestions exactly match the record (A: 63%), and the rest are drawn from headings cataloguers actually use for this collection.
- Grounding is unaffected by the list: 85 quotes verified, none rejected, nothing outside the vocabulary.
- Cost: the whole list goes into the prompt (about 80,000 extra input tokens), so each interview costs 27 to 34 cents instead of 12 to 17. Still small, but a full authority file would need a different design.
- This is the figure to quote as the realistic one so far, with A as the upper bound and B as the floor.
Next for condition C
- Look at the 22 available headings the model missed and what it chose instead (near-synonyms, broader or narrower forms).
--min-records 2(1,786 headings): does a shorter list of common headings help?- A shortlist step: retrieve the few hundred headings most related to the transcript (or let the model name topics freely, then map each to the nearest headings in the list) and offer only those. Also needed for the museum's full authority file.
- Repeat runs to measure run-to-run variation at this list size.
End-to-end test E1: from the recordings, no human transcript
Interview RG-50.030.0001 (Kochalski) run from the six video parts delivered in Box: faster-whisper large-v3 on a Colab T4, speaker roles from question marks, the automatic transcript analysed as it is (no correction), protocol v4 settings, condition A. Compared with the museum's transcript (curatorbot compare) and catalogue.
| Date | Audio length | Transcription time | WER vs museum transcript | Turns Q / A (automatic vs museum) | Place names kept | Model only: precision@10 / R-precision | With rules: precision / recall | Quotes verified / rejected | Journeys drawn / total | Model time, cost |
|---|---|---|---|---|---|---|---|---|---|---|
| 2026-10-28 | 02:58:44 (6 parts) | 23 min 38 s on a T4, including audio extraction (7.6 times faster than real time); 12 listening flags | 30.9% (17,359 museum words, 19,688 automatic) | 135 / 146 vs 123 / 123 | 14 of 14 (100%) | 50% (5/10) / 31% (5/16), 19 suggestions | 36% (8/22) / 50% (8/16) | 33 / 0 (0 outside vocabulary) | 4 (3 exact) / 13; not drawn: 4 unnamed, 5 not found | 65 s; 38,650 in / 9,909 out tokens; about US$0.18 |
Compare with RG-50.030.0001 run 3 (museum transcript, same settings): precision@10 60%, R-precision 44%, with rules 46% / 62%.
Place mentions placed: 106 of 114 (93%), against 91 of 112 (81%) from the museum transcript.
Place names kept: every one of the 14 distinct places the narrator names in the museum transcript also appears in the automatic transcript. A first count gave 23 of 25 with two "missed", "Ghetto Warsaw. It" and "Warsaw. There": the comparison read capitalised words across a full stop and counted one place once per spelling. Fixed (names read within sentences; one place said in several forms counts once) and rerun on the same transcripts.
What E1 shows
- The pipeline runs end to end from video with no human transcript: six parts, three hours of audio, a brief with every quote verified (33 of 33) against the automatic transcript. About 25 minutes in all on a Colab T4 (24 minutes transcription, about a minute for the brief), and about 18 US cents for the model. GPU time is not costed here.
- Heading quality dropped a little, within run-to-run variation. Model-only exact matches 5 of 16 against 7 of 16 from the museum transcript; with rules 8 against 10. Earlier repeat runs on the same transcript moved by one or two matches, so one run each cannot separate the effect of transcription errors from ordinary variation. Repeat runs on both transcripts are needed before claiming a cost.
- No place name was lost in transcription: all 14 places named in the museum transcript are in the automatic one.
- More places were placed from the automatic transcript (93% against 81%). Whisper writes names in their standard spellings, where the museum transcript has phonetic forms marked "(ph)" or blanks. To check by listing the place names where the two transcripts differ.
- Word error rate 30.9% overstates the errors. The automatic transcript has 13% more words: the museum transcript is lightly edited (false starts, repetitions and some interjections left out), and every omission counts as an error. A fairer figure needs a verbatim reference for a sample of passages.
- Speaker turns are found from question marks only: 135 questions and 146 answers against 123 each. Turn errors move some interviewer speech into narrator turns and back; the validators still hold, because quotes are checked against the text whatever its role.
Readings for slides
Use the v4 figures; earlier ones are kept for the record. Always state the condition-A caveat with heading figures.
- Headline: Across three released USHMM interviews, CuratorBot proposed headings that matched 70% of the museum's own catalogue headings exactly (37 of 53), and 54% of everything it proposed was an exact match. Every one of its 89 quotes was verified against the transcript. Each interview took under a minute and cost 12 to 17 US cents. (Catalogue headings were included in the vocabulary offered to the model; three interviews; transcripts unverified by the museum.)
- Top suggestions: 63% of the model's top-10 suggestions matched the catalogue exactly (53% to 57% in two earlier runs).
- How we got there: an error analysis of the first runs showed the model was limited to 12 suggestions while records hold 13 to 20; lifting the limit raised recall from 42% to 70% without lowering precision.
- Places: 79% of place mentions placed on the map; 19 of 27 journeys drawn. Of the 8 not drawn, 6 involve places the testimony never names; they are listed for a curator, not guessed. Phonetic spellings ("Varka (ph)") are matched and flagged as uncertain.
- Many "wrong" suggestions are defensible: e.g. Righteous Gentiles in the Holocaust ("But he was a good man, and he hid me with neighbors.") for Kochalski. A cataloguer's judgement would likely raise precision.
- Safeguard: when a weaker fallback model answered, the validators rejected 9 of its 18 quoted items.
- Limits to state: condition A vocabulary; three interviews; form-heading rules designed on the same records; museum transcripts unverified, with errors ("Pomortsy" for Pabianice, a blank hometown, phonetic spellings).
- Condition B (shared demo list only): when the catalogue's headings were taken out of the list, only 11 of 53 remained available, and CuratorBot found all 11. With the right headings missing it offered the nearest available ones (precision 19%), every one still backed by a verified quote. The vocabulary, not the model, sets the limit. (Demo list partly shaped by these records.)
- Condition C (realistic list, the figure to quote): offered the 5,346 headings used across the rest of the USHMM oral history collection, CuratorBot matched 34% of the catalogue's headings exactly (18 of 53), with 40% of its top-10 suggestions exact matches and every quote verified, for about 30 US cents an interview. Three quarters of the headings were available; finding the right one among thousands is now the limit, which a shortlist step should address. (Three interviews; one run each.)
- End to end (E1, one interview): from three hours of video with no human transcript, CuratorBot produced a brief in about 25 minutes on a Colab T4 GPU for about 18 US cents of model use. Every place the narrator named survived automatic transcription (14 of 14), 93% of place mentions were placed on the map, and all 33 quotes were verified. Heading matches were slightly lower than from the museum's transcript (5 against 7 of 16, model only), within the variation seen between repeat runs. (Condition A; one interview; one run each.)
Notes on misses
RG-50.030.0034, run 2
| Kind | Catalogue headings | Reading |
|---|---|---|
| Form and genre headings | Women--Personal narratives.; Holocaust survivors--Interviews.; Holocaust, Jewish (1939-1945)--Poland--Personal narratives. | Added by cataloguing convention; nothing in the testimony points to them |
| Broad period heading | Poland--History--Occupation, 1939-1945. | Content miss: the occupation runs through the testimony |
| Named-place heading | Litzmannstadt-Getto (Łódź, Poland) | To check whether the narrator was in the Łódź ghetto or only born in Łódź |
| Narrow topical heading | Ukraine--Armed Forces--Poland. | Probably a brief passage |
| Right main heading, different subdivision | Jews--Poland--Czestochowa.; World War, 1939-1945--Campaigns--Poland.; World War, 1939-1945--Conscript labor--Poland--Czestochowa. | Near misses |
Suggestions not in the catalogue, with their evidence:
- Soviet Union. Red Army (p017): "I saw Russian tanks coming up to the city, waving to us"
- Partisans (p011): "a lot of partisans, walked into that group to go into the ghetto"
Both are grounded in the testimony; whether they belong in the record is a curator's judgement, and the scores count them against CuratorBot.
RG-50.030.0002, run 1
| Kind | Catalogue headings | Reading |
|---|---|---|
| Transcript error | Pabianice (Poland); Jewish ghettos--Poland--Pabianice.; Jews--Poland--Pabianice. | The museum transcript renders Pabianice as "Pomortsy", so the name never appears. All three missed. A limit for any text-based method; a case for listening flags and gazetteer variants |
| Heading form | Jewish ghettos--Poland--Łódź. | CuratorBot chose the superseded form "Ghettos, Jewish--Poland--Łódź" from the demo vocabulary, which holds both. Fix the vocabulary to current forms |
| Content misses | Łódź (Poland); Poland--Emigration and immigration.; United States--Emigration and immigration. | Present in the testimony; the 10 slots went to camp and survivor headings instead |
| Form heading | Women--Personal narratives. | Convention, as in RG-50.030.0034 |
Suggestions counted as extras:
- Auschwitz (Concentration camp) (p122) and Bergen-Belsen (Concentration camp) (p154): in the record as corporate names; correct under reference v2.
- Holocaust survivors (p002): plausible, not in the record.
- Ghettos, Jewish--Poland--Łódź (p090), quote "I worked in a brassiere factory.": right topic, superseded form, and weak evidence. The validators check that a quote exists, not that it supports the heading.
Reference v2 runs
RG-50.030.0034, run 3: 1 of 4 corporate names found (HASAG Czestochowianka); missed Hugo Schneider Aktiengesellschaft., Germany. Gestapo., Czestochowa (Concentration camp). "Łódź (Poland)" and "Weapons industry--Poland--Czestochowa." were matched in run 2 but not here: run-to-run variation, possibly increased by the larger vocabulary. Extras: 1939-1945 (p006; a bare chronological term in the demo vocabulary, not a usable heading), Soviet Union. Red Army (p017), Treblinka (Concentration camp) (p006: "They were shipped to Treblinka because they needed another hundred men for the transport").
RG-50.030.0002, run 3: the vocabulary fix worked ("Jewish ghettos--Poland--Łódź." now matched exactly), and Auschwitz and Bergen-Belsen now count. Still missed: the two Pabianice headings (transcript error), "Łódź (Poland)", both emigration headings, "Women--Personal narratives.", "Birkenau (Concentration camp)", "Red Cross and Red Crescent.". Extras: Great Britain. Army (p237, quote "The liberators treated us very nice.": historically right for Bergen-Belsen, but the quote does not name the British), Holocaust survivors--United States (p255: "May 1946 we came here.").
RG-50.030.0001, run 1: exact matches: Getto warszawskie (Warsaw, Poland); Jewish orphanages--Poland--Warsaw.; Passing (Identity)--Poland.; Jewish children in the Holocaust--Poland. Main heading only: World War, 1939-1945--Jewish resistance--Poland.; World War, 1939-1945--Children--Poland. Missed: form headings (Men--Personal narratives.; Holocaust survivors--United States--Interviews.; Holocaust, Jewish (1939-1945)--Personal narratives.), both emigration headings, Jews--Poland., Jewish ghettos--Poland--Warsaw., Jewish orphans--Poland--Warsaw., Warsaw (Poland)--History--Warsaw Ghetto Uprising, 1943., Hiding places--Poland--Warsaw. Extras: 1939-1945 (p034, again a bare term, confidence 0.9), Antisemitism (p016), Hidden children (Holocaust) (p214: "Hiding through the rest of the war is the answer."), Righteous Gentiles in the Holocaust (p188: "But he was a good man, and he hid me with neighbors."). The one stopped item was not a bad quote: Claude proposed "Poland--Emigration and immigration" (a catalogue heading, with a correct quote at p238: "Eventually a year later I escaped from Poland and made my way to Germany.") without the vocabulary's final full stop, and the validator compared strings exactly. Validator fixed (headings now match ignoring case, spacing and a final full stop) and the run rescored without a new model call: the heading is now accepted, but at confidence 0.5 it ranks 11th of 11 valid suggestions, just below the top-10 cut, so the scores do not change. With k equal to the number of catalogue headings (16) it would count, which supports moving to R-precision.
Patterns across interviews (reference v2, latest runs)
| Pattern | Headings missed | Interviews | Candidate fix |
|---|---|---|---|
| Form and genre headings (personal narratives, interviews) | 7 | all three | Propose by rule from interview type and narrator |
| Postwar emigration (Poland--, United States--Emigration and immigration.) | 4 | Hilton, Kochalski | Prompt to cover the whole life course, including after liberation |
| Broad place or people headings (Łódź (Poland); Jews--Poland.) | 3 | all three | Prompt to pair each specific heading with its broader place heading |
| Transcript errors and gaps | 2 (Pabianice) | Hilton | Listening flags; gazetteer variants; ASR comparison |
| Near-synonyms in the vocabulary (Getto warszawskie / Jewish ghettos--Poland--Warsaw; Hiding places / Hidden children) | Kochalski | Cataloguers use both; suggest related headings together | |
| Bare term "1939-1945" taking a top slot | 2 extras | Bomba, Kochalski | Remove from the vocabulary |
Open methodological points
Next: run condition B (shared vocabulary only,
CURATORBOT_VOCAB_EXTRAunset) on the same three interviews, so the gap between the two conditions is measured.Rank rule headings after the model's suggestions (or keep them out of the ranked list) so they do not displace correct model headings.
Raise the model's heading limit from 12 towards the catalogue's typical 15 to 20, and compare by R-precision.
Match transcribers' phonetic spellings marked "(ph)" (e.g. "Varka" for Warka) with a fuzzy lookup, flagged as uncertain for a curator.
Report journeys with unnamed places ("a farm in a village") separately from named places the lookup could not find.
Clean the demo vocabulary: remove bare terms such as "1939-1945" and any remaining superseded forms; replace it with the museum's authority export when available.
Evaluation runs should stop rather than fall back to the second model, or be excluded when they do (as RG-50.030.0002 run 2 was). Add an option to
prepare_interview.pyto switch the fallback off for evaluation."Łódź (Poland)" was missed in both interviews' latest runs although Łódź is central to both: check whether the model avoids broad place headings when a narrower ghetto heading is chosen.
Check evidence relevance as well as existence (e.g. require the quote to mention the heading's place or topic, or have a second model judge support), and report weakly supported headings.
Candidate change: propose form and genre headings (e.g. "Holocaust survivors--Interviews.", "Women--Personal narratives.") by rule from the interview type and narrator, rather than asking the model. Test on all evaluation interviews before adopting, not on RG-50.030.0034 alone.
Ask a USHMM cataloguer to judge the extras, so precision can also be reported against an expert's view rather than the record alone.
Replace the fixed top-10 cut-off with k = number of catalogue headings (R-precision), and report the ceiling alongside recall.
Report condition A and condition B for each interview.
Use the museum's authority export as the vocabulary once available; record its version here.
Grow the sample before claiming a general figure; report per-interview results and the mean.