Spaces:
Running
Running
Commit History
7 days ago
-
Evaluation log: condition C (5,346 headings from the rest of the collection) and the three conditions compared
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEab3b5ef -
Evaluation log: condition B results on the three released interviews
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE2b1974a -
Evaluation condition C: vocabulary from the rest of the catalogued collection
scripts/harvest_ehri_headings.py reads the EHRI portal's EAD export of a collection (USHMM oral histories, about 1,150 interviews), collects each interview's subject, place, corporate and genre headings, leaves out the interviews under test, and writes a vocabulary with a report of how many of each test interview's headings remain available. CURATORBOT_VOCABULARY replaces the bundled demo list with such a file (or, later, an authority export). Method notes for conditions A, B and C in the evaluation log. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE15f22eb -
Evaluation log: E1 timings and cost; slide reading for the end-to-end test
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE15bf067 -
Evaluation log: E1 place names kept, 14 of 14 after the comparison fix
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE1a33936 -
Compare: place names stop at sentence ends; evaluation log: end-to-end test E1
The place-name check read capitalised words across a full stop ("Warsaw. There"), so one place was counted several times and reported as missed. Names are now read within sentences and one place said in several forms counts once. E1 results logged: the Kochalski interview from the six Box video parts, analysed from the automatic transcript. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEf8b32e0
8 days ago
-
End-to-end path: Colab notebook for recordings, transcript comparison, faster language ID
- notebooks/prepare_on_colab.ipynb rewritten: new id for the Box copy, evaluation settings, transcript saved to Drive at once, analyse as-is or after review, compare with the museum transcript. - New 'curatorbot compare': WER (jiwer), turns, and how many place names in the reference narrator turns survive automatic transcription. - Language ID once per 30-second stretch instead of per chunk (a few hundred passes for a three-hour interview, not thousands). - Rule headings ranked after the model's in the back-test. - Runbook section D and evaluation-log section E1 for the end-to-end test. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEcf6fdf5 -
Evaluation log: protocol v4 results, revised slide reading, condition-A caveat
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEaeeb37c -
Heading limit 20, phonetic place names, unnamed places counted separately
- CURATORBOT_MAX_HEADINGS (default 20, was a fixed 12); back-test adds the model's own R-precision. - Places marked '(ph)' are matched approximately (sound swaps, then close spellings in the collection's countries) and flagged uncertain with the spelling heard. GeoNames index widened to places of 500+ people in those countries; nearest place to a named anchor wins ties. - Journeys with unnamed places are labelled 'Not named in the testimony' and counted apart from named places not found. - Docs: protocol v4 in the evaluation log, schema, .env example. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE162cea3 -
Evaluation log: protocol v3 results for all three interviews, findings and next tests
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE22166ec -
Places, headings and scoring improvements from the first evaluation
Places - New curatorbot/places.py: period gazetteer first, then GeoNames towns (offline, diacritic-insensitive, alternate names), then approximate placement near a place the narrator names ('Pomortsy, near Lodz') or at a country centre. Each place records its source and precision. - Journeys gain fromNear/toNear; none is dropped silently: unplaced ends are shown as 'Not placed: needs a curator'. Coverage counts per brief. - Map frames the journeys, avoids label collisions, marks approximate places with dashed rings and points off-map arrows towards their destination. Headings - Prompt covers the whole life, pairs specific and broader place headings, and leaves generic form headings to a rule. - Form headings proposed by rule (data/form_headings.json), using a grounded narrator-gender quote; labelled 'Added by rule' in the viewer. Vocabulary and scoring - Base vocabulary: bare chronological terms removed, current 'Jewish ghettos--' forms, generic form headings added. E…3861ad6 -
Evaluation log: RG-50.030.0001 rescored after the validator fix
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEec02bc7 -
Validators: match headings ignoring a final full stop; count bad quotes and out-of-vocabulary headings separately
- 'Poland--Emigration and immigration' was stopped because the vocabulary form ends with a full stop. Headings now match ignoring case, spacing and a final full stop, and are stored in the vocabulary's own form. - validation now records quotes_rejected and out_of_vocabulary as well as the combined 'rejected'; How it works shows both. - New 'curatorbot revalidate' command reapplies the validators and the back-test to a saved brief without a model call. - The page now loads the viewer's script and styles with a version tag, so browsers pick up a new deploy without a hard refresh. - Evaluation log: corrected the quote claim and explained the stopped item. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE05cb6a1 -
Evaluation log: RG-50.030.0001 run 1, three-interview pooled figure, patterns across interviews
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE24bc20d -
Evaluation log: fill quote counts for RG-50.030.0002 run 3
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEcf98315 -
Evaluation log: reference v2 results for RG-50.030.0034 and RG-50.030.0002, pooled figure, excluded fallback run
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEd8eed0e
9 days ago
-
Evaluation log: RG-50.030.0002 run 1, reference v1 caveat, transcript and heading-form findings
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE153fd5b -
Evaluation log: RG-50.030.0034 run 2, corrected precision note, analysis of misses
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE7bade8d -
Evaluation log: confirm model, time and cost for RG-50.030.0034
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEbe9930d -
Evaluation log: method and first back-test result (RG-50.030.0034)
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEfaf6d4a