Spaces:
Running
Running
Commit History
7 days ago
-
Condition C harvest: find test interviews by record number or title as well as accession number
The EHRI export does not carry the accession numbers the catalogue files use, so nothing was excluded. Test interviews are now matched by accession number, the irn record number in the catalogue URL, or exact title (which also catches a second record of the same interview); excluded records are listed. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEc7c7f6c -
Evaluation condition C: vocabulary from the rest of the catalogued collection
scripts/harvest_ehri_headings.py reads the EHRI portal's EAD export of a collection (USHMM oral histories, about 1,150 interviews), collects each interview's subject, place, corporate and genre headings, leaves out the interviews under test, and writes a vocabulary with a report of how many of each test interview's headings remain available. CURATORBOT_VOCABULARY replaces the bundled demo list with such a file (or, later, an authority export). Method notes for conditions A, B and C in the evaluation log. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE15f22eb -
Progress reports for the analysis step
Analysing a three-hour transcript takes several minutes with no output. It now reports each stage: loading the language model, finding places, people and events turn by turn (with the time left), the model drafting the brief (a "still working" line every 30 seconds until it answers), checking quotes and headings, and placing. The notebook's analyse cell keeps the latest state in <ID>/out/progress.json like the transcription cell. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE8c6b0a2 -
Progress reports for long runs; notebook shows command output and the real error
- New curatorbot/progress.py: timestamped stage lines and a progress.json. - Audio extraction reports each part; transcription reports every 30 s with the time left; language detection reports its stretches. - A speech model that will not load says whether GPU libraries are the cause. - prepare_interview.py --progress-file; the notebook keeps progress in Drive (<ID>/out/progress.json) and streams command output, so failures show their reason instead of only an exit code. - Runbook: progress, and Lancaster's Hex cluster for larger batches. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE34aa951 -
Take Box deliveries as they come: inventory, recordings at any depth, batch notebook
- New curatorbot/intake.py and 'curatorbot inventory': one entry per submission folder, recordings found at any depth, missing parts, other files, durations; ids from accession numbers (with -BOX) or folder names. - prepare_interview.py --media accepts folders. - Notebook rewritten around CuratorBot/box/ (Box folders as downloaded): inventory, choose submissions, transcribe (resumable), analyse as-is or after review, compare; spaces in folder names handled. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE68f1146
8 days ago
-
Heading limit 20, phonetic place names, unnamed places counted separately
- CURATORBOT_MAX_HEADINGS (default 20, was a fixed 12); back-test adds the model's own R-precision. - Places marked '(ph)' are matched approximately (sound swaps, then close spellings in the collection's countries) and flagged uncertain with the spelling heard. GeoNames index widened to places of 500+ people in those countries; nearest place to a named anchor wins ties. - Journeys with unnamed places are labelled 'Not named in the testimony' and counted apart from named places not found. - Docs: protocol v4 in the evaluation log, schema, .env example. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE162cea3 -
Places, headings and scoring improvements from the first evaluation
Places - New curatorbot/places.py: period gazetteer first, then GeoNames towns (offline, diacritic-insensitive, alternate names), then approximate placement near a place the narrator names ('Pomortsy, near Lodz') or at a country centre. Each place records its source and precision. - Journeys gain fromNear/toNear; none is dropped silently: unplaced ends are shown as 'Not placed: needs a curator'. Coverage counts per brief. - Map frames the journeys, avoids label collisions, marks approximate places with dashed rings and points off-map arrows towards their destination. Headings - Prompt covers the whole life, pairs specific and broader place headings, and leaves generic form headings to a rule. - Form headings proposed by rule (data/form_headings.json), using a grounded narrator-gender quote; labelled 'Added by rule' in the viewer. Vocabulary and scoring - Base vocabulary: bare chronological terms removed, current 'Jewish ghettos--' forms, generic form headings added. E…3861ad6
22 days ago
-
Real-data demo: intake queue, catalogue comparison, private interview data
- Intake queue as the landing page: new submissions first, then released interviews with their catalogue scores, then the synthetic example - Interview status (under review, released, synthetic) in briefs, CLI and UI - Catalogue comparison tab for released interviews: headings found, missed (separating vocabulary gaps from model misses) and extra suggestions with their evidence - Real briefs load from a private Hugging Face dataset at start-up (and on /api/reload), so partner material never enters the code repository - Importer for the museum's released PDF transcripts (speaker initials, page headers and footers), so released interviews skip transcription - Runbook for the two data tracks; data policy and deployment guide updated; Colab notebook takes a status and a donor or catalogue file Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEc1d045b
27 days ago
-
CuratorBot prototype: pipeline, hosted demo and project kit
- curatorbot package: Q/A transcripts, spatio-textual 0.4.1 annotation, period gazetteer with review reasons, Anthropic/OpenAI agent with quote and vocabulary validators, heading back-test, CLI - FastAPI app and viewer for a Hugging Face Docker Space, with passcode- protected live analysis and optional short-clip transcription - Synthetic demonstration interview (DEMO-0001) - Tooling for released interviews: GPU prepare script and Colab notebook - Docs (deployment, data policy, brief schema) and a project kit for the Claude and ChatGPT projects; CI and Space deploy workflows Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE3bf43cb