Spaces:
Running
Running
Commit History
7 days ago
-
README as a standard project README; Space configuration moved to a Space card
GitHub showed the Hugging Face Space settings as a table at the top of the README. They now live in space/README.md, which the deploy step puts in place of README.md only in the copy pushed to the Space. The README describes the project, its features, pipeline, installation, commands, configuration, evaluation, layout and principles. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE55cd90c -
Evaluation log: condition C (5,346 headings from the rest of the collection) and the three conditions compared
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEab3b5ef -
Evaluation log: condition B results on the three released interviews
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE2b1974a -
Evaluation condition C: vocabulary from the rest of the catalogued collection
scripts/harvest_ehri_headings.py reads the EHRI portal's EAD export of a collection (USHMM oral histories, about 1,150 interviews), collects each interview's subject, place, corporate and genre headings, leaves out the interviews under test, and writes a vocabulary with a report of how many of each test interview's headings remain available. CURATORBOT_VOCABULARY replaces the bundled demo list with such a file (or, later, an authority export). Method notes for conditions A, B and C in the evaluation log. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE15f22eb -
Evaluation log: E1 timings and cost; slide reading for the end-to-end test
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE15bf067 -
Evaluation log: E1 place names kept, 14 of 14 after the comparison fix
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE1a33936 -
Compare: place names stop at sentence ends; evaluation log: end-to-end test E1
The place-name check read capitalised words across a full stop ("Warsaw. There"), so one place was counted several times and reported as missed. Names are now read within sentences and one place said in several forms counts once. E1 results logged: the Kochalski interview from the six Box video parts, analysed from the automatic transcript. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEf8b32e0 -
Transcription on GPU: provide the CUDA 12 libraries faster-whisper needs
Colab's system CUDA is now newer than the CUDA 12 that faster-whisper's engine (CTranslate2) is built for, so libcublas.so.12 was missing at the first GPU step. The setup cell installs nvidia-cublas-cu12 and cuDNN 9 from pip; CuratorBot loads them before the speech model, and the notebook also puts them on LD_LIBRARY_PATH for the run. A missing GPU library during transcription now gets the same explanation as one during model loading. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEdca0e6c -
Progress reports for long runs; notebook shows command output and the real error
- New curatorbot/progress.py: timestamped stage lines and a progress.json. - Audio extraction reports each part; transcription reports every 30 s with the time left; language detection reports its stretches. - A speech model that will not load says whether GPU libraries are the cause. - prepare_interview.py --progress-file; the notebook keeps progress in Drive (<ID>/out/progress.json) and streams command output, so failures show their reason instead of only an exit code. - Runbook: progress, and Lancaster's Hex cluster for larger batches. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE34aa951 -
Open in Colab badge on the notebook and in the README
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE3883b9b -
Take Box deliveries as they come: inventory, recordings at any depth, batch notebook
- New curatorbot/intake.py and 'curatorbot inventory': one entry per submission folder, recordings found at any depth, missing parts, other files, durations; ids from accession numbers (with -BOX) or folder names. - prepare_interview.py --media accepts folders. - Notebook rewritten around CuratorBot/box/ (Box folders as downloaded): inventory, choose submissions, transcribe (resumable), analyse as-is or after review, compare; spaces in folder names handled. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE68f1146
8 days ago
-
End-to-end path: Colab notebook for recordings, transcript comparison, faster language ID
- notebooks/prepare_on_colab.ipynb rewritten: new id for the Box copy, evaluation settings, transcript saved to Drive at once, analyse as-is or after review, compare with the museum transcript. - New 'curatorbot compare': WER (jiwer), turns, and how many place names in the reference narrator turns survive automatic transcription. - Language ID once per 30-second stretch instead of per chunk (a few hundred passes for a three-hour interview, not thousands). - Rule headings ranked after the model's in the back-test. - Runbook section D and evaluation-log section E1 for the end-to-end test. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEcf6fdf5 -
Evaluation log: protocol v4 results, revised slide reading, condition-A caveat
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEaeeb37c -
Heading limit 20, phonetic place names, unnamed places counted separately
- CURATORBOT_MAX_HEADINGS (default 20, was a fixed 12); back-test adds the model's own R-precision. - Places marked '(ph)' are matched approximately (sound swaps, then close spellings in the collection's countries) and flagged uncertain with the spelling heard. GeoNames index widened to places of 500+ people in those countries; nearest place to a named anchor wins ties. - Journeys with unnamed places are labelled 'Not named in the testimony' and counted apart from named places not found. - Docs: protocol v4 in the evaluation log, schema, .env example. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE162cea3 -
Evaluation log: protocol v3 results for all three interviews, findings and next tests
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE22166ec -
Places, headings and scoring improvements from the first evaluation
Places - New curatorbot/places.py: period gazetteer first, then GeoNames towns (offline, diacritic-insensitive, alternate names), then approximate placement near a place the narrator names ('Pomortsy, near Lodz') or at a country centre. Each place records its source and precision. - Journeys gain fromNear/toNear; none is dropped silently: unplaced ends are shown as 'Not placed: needs a curator'. Coverage counts per brief. - Map frames the journeys, avoids label collisions, marks approximate places with dashed rings and points off-map arrows towards their destination. Headings - Prompt covers the whole life, pairs specific and broader place headings, and leaves generic form headings to a rule. - Form headings proposed by rule (data/form_headings.json), using a grounded narrator-gender quote; labelled 'Added by rule' in the viewer. Vocabulary and scoring - Base vocabulary: bare chronological terms removed, current 'Jewish ghettos--' forms, generic form headings added. E…3861ad6 -
Evaluation log: RG-50.030.0001 rescored after the validator fix
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEec02bc7 -
Validators: match headings ignoring a final full stop; count bad quotes and out-of-vocabulary headings separately
- 'Poland--Emigration and immigration' was stopped because the vocabulary form ends with a full stop. Headings now match ignoring case, spacing and a final full stop, and are stored in the vocabulary's own form. - validation now records quotes_rejected and out_of_vocabulary as well as the combined 'rejected'; How it works shows both. - New 'curatorbot revalidate' command reapplies the validators and the back-test to a saved brief without a model call. - The page now loads the viewer's script and styles with a version tag, so browsers pick up a new deploy without a hard refresh. - Evaluation log: corrected the quote claim and explained the stopped item. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE05cb6a1 -
Evaluation log: RG-50.030.0001 run 1, three-interview pooled figure, patterns across interviews
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE24bc20d -
Evaluation log: fill quote counts for RG-50.030.0002 run 3
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEcf98315 -
Evaluation log: reference v2 results for RG-50.030.0034 and RG-50.030.0002, pooled figure, excluded fallback run
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEd8eed0e -
Raise the default model output limit to 16000 tokens
A full interview brief (RG-50.030.0002) needed about 6,000 output tokens, so the old 6000 limit cut Claude's answer off and the run fell back to the second model. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEde2ecdc
9 days ago
-
Evaluation log: RG-50.030.0002 run 1, reference v1 caveat, transcript and heading-form findings
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE153fd5b -
Evaluation log: RG-50.030.0034 run 2, corrected precision note, analysis of misses
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE7bade8d -
Evaluation log: confirm model, time and cost for RG-50.030.0034
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEbe9930d -
Evaluation log: method and first back-test result (RG-50.030.0034)
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEfaf6d4a
19 days ago
-
Model providers: Claude via OpenRouter by default, OpenAI fallback, any OpenAI-compatible endpoint
- New default provider openrouter (anthropic/claude-sonnet-5.5), with openai/gpt-4.1-mini tried once on outages, rate limits or unreadable answers; account errors (bad key, no credit) are not retried. - New 'compatible' provider for an institution's own gateway or local open models (vLLM, Ollama) via CURATORBOT_LLM_BASE_URL. - Plain-English errors for curators; details go to the server log. - /api/health and the How it works tab show the model and any fallback. - docs/model-providers.md explains the choice and data protection. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEfa829bf
20 days ago
-
Site-wide passcode: share a public Space with the team without exposing interviews
CURATORBOT_SITE_PASSCODE puts every page and API call behind a sign-in page (14-day cookie, cleared when the passcode changes). Scripts can use the X-Site-Passcode header. Docs explain how to share the Space safely. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEbe9517e
22 days ago
-
Use the Hugging Face username (ignatius) for the Space and dataset
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE96a766a -
Real-data demo: intake queue, catalogue comparison, private interview data
- Intake queue as the landing page: new submissions first, then released interviews with their catalogue scores, then the synthetic example - Interview status (under review, released, synthetic) in briefs, CLI and UI - Catalogue comparison tab for released interviews: headings found, missed (separating vocabulary gaps from model misses) and extra suggestions with their evidence - Real briefs load from a private Hugging Face dataset at start-up (and on /api/reload), so partner material never enters the code repository - Importer for the museum's released PDF transcripts (speaker initials, page headers and footers), so released interviews skip transcription - Runbook for the two data tracks; data policy and deployment guide updated; Colab notebook takes a status and a donor or catalogue file Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE9b35635 -
Deployment guide: Docker Spaces now need Hugging Face PRO; host on the personal account
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEffc24b0
27 days ago
-
CuratorBot prototype: pipeline, hosted demo and project kit
- curatorbot package: Q/A transcripts, spatio-textual 0.4.1 annotation, period gazetteer with review reasons, Anthropic/OpenAI agent with quote and vocabulary validators, heading back-test, CLI - FastAPI app and viewer for a Hugging Face Docker Space, with passcode- protected live analysis and optional short-clip transcription - Synthetic demonstration interview (DEMO-0001) - Tooling for released interviews: GPU prepare script and Colab notebook - Docs (deployment, data policy, brief schema) and a project kit for the Claude and ChatGPT projects; CI and Space deploy workflows Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE3bf43cb