Spaces:
Running
Running
Commit History
8 days ago
-
Evaluation condition C: vocabulary from the rest of the catalogued collection
scripts/harvest_ehri_headings.py reads the EHRI portal's EAD export of a collection (USHMM oral histories, about 1,150 interviews), collects each interview's subject, place, corporate and genre headings, leaves out the interviews under test, and writes a vocabulary with a report of how many of each test interview's headings remain available. CURATORBOT_VOCABULARY replaces the bundled demo list with such a file (or, later, an authority export). Method notes for conditions A, B and C in the evaluation log. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE15f22eb -
Transcription on GPU: provide the CUDA 12 libraries faster-whisper needs
Colab's system CUDA is now newer than the CUDA 12 that faster-whisper's engine (CTranslate2) is built for, so libcublas.so.12 was missing at the first GPU step. The setup cell installs nvidia-cublas-cu12 and cuDNN 9 from pip; CuratorBot loads them before the speech model, and the notebook also puts them on LD_LIBRARY_PATH for the run. A missing GPU library during transcription now gets the same explanation as one during model loading. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEdca0e6c -
Progress reports for long runs; notebook shows command output and the real error
- New curatorbot/progress.py: timestamped stage lines and a progress.json. - Audio extraction reports each part; transcription reports every 30 s with the time left; language detection reports its stretches. - A speech model that will not load says whether GPU libraries are the cause. - prepare_interview.py --progress-file; the notebook keeps progress in Drive (<ID>/out/progress.json) and streams command output, so failures show their reason instead of only an exit code. - Runbook: progress, and Lancaster's Hex cluster for larger batches. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE34aa951 -
Open in Colab badge on the notebook and in the README
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE3883b9b -
Take Box deliveries as they come: inventory, recordings at any depth, batch notebook
- New curatorbot/intake.py and 'curatorbot inventory': one entry per submission folder, recordings found at any depth, missing parts, other files, durations; ids from accession numbers (with -BOX) or folder names. - prepare_interview.py --media accepts folders. - Notebook rewritten around CuratorBot/box/ (Box folders as downloaded): inventory, choose submissions, transcribe (resumable), analyse as-is or after review, compare; spaces in folder names handled. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE68f1146
9 days ago
-
End-to-end path: Colab notebook for recordings, transcript comparison, faster language ID
- notebooks/prepare_on_colab.ipynb rewritten: new id for the Box copy, evaluation settings, transcript saved to Drive at once, analyse as-is or after review, compare with the museum transcript. - New 'curatorbot compare': WER (jiwer), turns, and how many place names in the reference narrator turns survive automatic transcription. - Language ID once per 30-second stretch instead of per chunk (a few hundred passes for a three-hour interview, not thousands). - Rule headings ranked after the model's in the back-test. - Runbook section D and evaluation-log section E1 for the end-to-end test. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEcf6fdf5 -
Places, headings and scoring improvements from the first evaluation
Places - New curatorbot/places.py: period gazetteer first, then GeoNames towns (offline, diacritic-insensitive, alternate names), then approximate placement near a place the narrator names ('Pomortsy, near Lodz') or at a country centre. Each place records its source and precision. - Journeys gain fromNear/toNear; none is dropped silently: unplaced ends are shown as 'Not placed: needs a curator'. Coverage counts per brief. - Map frames the journeys, avoids label collisions, marks approximate places with dashed rings and points off-map arrows towards their destination. Headings - Prompt covers the whole life, pairs specific and broader place headings, and leaves generic form headings to a rule. - Form headings proposed by rule (data/form_headings.json), using a grounded narrator-gender quote; labelled 'Added by rule' in the viewer. Vocabulary and scoring - Base vocabulary: bare chronological terms removed, current 'Jewish ghettos--' forms, generic form headings added. E…3861ad6
20 days ago
-
Model providers: Claude via OpenRouter by default, OpenAI fallback, any OpenAI-compatible endpoint
- New default provider openrouter (anthropic/claude-sonnet-5.5), with openai/gpt-4.1-mini tried once on outages, rate limits or unreadable answers; account errors (bad key, no credit) are not retried. - New 'compatible' provider for an institution's own gateway or local open models (vLLM, Ollama) via CURATORBOT_LLM_BASE_URL. - Plain-English errors for curators; details go to the server log. - /api/health and the How it works tab show the model and any fallback. - docs/model-providers.md explains the choice and data protection. Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAEfa829bf
23 days ago
-
Use the Hugging Face username (ignatius) for the Space and dataset
Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE96a766a -
Real-data demo: intake queue, catalogue comparison, private interview data
- Intake queue as the landing page: new submissions first, then released interviews with their catalogue scores, then the synthetic example - Interview status (under review, released, synthetic) in briefs, CLI and UI - Catalogue comparison tab for released interviews: headings found, missed (separating vocabulary gaps from model misses) and extra suggestions with their evidence - Real briefs load from a private Hugging Face dataset at start-up (and on /api/reload), so partner material never enters the code repository - Importer for the museum's released PDF transcripts (speaker initials, page headers and footers), so released interviews skip transcription - Runbook for the two data tracks; data policy and deployment guide updated; Colab notebook takes a status and a donor or catalogue file Claude-Session: https://claude.ai/code/session_01AteuhwysR4rHp4fqUteLAE9b35635