docs/real-data-runbook.md

Real data for the demo: released interviews and Box submissions

The demo combines two kinds of real interview, each showing something different:

	Released interviews	Box submissions
What they show	Accuracy: suggested headings scored against the cataloguers' own	The real intake job: a new interview, plus a donor check
Status in the app	released	under_review
Transcript	The museum's released transcript (PDF), no transcription needed	Transcribed on a GPU (Colab notebook)
Extra input	Catalogue record (catalogue.json)	Donor-reported details (donor.json)
Tabs that light up	Catalogue comparison	Donor check, listening flags, language timeline

Everything under private/ stays out of git, and so do the real briefs. Finished brief.json (and backtest.json) files go to a private Hugging Face dataset, which the Space downloads when it starts. The code repository keeps only the synthetic example, so it can be made public later without rewriting its history.

A. A released interview (about 30 minutes, no GPU)

1. Find the interview on collections.ushmm.org. From its record, copy the Subjects list exactly, one heading per line, and the page URL.
2. Download the transcript PDF (the record's "transcript" link, usually collections.ushmm.org/oh_findingaids/RG-…_trs_en.pdf) to private/<ID>/transcript.pdf.
3. Write private/<ID>/catalogue.json:

   {"id": "RG-50.030.0123", "title": "Oral history interview with …", "url": "https://collections.ushmm.org/search/catalog/irn…",
    "subjects": ["Heading one.", "Heading two--Subdivision."]}
   

4. For vocabulary condition A, collect the catalogue headings into a private extra vocabulary rather than editing curatorbot/data/vocabulary.txt:

   python scripts/build_eval_vocabulary.py private/*/catalogue.json -o private/eval-vocabulary.txt
   export CURATORBOT_VOCAB_EXTRA=private/eval-vocabulary.txt
   

   Leave CURATORBOT_VOCAB_EXTRA unset for condition B (shared vocabulary only), which shows the vocabulary gap. For evaluation runs also set CURATORBOT_LLM_FALLBACK_MODEL= (empty) so a failed call stops instead of switching model, and CURATORBOT_NER_MODEL=en_core_web_trf. For condition C (the realistic list), build the vocabulary from the rest of the collection instead. In a browser, open https://portal.ehri-project.eu/units/us-005578-irn504457 (the USHMM oral history collection), choose Export Metadata > EAD 2002 XML, and save the file as private/ehri/us-005578-irn504457.xml. Then:

   python scripts/harvest_ehri_headings.py private/ehri/us-005578-irn504457.xml \
     --exclude private/RG-*/catalogue.json --out private/condition-c
   export CURATORBOT_VOCABULARY=private/condition-c/vocabulary.txt
   unset CURATORBOT_VOCAB_EXTRA
   

   It reports how many of each test interview's headings remain available. Every interview being scored must be in --exclude; rebuild the list when the test set changes.
5. Import the transcript:

   python scripts/import_transcript_pdf.py private/<ID>/transcript.pdf -o private/<ID>/transcript.txt
   

   Check the speaker it chose as interviewer; rerun with --interviewer XX if wrong. Skim the .txt for page furniture it missed.
6. Build the brief and score it:

   python scripts/prepare_interview.py --id <ID> --status released \
     --transcript private/<ID>/transcript.txt --catalogue private/<ID>/catalogue.json
   

   This needs OPENROUTER_API_KEY (or another provider's key, see docs/model-providers.md) in your environment and prints recall against the catalogue.

Released transcripts carry no timestamps, so the brief cites turn numbers (p001, p002 …) instead of times.

To refresh a saved brief after a change to the validators or the place lookup, without a new model call:

curatorbot revalidate data/interviews/<ID>/brief.json --catalogue private/<ID>/catalogue.json

B. A Box submission (about 1 hour per interview hour, GPU)

1. Copy the parts from Box (e.g. Files for Ignatius / Interview 2: RG-50.030.0001.01.06.mp4 … .06.06.mp4) to Google Drive under CuratorBot/<ID>/media/.
2. Write donor.json there from the donor's description (only the fields the donor gave):

   {"interviewee": "…", "birth": "place, year", "languages": "…", "wartime": "…", "postwar": "…", "duration": "about N hours, 6 files"}
   

3. Open notebooks/prepare_on_colab.ipynb on a GPU runtime, set ID and STATUS = "under_review", and run it. It stops after transcription.
4. Review transcript.txt: speaker roles, names of people and places. Do not rewrite testimony.
5. Run the analysis cell. It writes brief.json with the donor check, language timeline and listening flags.

D. Submissions delivered as recordings (Box), and the end-to-end test

The notebook notebooks/prepare_on_colab.ipynb takes a delivery folder as it comes: one subfolder per submission (Box's "Interview 1" ... "Interview 10"), recordings at any depth, parts like RG-50.030.0001.01.06 checked for gaps, other files (forms, photographs) listed.

1. In Google Drive, put the Box folders, exactly as downloaded, under CuratorBot/box/. Copy private/eval-vocabulary.txt to CuratorBot/eval-vocabulary.txt for evaluation runs.
2. Before any Box material goes to the model: in your OpenRouter account's privacy settings, allow only providers that neither train on nor keep prompts.
3. Open the notebook with the Open in Colab badge at its top (or in the README): https://colab.research.google.com/github/SpaceTimeNarratives/curatorbot/blob/main/notebooks/prepare_on_colab.ipynb. While the repository is private, Colab asks to connect to GitHub the first time: allow it, tick access to private repositories, and, since the repository belongs to the SpaceTimeNarratives organisation, grant the organisation access if GitHub offers it. Failing that, use File > Upload notebook. Then Runtime > Change runtime type > T4 GPU, and add the GH_TOKEN and OPENROUTER_API_KEY secrets.
4. Run setup, settings and 1 Inventory. It prints each submission's ID (from the file names, e.g. RG-50.030.0001-BOX, or from the folder, e.g. BOX-Interview-5), recordings, durations, missing parts and other files. The same works locally on a copy: curatorbot inventory <folder> --probe.
5. Optional per submission, in CuratorBot/<ID>/: donor.json (the donor's description), catalogue.json (if the interview is catalogued), reference.txt (a museum transcript).
6. Set RUN to the folders to process and run 3 Transcribe, 4 Analyse and, where there is a reference.txt, 5 Compare. Results go to CuratorBot/<ID>/out/. A submission already transcribed is skipped, so rerun after a disconnect.
7. Copy out/ to private/<ID>/ on your machine for the log, and upload brief.json (and backtest.json) to the private dataset as usual.

Progress. The transcribe step prints its progress as it goes (audio extraction per part; transcription every 30 seconds with the time left; language detection) and keeps the latest state in CuratorBot/<ID>/out/progress.json, which can be opened in Drive from any device. If a step fails, the reason is printed above the error.

Beyond Colab. For batches, Lancaster's Hex cluster (GPU desktops with an A3000, 12 GB of GPU memory, and 32 GB of RAM; software runs in Docker containers) suits this work: faster-whisper large-v3 in float16 needs about 5 GB of GPU memory. The same prepare_interview.py --media <folder> --device cuda runs there inside a GPU container; a ready-made image is planned. faster-whisper's engine needs the CUDA 12 cuBLAS and cuDNN 9 libraries; where the system CUDA is newer (as on Colab now), install nvidia-cublas-cu12 and nvidia-cudnn-cu12==9.* with pip and CuratorBot loads them itself. Contacts for access are on the Hex page (lancaster.ac.uk/scc/research/research-facilities/hex). Check that university storage on Hex meets the data policy before copying recordings there.

End-to-end test. Box Interview 2 is RG-50.030.0001 (Kochalski), which the museum has released. Put private/RG-50.030.0001/catalogue.json as catalogue.json and private/RG-50.030.0001/transcript.txt as reference.txt in CuratorBot/RG-50.030.0001-BOX/, run with RUN = ['Interview 2'], EVALUATION = True, REVIEWED = False, and record the results in section E1 of the evaluation log.

C. Put them in the demo

One-time setup:

1. On huggingface.co: New → Dataset, owner ignatius, name curatorbot-data, Private.
2. Create a Hugging Face token with read access (a fine-grained token limited to that dataset is best).
3. In the Space's settings add the secret HF_TOKEN (that token) and the variable CURATORBOT_DATA_REPO = ignatius/curatorbot-data.

For each interview, prepare_interview.py writes data/interviews/<ID>/brief.json (and backtest.json for released interviews); the Colab notebook also copies them to CuratorBot/<ID>/out/ in Drive. Upload that folder to the dataset, keeping the interview id as the folder name:

huggingface-cli upload ignatius/curatorbot-data data/interviews/<ID> <ID> --repo-type dataset

or drag the two files into a new <ID>/ folder on the dataset's Files page. Then either restart the Space, or fetch without restarting:

curl -X POST -H "X-Demo-Passcode: <passcode>" https://<space-url>/api/reload

Do not git add real interviews: keep data/interviews/ in git for the synthetic example only. /api/health reports when the private dataset last synced and any error.

Checks before showing anyone

• Read every heading's quote and every contradicted donor claim against the transcript.
• Confirm the queue shows the right status for each interview.
• For released interviews, note the recall figures and which misses are vocabulary gaps.