# Real data for the demo: released interviews and Box submissions The demo combines two kinds of real interview, each showing something different: | | Released interviews | Box submissions | | --- | --- | --- | | What they show | Accuracy: suggested headings scored against the cataloguers' own | The real intake job: a new interview, plus a donor check | | Status in the app | `released` | `under_review` | | Transcript | The museum's released transcript (PDF), no transcription needed | Transcribed on a GPU (Colab notebook) | | Extra input | Catalogue record (`catalogue.json`) | Donor-reported details (`donor.json`) | | Tabs that light up | Catalogue comparison | Donor check, listening flags, language timeline | Everything under `private/` stays out of git, and so do the real briefs. Finished `brief.json` (and `backtest.json`) files go to a **private Hugging Face dataset**, which the Space downloads when it starts. The code repository keeps only the synthetic example, so it can be made public later without rewriting its history. ## A. A released interview (about 30 minutes, no GPU) 1. Find the interview on collections.ushmm.org. From its record, copy the **Subjects** list exactly, one heading per line, and the page URL. 2. Download the transcript PDF (the record's "transcript" link, usually `collections.ushmm.org/oh_findingaids/RG-…_trs_en.pdf`) to `private//transcript.pdf`. 3. Write `private//catalogue.json`: ```json {"id": "RG-50.030.0123", "title": "Oral history interview with …", "url": "https://collections.ushmm.org/search/catalog/irn…", "subjects": ["Heading one.", "Heading two--Subdivision."]} ``` 4. For vocabulary condition A, collect the catalogue headings into a private extra vocabulary rather than editing `curatorbot/data/vocabulary.txt`: ```bash python scripts/build_eval_vocabulary.py private/*/catalogue.json -o private/eval-vocabulary.txt export CURATORBOT_VOCAB_EXTRA=private/eval-vocabulary.txt ``` Leave `CURATORBOT_VOCAB_EXTRA` unset for condition B (shared vocabulary only), which shows the vocabulary gap. For evaluation runs also set `CURATORBOT_LLM_FALLBACK_MODEL=` (empty) so a failed call stops instead of switching model, and `CURATORBOT_NER_MODEL=en_core_web_trf`. For condition C (the realistic list), build the vocabulary from the rest of the collection instead. In a browser, open https://portal.ehri-project.eu/units/us-005578-irn504457 (the USHMM oral history collection), choose Export Metadata > EAD 2002 XML, and save the file as `private/ehri/us-005578-irn504457.xml`. Then: ```bash python scripts/harvest_ehri_headings.py private/ehri/us-005578-irn504457.xml \ --exclude private/RG-*/catalogue.json --out private/condition-c export CURATORBOT_VOCABULARY=private/condition-c/vocabulary.txt unset CURATORBOT_VOCAB_EXTRA ``` It reports how many of each test interview's headings remain available. Every interview being scored must be in `--exclude`; rebuild the list when the test set changes. 5. Import the transcript: ```bash python scripts/import_transcript_pdf.py private//transcript.pdf -o private//transcript.txt ``` Check the speaker it chose as interviewer; rerun with `--interviewer XX` if wrong. Skim the .txt for page furniture it missed. 6. Build the brief and score it: ```bash python scripts/prepare_interview.py --id --status released \ --transcript private//transcript.txt --catalogue private//catalogue.json ``` This needs `OPENROUTER_API_KEY` (or another provider's key, see `docs/model-providers.md`) in your environment and prints recall against the catalogue. Released transcripts carry no timestamps, so the brief cites turn numbers (p001, p002 …) instead of times. To refresh a saved brief after a change to the validators or the place lookup, without a new model call: ```bash curatorbot revalidate data/interviews//brief.json --catalogue private//catalogue.json ``` ## B. A Box submission (about 1 hour per interview hour, GPU) 1. Copy the parts from Box (e.g. `Files for Ignatius / Interview 2`: `RG-50.030.0001.01.06.mp4` … `.06.06.mp4`) to Google Drive under `CuratorBot//media/`. 2. Write `donor.json` there from the donor's description (only the fields the donor gave): ```json {"interviewee": "…", "birth": "place, year", "languages": "…", "wartime": "…", "postwar": "…", "duration": "about N hours, 6 files"} ``` 3. Open `notebooks/prepare_on_colab.ipynb` on a GPU runtime, set `ID` and `STATUS = "under_review"`, and run it. It stops after transcription. 4. Review `transcript.txt`: speaker roles, names of people and places. Do not rewrite testimony. 5. Run the analysis cell. It writes `brief.json` with the donor check, language timeline and listening flags. ## D. Submissions delivered as recordings (Box), and the end-to-end test The notebook `notebooks/prepare_on_colab.ipynb` takes a delivery folder as it comes: one subfolder per submission (Box's "Interview 1" ... "Interview 10"), recordings at any depth, parts like `RG-50.030.0001.01.06` checked for gaps, other files (forms, photographs) listed. 1. In Google Drive, put the Box folders, exactly as downloaded, under `CuratorBot/box/`. Copy `private/eval-vocabulary.txt` to `CuratorBot/eval-vocabulary.txt` for evaluation runs. 2. Before any Box material goes to the model: in your OpenRouter account's privacy settings, allow only providers that neither train on nor keep prompts. 3. Open the notebook with the **Open in Colab** badge at its top (or in the README): https://colab.research.google.com/github/SpaceTimeNarratives/curatorbot/blob/main/notebooks/prepare_on_colab.ipynb. While the repository is private, Colab asks to connect to GitHub the first time: allow it, tick access to private repositories, and, since the repository belongs to the SpaceTimeNarratives organisation, grant the organisation access if GitHub offers it. Failing that, use File > Upload notebook. Then Runtime > Change runtime type > T4 GPU, and add the `GH_TOKEN` and `OPENROUTER_API_KEY` secrets. 4. Run setup, settings and **1 Inventory**. It prints each submission's ID (from the file names, e.g. `RG-50.030.0001-BOX`, or from the folder, e.g. `BOX-Interview-5`), recordings, durations, missing parts and other files. The same works locally on a copy: `curatorbot inventory --probe`. 5. Optional per submission, in `CuratorBot//`: `donor.json` (the donor's description), `catalogue.json` (if the interview is catalogued), `reference.txt` (a museum transcript). 6. Set `RUN` to the folders to process and run **3 Transcribe**, **4 Analyse** and, where there is a `reference.txt`, **5 Compare**. Results go to `CuratorBot//out/`. A submission already transcribed is skipped, so rerun after a disconnect. 7. Copy `out/` to `private//` on your machine for the log, and upload `brief.json` (and `backtest.json`) to the private dataset as usual. **Progress.** The transcribe step prints its progress as it goes (audio extraction per part; transcription every 30 seconds with the time left; language detection) and keeps the latest state in `CuratorBot//out/progress.json`, which can be opened in Drive from any device. If a step fails, the reason is printed above the error. **Beyond Colab.** For batches, Lancaster's Hex cluster (GPU desktops with an A3000, 12 GB of GPU memory, and 32 GB of RAM; software runs in Docker containers) suits this work: faster-whisper large-v3 in float16 needs about 5 GB of GPU memory. The same `prepare_interview.py --media --device cuda` runs there inside a GPU container; a ready-made image is planned. faster-whisper's engine needs the CUDA 12 cuBLAS and cuDNN 9 libraries; where the system CUDA is newer (as on Colab now), install `nvidia-cublas-cu12` and `nvidia-cudnn-cu12==9.*` with pip and CuratorBot loads them itself. Contacts for access are on the Hex page (lancaster.ac.uk/scc/research/research-facilities/hex). Check that university storage on Hex meets the data policy before copying recordings there. **End-to-end test.** Box Interview 2 is RG-50.030.0001 (Kochalski), which the museum has released. Put `private/RG-50.030.0001/catalogue.json` as `catalogue.json` and `private/RG-50.030.0001/transcript.txt` as `reference.txt` in `CuratorBot/RG-50.030.0001-BOX/`, run with `RUN = ['Interview 2']`, `EVALUATION = True`, `REVIEWED = False`, and record the results in section E1 of the evaluation log. ## C. Put them in the demo One-time setup: 1. On huggingface.co: **New → Dataset**, owner `ignatius`, name `curatorbot-data`, **Private**. 2. Create a Hugging Face token with **read** access (a fine-grained token limited to that dataset is best). 3. In the Space's settings add the secret `HF_TOKEN` (that token) and the variable `CURATORBOT_DATA_REPO` = `ignatius/curatorbot-data`. For each interview, `prepare_interview.py` writes `data/interviews//brief.json` (and `backtest.json` for released interviews); the Colab notebook also copies them to `CuratorBot//out/` in Drive. Upload that folder to the dataset, keeping the interview id as the folder name: ```bash huggingface-cli upload ignatius/curatorbot-data data/interviews/ --repo-type dataset ``` or drag the two files into a new `/` folder on the dataset's **Files** page. Then either restart the Space, or fetch without restarting: ```bash curl -X POST -H "X-Demo-Passcode: " https:///api/reload ``` Do not `git add` real interviews: keep `data/interviews/` in git for the synthetic example only. `/api/health` reports when the private dataset last synced and any error. ## Checks before showing anyone - Read every heading's quote and every contradicted donor claim against the transcript. - Confirm the queue shows the right status for each interview. - For released interviews, note the recall figures and which misses are vocabulary gaps.