Spaces:
Running
Running
| # Real data for the demo: released interviews and Box submissions | |
| The demo combines two kinds of real interview, each showing something different: | |
| | | Released interviews | Box submissions | | |
| | --- | --- | --- | | |
| | What they show | Accuracy: suggested headings scored against the cataloguers' own | The real intake job: a new interview, plus a donor check | | |
| | Status in the app | `released` | `under_review` | | |
| | Transcript | The museum's released transcript (PDF), no transcription needed | Transcribed on a GPU (Colab notebook) | | |
| | Extra input | Catalogue record (`catalogue.json`) | Donor-reported details (`donor.json`) | | |
| | Tabs that light up | Catalogue comparison | Donor check, listening flags, language timeline | | |
| Everything under `private/` stays out of git, and so do the real briefs. Finished `brief.json` (and `backtest.json`) files go to a **private Hugging Face dataset**, which the Space downloads when it starts. The code repository keeps only the synthetic example, so it can be made public later without rewriting its history. | |
| ## A. A released interview (about 30 minutes, no GPU) | |
| 1. Find the interview on collections.ushmm.org. From its record, copy the **Subjects** list exactly, one heading per line, and the page URL. | |
| 2. Download the transcript PDF (the record's "transcript" link, usually `collections.ushmm.org/oh_findingaids/RG-…_trs_en.pdf`) to `private/<ID>/transcript.pdf`. | |
| 3. Write `private/<ID>/catalogue.json`: | |
| ```json | |
| {"id": "RG-50.030.0123", "title": "Oral history interview with …", "url": "https://collections.ushmm.org/search/catalog/irn…", | |
| "subjects": ["Heading one.", "Heading two--Subdivision."]} | |
| ``` | |
| 4. For vocabulary condition A, collect the catalogue headings into a private extra vocabulary rather than editing `curatorbot/data/vocabulary.txt`: | |
| ```bash | |
| python scripts/build_eval_vocabulary.py private/*/catalogue.json -o private/eval-vocabulary.txt | |
| export CURATORBOT_VOCAB_EXTRA=private/eval-vocabulary.txt | |
| ``` | |
| Leave `CURATORBOT_VOCAB_EXTRA` unset for condition B (shared vocabulary only), which shows the vocabulary gap. For evaluation runs also set `CURATORBOT_LLM_FALLBACK_MODEL=` (empty) so a failed call stops instead of switching model, and `CURATORBOT_NER_MODEL=en_core_web_trf`. | |
| For condition C (the realistic list), build the vocabulary from the rest of the collection instead. In a browser, open https://portal.ehri-project.eu/units/us-005578-irn504457 (the USHMM oral history collection), choose Export Metadata > EAD 2002 XML, and save the file as `private/ehri/us-005578-irn504457.xml`. Then: | |
| ```bash | |
| python scripts/harvest_ehri_headings.py private/ehri/us-005578-irn504457.xml \ | |
| --exclude private/RG-*/catalogue.json --out private/condition-c | |
| export CURATORBOT_VOCABULARY=private/condition-c/vocabulary.txt | |
| unset CURATORBOT_VOCAB_EXTRA | |
| ``` | |
| It reports how many of each test interview's headings remain available. Every interview being scored must be in `--exclude`; rebuild the list when the test set changes. | |
| 5. Import the transcript: | |
| ```bash | |
| python scripts/import_transcript_pdf.py private/<ID>/transcript.pdf -o private/<ID>/transcript.txt | |
| ``` | |
| Check the speaker it chose as interviewer; rerun with `--interviewer XX` if wrong. Skim the .txt for page furniture it missed. | |
| 6. Build the brief and score it: | |
| ```bash | |
| python scripts/prepare_interview.py --id <ID> --status released \ | |
| --transcript private/<ID>/transcript.txt --catalogue private/<ID>/catalogue.json | |
| ``` | |
| This needs `OPENROUTER_API_KEY` (or another provider's key, see `docs/model-providers.md`) in your environment and prints recall against the catalogue. | |
| Released transcripts carry no timestamps, so the brief cites turn numbers (p001, p002 …) instead of times. | |
| To refresh a saved brief after a change to the validators or the place lookup, without a new model call: | |
| ```bash | |
| curatorbot revalidate data/interviews/<ID>/brief.json --catalogue private/<ID>/catalogue.json | |
| ``` | |
| ## B. A Box submission (about 1 hour per interview hour, GPU) | |
| 1. Copy the parts from Box (e.g. `Files for Ignatius / Interview 2`: `RG-50.030.0001.01.06.mp4` … `.06.06.mp4`) to Google Drive under `CuratorBot/<ID>/media/`. | |
| 2. Write `donor.json` there from the donor's description (only the fields the donor gave): | |
| ```json | |
| {"interviewee": "…", "birth": "place, year", "languages": "…", "wartime": "…", "postwar": "…", "duration": "about N hours, 6 files"} | |
| ``` | |
| 3. Open `notebooks/prepare_on_colab.ipynb` on a GPU runtime, set `ID` and `STATUS = "under_review"`, and run it. It stops after transcription. | |
| 4. Review `transcript.txt`: speaker roles, names of people and places. Do not rewrite testimony. | |
| 5. Run the analysis cell. It writes `brief.json` with the donor check, language timeline and listening flags. | |
| ## D. Submissions delivered as recordings (Box), and the end-to-end test | |
| The notebook `notebooks/prepare_on_colab.ipynb` takes a delivery folder as it comes: one subfolder per submission (Box's "Interview 1" ... "Interview 10"), recordings at any depth, parts like `RG-50.030.0001.01.06` checked for gaps, other files (forms, photographs) listed. | |
| 1. In Google Drive, put the Box folders, exactly as downloaded, under `CuratorBot/box/`. Copy `private/eval-vocabulary.txt` to `CuratorBot/eval-vocabulary.txt` for evaluation runs. | |
| 2. Before any Box material goes to the model: in your OpenRouter account's privacy settings, allow only providers that neither train on nor keep prompts. | |
| 3. Open the notebook with the **Open in Colab** badge at its top (or in the README): https://colab.research.google.com/github/SpaceTimeNarratives/curatorbot/blob/main/notebooks/prepare_on_colab.ipynb. While the repository is private, Colab asks to connect to GitHub the first time: allow it, tick access to private repositories, and, since the repository belongs to the SpaceTimeNarratives organisation, grant the organisation access if GitHub offers it. Failing that, use File > Upload notebook. Then Runtime > Change runtime type > T4 GPU, and add the `GH_TOKEN` and `OPENROUTER_API_KEY` secrets. | |
| 4. Run setup, settings and **1 Inventory**. It prints each submission's ID (from the file names, e.g. `RG-50.030.0001-BOX`, or from the folder, e.g. `BOX-Interview-5`), recordings, durations, missing parts and other files. The same works locally on a copy: `curatorbot inventory <folder> --probe`. | |
| 5. Optional per submission, in `CuratorBot/<ID>/`: `donor.json` (the donor's description), `catalogue.json` (if the interview is catalogued), `reference.txt` (a museum transcript). | |
| 6. Set `RUN` to the folders to process and run **3 Transcribe**, **4 Analyse** and, where there is a `reference.txt`, **5 Compare**. Results go to `CuratorBot/<ID>/out/`. A submission already transcribed is skipped, so rerun after a disconnect. | |
| 7. Copy `out/` to `private/<ID>/` on your machine for the log, and upload `brief.json` (and `backtest.json`) to the private dataset as usual. | |
| **Progress.** The transcribe step prints its progress as it goes (audio extraction per part; transcription every 30 seconds with the time left; language detection) and keeps the latest state in `CuratorBot/<ID>/out/progress.json`, which can be opened in Drive from any device. If a step fails, the reason is printed above the error. | |
| **Beyond Colab.** For batches, Lancaster's Hex cluster (GPU desktops with an A3000, 12 GB of GPU memory, and 32 GB of RAM; software runs in Docker containers) suits this work: faster-whisper large-v3 in float16 needs about 5 GB of GPU memory. The same `prepare_interview.py --media <folder> --device cuda` runs there inside a GPU container; a ready-made image is planned. faster-whisper's engine needs the CUDA 12 cuBLAS and cuDNN 9 libraries; where the system CUDA is newer (as on Colab now), install `nvidia-cublas-cu12` and `nvidia-cudnn-cu12==9.*` with pip and CuratorBot loads them itself. Contacts for access are on the Hex page (lancaster.ac.uk/scc/research/research-facilities/hex). Check that university storage on Hex meets the data policy before copying recordings there. | |
| **End-to-end test.** Box Interview 2 is RG-50.030.0001 (Kochalski), which the museum has released. Put `private/RG-50.030.0001/catalogue.json` as `catalogue.json` and `private/RG-50.030.0001/transcript.txt` as `reference.txt` in `CuratorBot/RG-50.030.0001-BOX/`, run with `RUN = ['Interview 2']`, `EVALUATION = True`, `REVIEWED = False`, and record the results in section E1 of the evaluation log. | |
| ## C. Put them in the demo | |
| One-time setup: | |
| 1. On huggingface.co: **New → Dataset**, owner `ignatius`, name `curatorbot-data`, **Private**. | |
| 2. Create a Hugging Face token with **read** access (a fine-grained token limited to that dataset is best). | |
| 3. In the Space's settings add the secret `HF_TOKEN` (that token) and the variable `CURATORBOT_DATA_REPO` = `ignatius/curatorbot-data`. | |
| For each interview, `prepare_interview.py` writes `data/interviews/<ID>/brief.json` (and `backtest.json` for released interviews); the Colab notebook also copies them to `CuratorBot/<ID>/out/` in Drive. Upload that folder to the dataset, keeping the interview id as the folder name: | |
| ```bash | |
| huggingface-cli upload ignatius/curatorbot-data data/interviews/<ID> <ID> --repo-type dataset | |
| ``` | |
| or drag the two files into a new `<ID>/` folder on the dataset's **Files** page. Then either restart the Space, or fetch without restarting: | |
| ```bash | |
| curl -X POST -H "X-Demo-Passcode: <passcode>" https://<space-url>/api/reload | |
| ``` | |
| Do not `git add` real interviews: keep `data/interviews/` in git for the synthetic example only. `/api/health` reports when the private dataset last synced and any error. | |
| ## Checks before showing anyone | |
| - Read every heading's quote and every contradicted donor claim against the transcript. | |
| - Confirm the queue shows the right status for each interview. | |
| - For released interviews, note the recall figures and which misses are vocabulary gaps. | |