README-DEPLOY.md

ResearchBee v1.4 — verified-sources release

What changed, in one line

Policy facts now come from databases, not from the language model. Where no source could be reached, the system says so instead of filling the gap.

---

1. HuggingFace Space — researchbee-hf/

File	Status
app.py	modified — DOI-first /api/check-license, registry-backed /api/find-repository, fixed publisher matcher, Khazna startup + 23:30 refresh, new /api/health
prompts.py	modified — "ResearchPilot" → "ResearchBee" everywhere; LICENSE_SYSTEM_PROMPT can no longer state policy facts; new ADVISORY_NOTE_PROMPT; REPO_SYSTEM_PROMPT constrained to registry selection
oaworks.py	new — OA.Works Permissions client, schema-confirmed against T&F / Elsevier / Springer
doi_utils.py	new — DOI extraction from URLs and text; ISSN mod-11 validation
repositories.py	new — 25-entry curated repository registry
khazna_loader.py	new — loads the harvested Khazna index from an HF Dataset
requirements.txt	modified — added huggingface_hub, apscheduler

Upload all seven. ku_knowledge.py, openalex.py, scimago_index.py, utils.py, Dockerfile, and the SCImago CSV are unchanged — leave them.

Before you deploy

1. Set KU_ROR in oaworks.py. Find it at https://ror.org (search "Khalifa University"). This matters more than it looks: Elsevier publishes a 12-month embargo for UK institutions and 24 months for everyone else, and the ROR lets OA.Works pick the right one.
2. Optionally set KHAZNA_DATASET and HF_TOKEN in Space secrets.
3. Check GET /api/health after deploy — it reports every source's status.

Khazna lookups return checked: false until the harvest has run. That is correct behaviour, not a failure.

---

2. GitHub Pages — researchbee-github/

File	Status
index.html	modified — DOI/URL field added as the primary input
js/license.js	modified — sends the DOI, renders provenance, Khazna status, deposit statement
js/render_license.js	new — the new render helpers
css/style.css	modified — styles appended at the end
khazna_oai.py	new — the OAI-PMH harvester (runs in Actions, not in the Space)
.github/workflows/harvest-khazna.yml	new — nightly 23:00 Dubai harvest
test_publishers.py	new — validation harness

Other JS files are untouched.

---

3. Khazna harvest — first run

# See the deposit gap before automating anything
python khazna_oai.py gap

# Verify the parser against real records
python khazna_oai.py inspect --set publications:withFiles --format mods -n 2 --raw
python khazna_oai.py inspect --set openaire --format oai_dc -n 2 --raw

Then:

1. Create an HF Dataset repo (default: nikeshn/researchbee-data)
2. Add HF_TOKEN as a GitHub repository secret
3. Actions → "Harvest Khazna" → Run workflow (first full run)
4. Nightly thereafter: Sunday full, Mon–Sat incremental

Sunday is full because Khazna's OAI reports deletedRecord: no — incrementals can never learn that a record was withdrawn.

---

4. How to tell it is working

With a DOI — green "Verified · OA.Works Permissions" badge, per-version rights, the publisher's required deposit statement with a copy button, a source panel with evidence links, and Khazna status.

Without a DOI — amber "Unverified · journal-level estimate", all version blocks "Unclear", a prompt to enter a DOI. The backend forcibly downgrades any policy_status: "Confirmed" the model returns, so this cannot be bypassed.

Khazna unavailable — "Khazna status not checked", explicitly distinguished from "Not yet in Khazna".

Test DOIs from this build:

10.1080/19322909.2023.2221477   T&F      — no embargo stated
10.1016/j.jclepro.2023.136775   Elsevier — 24 months (not 12; see below)
10.1007/s11192-023-04812-4      Springer — preprint immediate, AAM 12 months

---

5. Things worth knowing

The Elsevier embargo trap. OA.Works returns two permissions with identical score: 1100 — 12 months (UK) and 24 months (everyone else). Ranking by score alone silently picks 12. _permission_rank() in oaworks.py breaks the tie by the longer embargo. Do not "optimise" that away.

Absent is not zero. T&F records carry no embargo field. The system reports "Not stated in the permission record" rather than "no embargo", because T&F's published policy is normally 12 months and a wrong permissive answer costs the library a takedown notice.

Publisher matcher. Now covers imprints whose SCImago string never says the parent publisher — W.B. Saunders, Academic Press, Cell Press, BioMed Central, Palgrave, Blackwell, Informa, Dove Medical. Q1 APC badge coverage went from 45.8% to 51.1% (+497 journals). AMBIGUOUS_ALIASES prevents "Maximum Academic Press" from being mistaken for Elsevier.

Repository registry. The model selects entries by id and writes the fit_reason; every factual field is overwritten from repositories.py. Unknown ids are dropped. Entries you have not personally verified are left blank rather than guessed — please fill them in and bump verified.

---

6. Still open

• KU_ROR is empty
• Repository registry has blank fields awaiting your verification
• Wiley, IEEE and ACS not yet validated against OA.Works (test_publishers.py)
• Worth telling OA.Works about the UK/non-UK tie: help@openaccessbutton.org