Upload 52 files

.env.example

@@ -27,3 +27,9 @@ USAGE_PUSH_ENABLED=false
 # Optional model configuration
 ASJC_LLM_MODEL=gpt-4.1-nano
 ANTHROPIC_API_KEY=
+
+# Optional re3data.org supplement for the data-repository finder (v1.8.3)
+# Set to false to hide the "More repositories from re3data.org" list.
+RE3DATA_ENABLED=true
+# Upper bound on the re3data lookup; it runs alongside the model call.
+RE3DATA_TIMEOUT_SECONDS=12

DEPLOYMENT-LINKS.md

@@ -1,67 +1,67 @@
-# ResearchBee hyperlink update — 20 September 2026
-
-## Upload the backend first
-
-1. Keep a backup of your current Hugging Face Space and LibGuides HTML.
-2. Extract `ResearchBee_HuggingFace_Backend.zip`. Upload **the contents of its `researchbee` folder** to the root of your existing Docker Space. Replace matching files. Do not upload the ZIP itself as the application.
-3. Include the complete `tests/` directory, `trusted_links.py`, `journal_links_curated.json`, `journals.db`, `scimagojr_2025.csv`, requirements files, Dockerfile and all other supplied application files. The Docker build executes the Python tests. There is no new paid service or dependency.
-4. Keep your existing Space secrets and configuration. Credentials belong only in Hugging Face Secrets; do not put them in the frontend. The existing OpenAlex integration and optional `OPENALEX_API_KEY` are reused.
-5. Wait for the build. If the existing pinned packages cannot be resolved or a test fails, do not bypass the build gate. Check the build log before publishing the frontend. The preparation environment could not install these dependencies or run Docker.
-6. Check `/api/health`, then try a journal search and the new “Find author guidelines” action. A publisher may reject automated access; an unavailable-guidance result with the retained homepage is expected in that case.
-
-## Replace the LibGuides frontend
-
-1. Open `ResearchBee_LibGuides.html` in a text editor and copy its complete contents into the existing LibGuides HTML/source block, replacing the old fragment.
-2. The file remains one self-contained HTML fragment with embedded CSS and JavaScript. Do not add a second copy alongside the old one.
-3. `RB_API_BASE` retains `https://nikeshn-researchbee.hf.space`. Change only that setting if your Space address changes.
-4. Preview desktop and mobile layouts in LibGuides before publishing. Test English and Arabic, journal details, additional results, comparisons, article sharing, repository selection, and reset while an author-guidance request is pending.
-5. Revert the HTML and backend together from your backup if necessary.
-
-## What changed
-
-- Journal titles and prominent website/author-guidance actions in detailed cards, additional results and comparison views. Existing branding, mascot, institutional text, copyright, language options, feedback destination and workflows were retained from the separately supplied frontend.
-- Canonical database identities and ISSNs gate homepage metadata. OpenAlex responses must match the requested ISSN; a title search must match the complete normalized title. No LLM-supplied URLs are used by the new link controls.
-- New `POST /api/journal-links` accepts a canonical Scopus ID and/or ISSN, never an arbitrary URL. Publisher-page discovery runs only when the user clicks. It follows actual HTML anchors on an identity-matched publisher page and checks the discovered destination.
-- Official publisher domains must match the canonical publisher. HTTPS only, public DNS addresses, pinned connection IP with original TLS verification/Host, manual redirect validation, at most three requests per page, two pages per discovery, 512 KB per page, socket/page/overall discovery limits and four concurrent discovery operations.
-- Checked guidance is cached for 24 hours; unsuccessful guidance and metadata lookups for one hour. Caches are bounded to 512 records per process and reset on Space restart. Concurrent clicks for the same journal share the lookup. Publisher content can change after a check.
-- Evidence labels distinguish curated, checked, metadata-only and unavailable. Provider update/retrieval dates and curated review dates retain their meaning. A lookup attempt is not called a review. “Checked” means the official page/link was retrieved successfully, not that every instruction was reviewed by library staff.
-- Guidance is labelled journal-specific only when the checked page heading and journal path support that classification. Other discovered instructions are explicitly general publisher guidance.
-- Category quartiles are read from the supplied CSV using the matching SCImago source ID plus overlapping canonical ISSNs. The metric year is read from the CSV's `Total Docs. (2025)` header. Each category retains its own quartile; the best journal quartile is not substituted.
-- Category rankings use exact, unambiguous category-code matches from the supplied taxonomy; quartile badges use the journal's SCImago profile. Missing/ambiguous mappings show “Not available”. Scopus source links use Scopus IDs independently. No CiteScore quartiles are inferred or added.
-- Repository names and visit buttons use the existing curated registry. Primary/secondary deposit choices receive URLs from their resolved registry IDs, ignoring model-generated URLs. Existing registry source and review information is retained.
-- Late discovery responses cannot change results after reset or a newer search. Request failures preserve the homepage and allow a retry.
-
-## Evidence limitations and maintenance
-
-There was no curated author-guidelines catalogue in the upload. `journal_links_curated.json` therefore starts empty; no remembered or guessed publisher paths have been added. Staff can add records keyed by canonical Scopus source ID with `issns`, `url`, `source`, `scope` (`journal-specific` or `general-publisher`), and an actual `reviewed_at` date. The ISSNs and official publisher domains are validated. Do not populate dates without a real review. Extend the publisher-domain allowlist only after verifying publisher ownership.
-
-The supplied taxonomy has duplicate names under different codes, and some CSV categories have no code match. Those category names remain visible with their original quartile and profile link, but their category-ranking link says “Not available”. This is deliberate: resolving them requires an authoritative SCImago category crosswalk. An attempt to retrieve the live SCImago rankings page was blocked with HTTP 403 in this environment.
-
-Some publishers require JavaScript, cookies or authentication, use another platform/domain, omit a discoverable author link, or block automated requests. The resolver does not guess a path or bypass access controls. It retains the metadata homepage and reports unavailable guidance. A homepage labelled metadata-only has not been tested for reachability.
-
-## Validation performed
-
-- 17 offline Python link-resolver tests passed using the actual bundled database and CSV plus mocked upstream responses: canonical identity conflicts, metadata identity, source dates, missing evidence, category-specific quartiles, separate identifiers, real relative-anchor extraction, general guidance labels, successful/negative/expired/bounded caches, concurrent duplicate requests, failed requests, unsafe URLs/private DNS/redirects, and curated repository selection.
-- 12 Node VM frontend behavior checks passed against the actual embedded functions: safe URL rendering, clickable titles, evidence labels, distinct category quartiles, reset/session invalidation, successful result updates and failed-request behavior.
-- 7 existing dependency-free frontend/repository regression checks passed.
-- Python compilation and embedded JavaScript syntax checks passed.
-- Full original tests directory retained. Added two FastAPI endpoint integration tests for identity-only input and rate limiting; these are included in the Docker pytest gate but could not be executed here.
-
-## Not verified here
-
-The environment could not install the pinned Python requirements. Therefore the full FastAPI/pytest suite, API server startup and Docker build were not run. Playwright was present but no browser executable was installed, so no visual/browser or real LibGuides test was completed. Node VM checks are not browser tests. Live OpenAlex, publisher discovery, credential-dependent services and production deployment were not verified. The release has not been deployed.
-
-Reproduce checks in a configured development environment:
-
-```bash
-python -m pip install -r requirements-dev.txt
-python -m compileall -q .
-python -m pytest -q
-node tests/test_frontend_links.cjs
-```
-
-The standalone offline link tests need only the Python standard library:
-
-```bash
-python -m unittest discover -s tests -p test_trusted_links.py -v
-```
+# ResearchBee hyperlink update — 20 September 2026
+
+## Upload the backend first
+
+1. Keep a backup of your current Hugging Face Space and LibGuides HTML.
+2. Extract `ResearchBee_HuggingFace_Backend.zip`. Upload **the contents of its `researchbee` folder** to the root of your existing Docker Space. Replace matching files. Do not upload the ZIP itself as the application.
+3. Include the complete `tests/` directory, `trusted_links.py`, `journal_links_curated.json`, `journals.db`, `scimagojr_2025.csv`, requirements files, Dockerfile and all other supplied application files. The Docker build executes the Python tests. There is no new paid service or dependency.
+4. Keep your existing Space secrets and configuration. Credentials belong only in Hugging Face Secrets; do not put them in the frontend. The existing OpenAlex integration and optional `OPENALEX_API_KEY` are reused.
+5. Wait for the build. If the existing pinned packages cannot be resolved or a test fails, do not bypass the build gate. Check the build log before publishing the frontend. The preparation environment could not install these dependencies or run Docker.
+6. Check `/api/health`, then try a journal search and the new “Find author guidelines” action. A publisher may reject automated access; an unavailable-guidance result with the retained homepage is expected in that case.
+
+## Replace the LibGuides frontend
+
+1. Open `ResearchBee_LibGuides.html` in a text editor and copy its complete contents into the existing LibGuides HTML/source block, replacing the old fragment.
+2. The file remains one self-contained HTML fragment with embedded CSS and JavaScript. Do not add a second copy alongside the old one.
+3. `RB_API_BASE` retains `https://nikeshn-researchbee.hf.space`. Change only that setting if your Space address changes.
+4. Preview desktop and mobile layouts in LibGuides before publishing. Test English and Arabic, journal details, additional results, comparisons, article sharing, repository selection, and reset while an author-guidance request is pending.
+5. Revert the HTML and backend together from your backup if necessary.
+
+## What changed
+
+- Journal titles and prominent website/author-guidance actions in detailed cards, additional results and comparison views. Existing branding, mascot, institutional text, copyright, language options, feedback destination and workflows were retained from the separately supplied frontend.
+- Canonical database identities and ISSNs gate homepage metadata. OpenAlex responses must match the requested ISSN; a title search must match the complete normalized title. No LLM-supplied URLs are used by the new link controls.
+- New `POST /api/journal-links` accepts a canonical Scopus ID and/or ISSN, never an arbitrary URL. Publisher-page discovery runs only when the user clicks. It follows actual HTML anchors on an identity-matched publisher page and checks the discovered destination.
+- Official publisher domains must match the canonical publisher. HTTPS only, public DNS addresses, pinned connection IP with original TLS verification/Host, manual redirect validation, at most three requests per page, two pages per discovery, 512 KB per page, socket/page/overall discovery limits and four concurrent discovery operations.
+- Checked guidance is cached for 24 hours; unsuccessful guidance and metadata lookups for one hour. Caches are bounded to 512 records per process and reset on Space restart. Concurrent clicks for the same journal share the lookup. Publisher content can change after a check.
+- Evidence labels distinguish curated, checked, metadata-only and unavailable. Provider update/retrieval dates and curated review dates retain their meaning. A lookup attempt is not called a review. “Checked” means the official page/link was retrieved successfully, not that every instruction was reviewed by library staff.
+- Guidance is labelled journal-specific only when the checked page heading and journal path support that classification. Other discovered instructions are explicitly general publisher guidance.
+- Category quartiles are read from the supplied CSV using the matching SCImago source ID plus overlapping canonical ISSNs. The metric year is read from the CSV's `Total Docs. (2025)` header. Each category retains its own quartile; the best journal quartile is not substituted.
+- Category rankings use exact, unambiguous category-code matches from the supplied taxonomy; quartile badges use the journal's SCImago profile. Missing/ambiguous mappings show “Not available”. Scopus source links use Scopus IDs independently. No CiteScore quartiles are inferred or added.
+- Repository names and visit buttons use the existing curated registry. Primary/secondary deposit choices receive URLs from their resolved registry IDs, ignoring model-generated URLs. Existing registry source and review information is retained.
+- Late discovery responses cannot change results after reset or a newer search. Request failures preserve the homepage and allow a retry.
+
+## Evidence limitations and maintenance
+
+There was no curated author-guidelines catalogue in the upload. `journal_links_curated.json` therefore starts empty; no remembered or guessed publisher paths have been added. Staff can add records keyed by canonical Scopus source ID with `issns`, `url`, `source`, `scope` (`journal-specific` or `general-publisher`), and an actual `reviewed_at` date. The ISSNs and official publisher domains are validated. Do not populate dates without a real review. Extend the publisher-domain allowlist only after verifying publisher ownership.
+
+The supplied taxonomy has duplicate names under different codes, and some CSV categories have no code match. Those category names remain visible with their original quartile and profile link, but their category-ranking link says “Not available”. This is deliberate: resolving them requires an authoritative SCImago category crosswalk. An attempt to retrieve the live SCImago rankings page was blocked with HTTP 403 in this environment.
+
+Some publishers require JavaScript, cookies or authentication, use another platform/domain, omit a discoverable author link, or block automated requests. The resolver does not guess a path or bypass access controls. It retains the metadata homepage and reports unavailable guidance. A homepage labelled metadata-only has not been tested for reachability.
+
+## Validation performed
+
+- 17 offline Python link-resolver tests passed using the actual bundled database and CSV plus mocked upstream responses: canonical identity conflicts, metadata identity, source dates, missing evidence, category-specific quartiles, separate identifiers, real relative-anchor extraction, general guidance labels, successful/negative/expired/bounded caches, concurrent duplicate requests, failed requests, unsafe URLs/private DNS/redirects, and curated repository selection.
+- 12 Node VM frontend behavior checks passed against the actual embedded functions: safe URL rendering, clickable titles, evidence labels, distinct category quartiles, reset/session invalidation, successful result updates and failed-request behavior.
+- 7 existing dependency-free frontend/repository regression checks passed.
+- Python compilation and embedded JavaScript syntax checks passed.
+- Full original tests directory retained. Added two FastAPI endpoint integration tests for identity-only input and rate limiting; these are included in the Docker pytest gate but could not be executed here.
+
+## Not verified here
+
+The environment could not install the pinned Python requirements. Therefore the full FastAPI/pytest suite, API server startup and Docker build were not run. Playwright was present but no browser executable was installed, so no visual/browser or real LibGuides test was completed. Node VM checks are not browser tests. Live OpenAlex, publisher discovery, credential-dependent services and production deployment were not verified. The release has not been deployed.
+
+Reproduce checks in a configured development environment:
+
+```bash
+python -m pip install -r requirements-dev.txt
+python -m compileall -q .
+python -m pytest -q
+node tests/test_frontend_links.cjs
+```
+
+The standalone offline link tests need only the Python standard library:
+
+```bash
+python -m unittest discover -s tests -p test_trusted_links.py -v
+```

README-DEPLOY.md

@@ -1,3 +1,7 @@
+# ResearchBee upload instructions (current: 1.8.3 / interface v1.9.3)
+
+> **1.8.3 quick path.** Upload the extracted backend files (including the new `re3data.py`, `tests/test_re3data.py`, `tests/test_subject_nearest.py`, `tests/test_frontend_re3data.cjs` and `tests/fixtures/`) to the Space, wait for the build, and confirm `/api/health` shows `version: "1.8.3"` and a `sources.re3data` entry. Then replace the LibGuides fragment with `ResearchBee_LibGuides.html` and confirm the footer shows **Interface v1.9.3**. Post-deploy checks: browse by subject "physics" (journals appear with an amber "nearest verified subject area" notice); run a repository search with discipline "Materials science" (a "More repositories from re3data.org" section appears below the curated cards). No new secret is required. The sections below are the 1.8.2 guide and still apply.
+
 # ResearchBee 1.8.2 — upload instructions
 
 Prepared 10 September 2026 from your uploaded HF backend and the separately uploaded LibGuides text. Your original uploads were preserved.
@@ -39,6 +43,8 @@ The API address remains `https://nikeshn-researchbee.hf.space`. No frontend API
 | `CLOUDFLARE_WORKER_URL`, `CLOUDFLARE_WORKER_TOKEN` | Keep existing values if using the external usage dashboard. Credentials stay on the server. |
 | `USAGE_PUSH_ENABLED` | Preserve existing setting; use `true` for periodic usage pushes. |
 | `OPENALEX_API_KEY` | Optional supported OpenAlex credential. Configure it if required by your OpenAlex access plan. |
+| `RE3DATA_ENABLED` | Optional (1.8.3). Default `true`. Set `false` to hide the re3data.org supplementary list. |
+| `RE3DATA_TIMEOUT_SECONDS` | Optional (1.8.3). Default `12`. Upper bound on the re3data lookup. |
 | `RESEARCHBEE_MODEL`, `RESEARCHBEE_MODEL_LIGHT` | Defaults remain `gpt-4.1-mini` and `gpt-4.1-nano`. Use models your account supports. |
 
 No new secret is mandatory for the code repairs themselves. Existing features still depend on their corresponding valid credentials and services.

README.md

@@ -1,66 +1,71 @@
----
-title: ResearchBee
-emoji: 🐝
-colorFrom: indigo
-colorTo: purple
-sdk: docker
-app_port: 7860
-pinned: false
-license: mit
-short_description: Journal selection, Open Access and research data assistant
----
-
-# ResearchBee — hyperlink update
-
-See **DEPLOYMENT-LINKS.md** for this update, upload instructions, executed checks and untested limitations. The following notes describe the supplied base release.
-
-## Base release 1.8.2
-
-Publishing, Open Access and research data guidance for Khalifa University researchers.
-
-This release repairs the supplied backend and uses the separately uploaded LibGuides fragment as the frontend baseline. It retains the existing HF endpoint and `journal_discovery_v2` schema.
-
-## Included functions
-
-- Journal discovery by manuscript or subject, verified identities and subject quartiles, preferred/excluded titles, comparisons and exports.
-- On-demand journal policies and DOI-based article-sharing checks with version, destination, licence, funder and embargo context.
-- Dataset repository selection from a curated registry, with deterministic constraint checks.
-- Source-aware English/Arabic chat, tool shortcuts, related literature and cover-letter drafts.
-- Health reporting, authenticated Khazna refresh and administration/usage monitoring.
-
-## Main repairs
-
-- Repaired 7,721 missing subject-quartile values in 7,096 existing journal records using the bundled 2025 CSV. Remaining unknowns stay unknown.
-- Retained eligible preferred journals through both shortlist limits; removed inflated fallback match scores.
-- Sharing advice evaluates complete policy options. Permission, embargo timing and destination conditions cannot be borrowed from incompatible options.
-- Repository recommendations enforce declared capabilities and sensitive-data restrictions. Khazna dataset advice is metadata registration, with file-hosting capabilities requiring confirmation.
-- Fixed chat rendering, question chips, history limits, SSE interruptions, autocomplete identity, embargo controls, CSV safety and long-abstract cover letters.
-- Added Arabic knowledge matching, interface translations, focus/keyboard improvements and source-review warnings.
-- Added bounded requests, paid-route quotas/concurrency, safer admin sessions, server-only usage credentials, redacted failure logging and lifecycle repairs.
-- OpenAlex receives broad validated taxonomy terms for manuscript evidence searches, not the unpublished title or abstract. It still receives journal identifiers and public literature queries.
-- Disabled the unsupported journal-level Altmetric score and unrepresentative percentile calculation.
-
-## Deploy
-
-See **README-DEPLOY.md**. Upload the complete extracted release to the existing Docker Space, including `journals.db`, new helper modules and `tests/`. Then replace the entire LibGuides fragment with `ResearchBee_LibGuides.html`.
-
-The Docker build installs pinned dependencies, runs `pip check`, compiles Python and runs the full Python test suite before creating the runtime image. Production runs as a non-root user.
-
-## Validation
-
-See **VALIDATION_1.8.2.md** for executed checks and limitations. Local verification used Python 3.12 and Node 24; the container targets Python 3.11. The Docker build and real FastAPI HTTP smoke suite could not be executed in the preparation environment. Live credentials, external service availability and rendered LibGuides browser behavior require the post-upload checks.
-
-```bash
-python -m pip install -r requirements-dev.txt
-python -m pytest -q
-python tests/run_offline_release_checks.py
-```
-
-## Evidence boundaries
-
-Journal identities and metrics come from the supplied local database and 2025 CSV. This release does not refresh them from current Scopus/SCImago sources. Fit scores support comparison; they do not estimate acceptance probability. Current APCs, title-level KU eligibility and unstructured policy conditions need authoritative confirmation.
-
-Khazna lookup reports a loaded snapshot. Its source harvest date is absent from the supplied snapshot format, so a missing DOI is not proof of current repository absence.
-
-Questions: library@ku.ac.ae; Khazna: khazna@ku.ac.ae.
-
+---
+title: ResearchBee
+emoji: 🐝
+colorFrom: indigo
+colorTo: purple
+sdk: docker
+app_port: 7860
+pinned: false
+license: mit
+short_description: Journal selection, Open Access and research data assistant
+---
+
+# ResearchBee 1.8.3 (interface v1.9.3)
+
+**What's new:** nearest-subject matching for journal browse (e.g. "physics" → Physics and Astronomy, disclosed to the user) and a clearly labelled re3data.org supplement in the data-repository finder. See **RELEASE_1.8.3.md** for details, validation and post-deploy checks. Deploy as described in README-DEPLOY.md; the new optional settings are `RE3DATA_ENABLED` and `RE3DATA_TIMEOUT_SECONDS`.
+
+## Earlier: hyperlink update
+
+See **DEPLOYMENT-LINKS.md** for this update, upload instructions, executed checks and untested limitations. The following notes describe the supplied base release.
+
+## Base release 1.8.2
+
+Publishing, Open Access and research data guidance for Khalifa University researchers.
+
+This release repairs the supplied backend and uses the separately uploaded LibGuides fragment as the frontend baseline. It retains the existing HF endpoint and `journal_discovery_v2` schema.
+
+## Included functions
+
+- Journal discovery by manuscript or subject, verified identities and subject quartiles, preferred/excluded titles, comparisons and exports.
+- On-demand journal policies and DOI-based article-sharing checks with version, destination, licence, funder and embargo context.
+- Dataset repository selection from a curated registry, with deterministic constraint checks.
+- Source-aware English/Arabic chat, tool shortcuts, related literature and cover-letter drafts.
+- Health reporting, authenticated Khazna refresh and administration/usage monitoring.
+
+## Main repairs
+
+- Repaired 7,721 missing subject-quartile values in 7,096 existing journal records using the bundled 2025 CSV. Remaining unknowns stay unknown.
+- Retained eligible preferred journals through both shortlist limits; removed inflated fallback match scores.
+- Sharing advice evaluates complete policy options. Permission, embargo timing and destination conditions cannot be borrowed from incompatible options.
+- Repository recommendations enforce declared capabilities and sensitive-data restrictions. Khazna dataset advice is metadata registration, with file-hosting capabilities requiring confirmation.
+- Fixed chat rendering, question chips, history limits, SSE interruptions, autocomplete identity, embargo controls, CSV safety and long-abstract cover letters.
+- Added Arabic knowledge matching, interface translations, focus/keyboard improvements and source-review warnings.
+- Added bounded requests, paid-route quotas/concurrency, safer admin sessions, server-only usage credentials, redacted failure logging and lifecycle repairs.
+- OpenAlex receives broad validated taxonomy terms for manuscript evidence searches, not the unpublished title or abstract. It still receives journal identifiers and public literature queries.
+- Disabled the unsupported journal-level Altmetric score and unrepresentative percentile calculation.
+
+## Deploy
+
+See **README-DEPLOY.md**. Upload the complete extracted release to the existing Docker Space, including `journals.db`, new helper modules and `tests/`. Then replace the entire LibGuides fragment with `ResearchBee_LibGuides.html`.
+
+The Docker build installs pinned dependencies, runs `pip check`, compiles Python and runs the full Python test suite before creating the runtime image. Production runs as a non-root user.
+
+## Validation
+
+See **VALIDATION_1.8.2.md** for executed checks and limitations. Local verification used Python 3.12 and Node 24; the container targets Python 3.11. The Docker build and real FastAPI HTTP smoke suite could not be executed in the preparation environment. Live credentials, external service availability and rendered LibGuides browser behavior require the post-upload checks.
+
+```bash
+python -m pip install -r requirements-dev.txt
+python -m pytest -q
+node tests/test_frontend_links.cjs
+node tests/test_frontend_re3data.cjs
+```
+
+## Evidence boundaries
+
+Journal identities and metrics come from the supplied local database and 2025 CSV. This release does not refresh them from current Scopus/SCImago sources. Fit scores support comparison; they do not estimate acceptance probability. Current APCs, title-level KU eligibility and unstructured policy conditions need authoritative confirmation.
+
+Khazna lookup reports a loaded snapshot. Its source harvest date is absent from the supplied snapshot format, so a missing DOI is not proof of current repository absence.
+
+Questions: library@ku.ac.ae; Khazna: khazna@ku.ac.ae.
+

RELEASE_1.8.3.md

@@ -0,0 +1,62 @@
+# ResearchBee 1.8.3 — change record (30 October 2026)
+
+Backend `1.8.3`, LibGuides interface `v1.9.3`. Earlier features are unchanged unless a change is listed here.
+
+## 1. Journal browse by subject: nearest-subject matching
+
+**Problem.** A subject that was not a literal SCImago/ASJC name (for example "physics", "maths", "biology", "machine learning") depended entirely on the LLM classifier. When the classifier was unavailable, or answered `not_found` for a broad term, the user saw no journals.
+
+**Change.** `/api/browse-subject` now resolves subjects in this order:
+
+1. Exact vocabulary term, as before.
+2. Conservative spelling repair, as before.
+3. **New:** deterministic nearest match in `scimago_index.nearest_subject_term()`:
+   - a reviewed alias table (about 200 common discipline names, for example maths → Mathematics, machine learning → Artificial Intelligence, petroleum engineering → Energy Engineering and Power Technology, criminology → Sociology and Political Science);
+   - whole-word containment, for example physics → area "Physics and Astronomy", or electrical engineering → "Electrical and Electronic Engineering". Parenthetical qualifiers are ignored, and generic words such as "science", "engineering" or "learning" never decide a match on their own.
+4. LLM classifier (closed vocabulary), as before.
+5. **New:** if the classifier fails or returns no mapping, a last-resort partial match on a distinctive word (for example "quantum computing networks" stays unmatched, while "geology rocks" maps to Geology).
+
+Every nearest or synonym match is disclosed to the user, for example: *SCImago and Scopus have no exact subject category for "physics". ResearchBee selected the nearest verified subject area "Physics And Astronomy".* Up to four verified alternative categories are offered as one-click chips. The disclosure note is highlighted in amber in the interface.
+
+Every alias target is validated against the bundled vocabulary when it is used, and a test fails if any target disappears. Meaningless input ("xyzzy") still returns no journals. Deterministic matches avoid an LLM call, which reduces cost and latency.
+
+Also fixed: the spelling-correction notice was shown twice.
+
+## 2. Data repository finder: re3data.org supplement
+
+The curated KU Library registry stays the only source of recommendations, deposit strategy and Khazna guidance. New module `re3data.py` adds a separate list: **More repositories from re3data.org**, labelled *Not reviewed by KU Library*.
+
+- Runs in parallel with the model call, so it adds no latency in normal operation. It is bounded by `RE3DATA_TIMEOUT_SECONDS` (default 12).
+- **Privacy:** only the *Discipline / domain* field is sent to re3data. The dataset title, description and notes are never sent.
+- Uses the re3data beta search API and v1 repository records, with namespace-agnostic XML parsing that refuses DTDs and is capped at 2 MB.
+- Excludes repositories that are already in the curated registry, closed (end date), closed to uploads, or purely institutional. These are usually limited to their own staff.
+- For datasets marked sensitive or personal, only repositories that list restricted, embargoed or closed access are shown, with the standard approval warning.
+- Ranks by subject/keyword/name evidence, then DOI, certification, open upload and versioning. Shows up to five.
+- Shows subjects, PID system, upload and access type, versioning, certificates, country, the repository link, the re3data record link and the record's own last-update date.
+- The list opens automatically when the curated registry produced fewer than two candidates. Otherwise it is collapsed.
+- Failures are isolated. An unreachable re3data service shows a short note and a "Search re3data.org yourself" link, and the curated results are unaffected.
+- Caches: 24 hours for searches, 7 days for records, 10 minutes for failures. All are bounded.
+- `/api/health` reports `sources.re3data`. Set `RE3DATA_ENABLED=false` to switch the list off.
+
+Also fixed: the curated cards' "Verify on re3data/FAIRsharing" links now URL-encode repository names.
+
+## 3. Frontend
+
+- Based on the live LibGuides fragment supplied on 30 October 2026 (KU styling). Its JavaScript was identical to the 1.8.2 package copy.
+- Adds amber styling for nearest/synonym subject notes, the re3data section (desktop and mobile), and Arabic labels for the new strings.
+- The footer reads **Interface v1.9.3**.
+
+## Validation
+
+- Python 3.11 (Docker target) and 3.13 with pinned requirements: `pip check` clean, `compileall` clean, **78 tests passed**. These are the 37 existing tests plus 27 subject tests and 14 re3data tests.
+- Node: 12 existing and 16 new frontend checks passed. The embedded script passes a syntax check.
+- Real-browser end-to-end test (Chromium/Playwright) of the live fragment against the real backend under uvicorn, with only the LLM and the re3data transport mocked:
+  - "physics" returns 15 verified journals with the nearest-area notice and alternative chips, and an alternative chip runs an exact search;
+  - the repository flow shows the curated cards, then the re3data section with safe links;
+  - no horizontal overflow at 390 px;
+  - no JavaScript errors.
+
+## Not verified here
+
+- The live re3data API was not reachable from the preparation environment (egress blocked). Parsing was built against re3data's published schema and tested with fixtures in that shape. After deployment, run one repository search and check that the re3data section shows real entries, or that `/api/health → sources.re3data.last_error` is empty.
+- The live OpenAI classifier and the Docker build on Hugging Face were not executed. The same pytest gate was run locally on Python 3.11.

SHA256SUMS.json

@@ -1,62 +1,73 @@
-{
-  ".dockerignore": "acbfa007f9cc614a581b5517709de7fb6df4a1836bc7c0c7bee7b96d020f3b83",
-  ".env.example": "3b015ac559145433fffa041d0de37bc0e0e1d823dd8b7ca08284384232c7d6c1",
-  ".gitattributes": "4d72e9b97f1cb0fd76258dec00ff2a9ac929fafab43ac1e68da5d4ab82149622",
-  "DEPLOYMENT-LINKS.md": "57ee8ab82d4bf157899b7fb265938ca568860b4b8160423edb14740239e62121",
-  "Dockerfile": "a5289268fe287dd5677b37ce60017563f7294729bed078c935cfce3fc010d280",
-  "README-DEPLOY.md": "c0906dc9e26fbe235109e0b70bc3c12aed33e4f3e50fff30fc7987d5b758da2c",
-  "README.md": "470e323af6c7c2ca4d487d1ce169817ebc5585fc322b7f3a384a22cc99d2abb0",
-  "RELEASE_1.7.0.md": "334d4cb6215ff7ab540c539d3c3f8d2830641ced95c4f9325fdd86c78b7488ce",
-  "RELEASE_1.8.0.md": "4d85c1711c33cd314c799d27e59795b62fe9104656d79cd76bba6af538b9ad1e",
-  "RELEASE_1.8.1.md": "500ba91d0737c6198a453de2bd539827b019185c34d68d32d325a2dc5e106fa4",
-  "RELEASE_1.8.2.md": "b006bed1643a39a8f79930ffb682c37c64ba00a3752b9a056993791153810f88",
-  "RELEASE_A_NOTES.md": "2c9d4dd7b8e787ada145b2393578c6b3499370fb4be2a89100bd555ed8b96a42",
-  "RELEASE_OPF_WEB_1.5.2.md": "b4cc66479031664e35026fdbb57ec0a83468e1d54100ac022100b2849df67b77",
-  "ResearchBee_LibGuides.html": "6d2946b8322a5c59c8d321d764cfb000196d5d5fe763380f8bf4c21feb0d1dd8",
-  "VALIDATION_1.8.2.md": "2c5ae04d43ebe3e3867bd5fbb4bdf303cebf03a7dc85dddc0079844b3ec82f41",
-  "admin.py": "7db70eb130f8bc4d9e98f566ce86315fd2dfa4cbe90e3415dba037aa13d28421",
-  "app.py": "9bd3acbd331080d97fef89314df0d298bcd44f2e6e50c1be3adf48ebf5e0b1f4",
-  "asjc_classifier.py": "c689f9d1fcd3ac15b4a0bd711153acbc9cb8780caed220f0c5d6afcde9af6fa6",
-  "asjc_eval.jsonl": "cc762d1d81532574a2270b98758324dc7c251b1d22a58604045ca5fa02a1b684",
-  "build_centroids.py": "48b3064a7b261195824fdedf84708ebb6e36954b9160d4b1722235d548e93e42",
-  "chat_knowledge.py": "d8b69d510ae847bb46f2c054a1458df7fe4a8db237dfb94ae515cfd4dff414ad",
-  "doi_utils.py": "7026a3913998b94aa4822f7cbbe9bb779ce52880f6e5a180959b37a19de9f8f9",
-  "harvest_openalex.py": "3c3cedd28c0892a10126249d892cd27275131d5295669d10d201ebae9ea4da3d",
-  "journal_discovery.py": "b69d282411eb4fecd3bcdb354cc34ac27f705b5b1d71ae600372d75a4672c12d",
-  "journal_links_curated.json": "ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356",
-  "journal_lookup.py": "4d7e1638b48cfd1c908b217c3e4b8bf145932a9f34d3cb762092a99e4a817aa0",
-  "journals.db": "6553cefa4515cbeea4631fcd5f59a4b8bdb756fe83701788e8a11fb9298b2404",
-  "khazna_loader.py": "3e944cbfa116647ff548015b37e083134da9aa2aafbda0e957be6c31ea7ac9c7",
-  "knowledge_base.json": "1838c1d1fd5456c16308ebfaba237b8516aa01df5901a83e7290bb73b55b5bc6",
-  "ku_knowledge.py": "b54c5acd3c26783e5b44872897810d04ae0257df367b9fde62e8b6bb6e5f9745",
-  "oaworks.py": "491ae10ae479001fd1218d3d9c623cb39f4e8c222fc3bdf3420b2a8f03328410",
-  "openalex.py": "434db8364c7f2893161f577a4008974d7ad6d6666b1891dac9fa1e9e05c75618",
-  "openpolicyfinder.py": "9f55d2e744b6f784422a0731f01c0b727c3cc5b001da009889e8dcf83288f081",
-  "policy_evaluation.py": "287df1a54c2d4893cd495da6d95ee8049f480bf0ff75879e52f2b0ede7307dab",
-  "prompts.py": "5b40d1be17b41b8a51231378bf70dee4056d5373c1a739b1a964094f93850359",
-  "repair_subject_quartiles.py": "91ddce1c0a08fa802923ff1806d5333617f4262eca686ef50ae35b3f35dd5950",
-  "repositories.py": "fcde7418a711296acfe9c552b284a4c79aa4e741034a6d50854880bfcfae0562",
-  "requirements-dev.txt": "e5bfd68d8ff1bdd7151344031c2e4cb97b187d099f681f4007b6d96ad7dfbdf8",
-  "requirements.txt": "179482f7f9e3177d035ef04101719117524476a98271dbafab3e3f18a17ce149",
-  "researchbee_journals.py": "623272ee184e37bf08b4fddd887f7aabd10b6c6cb1999416ddbcc0f5c72309c6",
-  "runtime_security.py": "8d3466aba8a11fb729f30bf83cd56596b53e935cb88f144539a20aaf9ac7d762",
-  "scimago_index.py": "63c76aff39873989cfdc287c1d3e9320b9bf839139e1cd65d0d7d8d4b6c53832",
-  "scimagojr_2025.csv": "a536c49b0854e96c66350c92b6f50dcfed26845e7c5545e5ebc65c1f96058fe2",
-  "style.css": "10762d70e0c991a6bb989c56bf6d22bb2e53ba902d1c9bc4ce7011d8d9f98699",
-  "subject_quartile_repair.json": "13fb100a9aaabacd9182551ccae8b531da8ae878789692402adde245d8300edb",
-  "tests/a": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-  "tests/conftest.py": "d8a6d008056392e4dbcf83d1970c8d077d94cd8363804727a8ca0c5fe4538edf",
-  "tests/test_frontend_links.cjs": "2b9eaab29a62de749219d1da27d2c6ea68c6774a17a205c837cbc187fc6fe1b9",
-  "tests/test_frontend_release_a.py": "ae8f029cd7a6e3ebc404c6cffdf35b3a9a3188cb65f93ff5f9c0cbcfb344947a",
-  "tests/test_journal_policy_guard.py": "27ece48b44981ba75a0335dbcca5ec5a09dfa57f5976b037e2dbd7155ff4990b",
-  "tests/test_khazna_reload_auth.py": "c9fe67180ca23b5d0adf0d8f91bd21e256d929a993a61f07a02ae7d111091013",
-  "tests/test_oaworks_tri_state.py": "0bd94f8af28d39c01c42a821ad79604699690d8801ffcc39e5189b6add5afee2",
-  "tests/test_repository_schema.py": "658880256ca07349737a52c602544a98153f0c995125970d13cc5fb9c9caf64f",
-  "tests/test_subject_quartiles.py": "c93529b2e2501f2e4f378dbd73d37cabb388040d907566404f19b97980789676",
-  "tests/test_trusted_links.py": "5d5ffb6547d86e20601374a8c637be162824b6947fdefec114d621921d49bcbc",
-  "tests/test_trusted_links_api.py": "c11c5e990d83a67627104f6ad41dcc1b3d8380a76354a4806ab9702e6dafc1b3",
-  "train_asjc.py": "caff809252a7e3234a4c22f113c4df56e2aafbcfa45398fcfbed3061755d1dae",
-  "trusted_links.py": "dfbc4490dfb4b80e1b1aa0089fcf39c6cf0866b42d5a032367dae255e25da004",
-  "usage.py": "5cbe805a1732ed5a11d2a420cf085b6baef63f5c5eff20de3126f6a208848df3",
-  "utils.py": "57e37e6b426159b5233657bfd442671150647c7b5f291a2d1a6ba5938b3a42b7"
-}
+{
+  ".dockerignore": "acbfa007f9cc614a581b5517709de7fb6df4a1836bc7c0c7bee7b96d020f3b83",
+  ".env.example": "b3ad9769d1306238eec831d7a818a21b15cd7cdb3df8b670d49831fb10beb6b8",
+  ".gitattributes": "4d72e9b97f1cb0fd76258dec00ff2a9ac929fafab43ac1e68da5d4ab82149622",
+  "DEPLOYMENT-LINKS.md": "f58b06a9ebf2558c3699025038c5671d1eb45c3e3ff989c9300ed586800a9347",
+  "Dockerfile": "a5289268fe287dd5677b37ce60017563f7294729bed078c935cfce3fc010d280",
+  "README-DEPLOY.md": "74103d9cc21b69354e5c978910edd7e22b24cf21c3ef657f82102178b0cd962c",
+  "README.md": "4b8cb6a850fdecf36e0d8efb70ec7fe9c916edbfa82db367d9126c9df51166e4",
+  "RELEASE_1.7.0.md": "334d4cb6215ff7ab540c539d3c3f8d2830641ced95c4f9325fdd86c78b7488ce",
+  "RELEASE_1.8.0.md": "4d85c1711c33cd314c799d27e59795b62fe9104656d79cd76bba6af538b9ad1e",
+  "RELEASE_1.8.1.md": "500ba91d0737c6198a453de2bd539827b019185c34d68d32d325a2dc5e106fa4",
+  "RELEASE_1.8.2.md": "b006bed1643a39a8f79930ffb682c37c64ba00a3752b9a056993791153810f88",
+  "RELEASE_1.8.3.md": "9d0aace78e561326ab1c74e63dc801029fc2c958b656d82f6325242a2476c491",
+  "RELEASE_A_NOTES.md": "2c9d4dd7b8e787ada145b2393578c6b3499370fb4be2a89100bd555ed8b96a42",
+  "RELEASE_OPF_WEB_1.5.2.md": "b4cc66479031664e35026fdbb57ec0a83468e1d54100ac022100b2849df67b77",
+  "ResearchBee_LibGuides.html": "2ad265c90e98d2d30ded0754d8fefb2f0137b3f1f0b8416aaacfc42718764219",
+  "VALIDATION_1.8.2.md": "2c5ae04d43ebe3e3867bd5fbb4bdf303cebf03a7dc85dddc0079844b3ec82f41",
+  "admin.py": "7db70eb130f8bc4d9e98f566ce86315fd2dfa4cbe90e3415dba037aa13d28421",
+  "app.py": "ded10bb7abc502c1142c6a513ff1f1d4950f70d4e47df5351cbb0172ef849a23",
+  "asjc_classifier.py": "c689f9d1fcd3ac15b4a0bd711153acbc9cb8780caed220f0c5d6afcde9af6fa6",
+  "asjc_eval.jsonl": "cc762d1d81532574a2270b98758324dc7c251b1d22a58604045ca5fa02a1b684",
+  "build_centroids.py": "48b3064a7b261195824fdedf84708ebb6e36954b9160d4b1722235d548e93e42",
+  "chat_knowledge.py": "d8b69d510ae847bb46f2c054a1458df7fe4a8db237dfb94ae515cfd4dff414ad",
+  "doi_utils.py": "7026a3913998b94aa4822f7cbbe9bb779ce52880f6e5a180959b37a19de9f8f9",
+  "harvest_openalex.py": "3c3cedd28c0892a10126249d892cd27275131d5295669d10d201ebae9ea4da3d",
+  "journal_discovery.py": "b69d282411eb4fecd3bcdb354cc34ac27f705b5b1d71ae600372d75a4672c12d",
+  "journal_links_curated.json": "e3566b3a06430868d71e9287dfd6c6c520a3da027aabea01951d407ee131dc2f",
+  "journal_lookup.py": "4d7e1638b48cfd1c908b217c3e4b8bf145932a9f34d3cb762092a99e4a817aa0",
+  "journals.db": "6553cefa4515cbeea4631fcd5f59a4b8bdb756fe83701788e8a11fb9298b2404",
+  "khazna_loader.py": "3e944cbfa116647ff548015b37e083134da9aa2aafbda0e957be6c31ea7ac9c7",
+  "knowledge_base.json": "1838c1d1fd5456c16308ebfaba237b8516aa01df5901a83e7290bb73b55b5bc6",
+  "ku_knowledge.py": "b54c5acd3c26783e5b44872897810d04ae0257df367b9fde62e8b6bb6e5f9745",
+  "oaworks.py": "491ae10ae479001fd1218d3d9c623cb39f4e8c222fc3bdf3420b2a8f03328410",
+  "openalex.py": "1f846a13c46d7dd4ed70494a552a1bfd596cf764d7fb7f8b89ac3389a9cf195c",
+  "openpolicyfinder.py": "9f55d2e744b6f784422a0731f01c0b727c3cc5b001da009889e8dcf83288f081",
+  "policy_evaluation.py": "287df1a54c2d4893cd495da6d95ee8049f480bf0ff75879e52f2b0ede7307dab",
+  "prompts.py": "5b40d1be17b41b8a51231378bf70dee4056d5373c1a739b1a964094f93850359",
+  "re3data.py": "fee9fb9a36c77b9ab2da9e63baa54f7f74b88a864ac5d39df0d6804f6904c25e",
+  "repair_subject_quartiles.py": "91ddce1c0a08fa802923ff1806d5333617f4262eca686ef50ae35b3f35dd5950",
+  "repositories.py": "f0f1673a2b73a8afc9be6b20a96bd3396336513550570d8b5048000ed75fa5a4",
+  "requirements-dev.txt": "e5bfd68d8ff1bdd7151344031c2e4cb97b187d099f681f4007b6d96ad7dfbdf8",
+  "requirements.txt": "179482f7f9e3177d035ef04101719117524476a98271dbafab3e3f18a17ce149",
+  "researchbee_journals.py": "623272ee184e37bf08b4fddd887f7aabd10b6c6cb1999416ddbcc0f5c72309c6",
+  "runtime_security.py": "8d3466aba8a11fb729f30bf83cd56596b53e935cb88f144539a20aaf9ac7d762",
+  "scimago_index.py": "8ec98c45527c7af397b9c3d0d1f4a23319e2bebbcb0c2642727559b9c3909a80",
+  "scimagojr_2025.csv": "a536c49b0854e96c66350c92b6f50dcfed26845e7c5545e5ebc65c1f96058fe2",
+  "style.css": "10762d70e0c991a6bb989c56bf6d22bb2e53ba902d1c9bc4ce7011d8d9f98699",
+  "subject_quartile_repair.json": "13fb100a9aaabacd9182551ccae8b531da8ae878789692402adde245d8300edb",
+  "tests/a": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+  "tests/conftest.py": "d8a6d008056392e4dbcf83d1970c8d077d94cd8363804727a8ca0c5fe4538edf",
+  "tests/fixtures/re3data_r3d100010134.xml": "7d6a2c0c6a655529dec1d69af12ec949369268d75ddaa7723e3ca5449d304683",
+  "tests/fixtures/re3data_r3d100012345.xml": "6ad59819be352abc72b34127aa59813ee128196651783d051224f6e7355331cd",
+  "tests/fixtures/re3data_r3d100012346.xml": "ef4d110920251213e1da2fa4de7ac34b047164e59923390e9df3a6f075b3f2c8",
+  "tests/fixtures/re3data_r3d100012347.xml": "7aea33c7ad7c38d7c6288a913237eaed235d40c0162af7bb5b7cd2f4a6cd3775",
+  "tests/fixtures/re3data_r3d100012348.xml": "2220f3b9ee3665707c161cec6b0d07019541a902a6f028d8aeb1ff6ae14de458",
+  "tests/fixtures/re3data_search_materials.xml": "9cd339477b3184a48027316baa576fcb367353356311c2c3e8bd3420f7673a83",
+  "tests/test_frontend_links.cjs": "33bf2edca82a17ef98df86221beb1ca0fddfaf52289dfc9a7cc161e657c7e551",
+  "tests/test_frontend_re3data.cjs": "1f934c661982b6e00a0676ac2e5b36b4aef302d5b0a4d41858ea8a24a68f0e41",
+  "tests/test_frontend_release_a.py": "ae8f029cd7a6e3ebc404c6cffdf35b3a9a3188cb65f93ff5f9c0cbcfb344947a",
+  "tests/test_journal_policy_guard.py": "27ece48b44981ba75a0335dbcca5ec5a09dfa57f5976b037e2dbd7155ff4990b",
+  "tests/test_khazna_reload_auth.py": "c9fe67180ca23b5d0adf0d8f91bd21e256d929a993a61f07a02ae7d111091013",
+  "tests/test_oaworks_tri_state.py": "0bd94f8af28d39c01c42a821ad79604699690d8801ffcc39e5189b6add5afee2",
+  "tests/test_re3data.py": "62388760aed721d86172bc617cc88ab3e2cb20eb3f2974aff9c2f3b5ff5f76a2",
+  "tests/test_repository_schema.py": "658880256ca07349737a52c602544a98153f0c995125970d13cc5fb9c9caf64f",
+  "tests/test_subject_nearest.py": "d9d4e9f07ae5399201fb376885171378d5f47842be4b41072c476e86ce80be06",
+  "tests/test_subject_quartiles.py": "c93529b2e2501f2e4f378dbd73d37cabb388040d907566404f19b97980789676",
+  "tests/test_trusted_links.py": "05e2e6a614ab61bbf44bb9b937ef00b831ec4f5e7c1dd2b2557c536e56feb8ba",
+  "tests/test_trusted_links_api.py": "b3d5d1f765fc451f13e514559a1ee53dbd1b7c51ad6a3736a47c75b878d619e0",
+  "train_asjc.py": "caff809252a7e3234a4c22f113c4df56e2aafbcfa45398fcfbed3061755d1dae",
+  "trusted_links.py": "8cbc9e55a9682af3991ea42cd040df89bdce02347919114bdb7236e97ad97c00",
+  "usage.py": "5cbe805a1732ed5a11d2a420cf085b6baef63f5c5eff20de3126f6a208848df3",
+  "utils.py": "57e37e6b426159b5233657bfd442671150647c7b5f291a2d1a6ba5938b3a42b7"
+}

journal_links_curated.json

@@ -1 +1 @@
-{}
+{}

openalex.py

@@ -1,307 +1,307 @@
-"""
-OpenAlex enrichment — journal metadata + peer percentile calculation.
-Optional server-side API key raises the available request budget.
-The obsolete sample-derived percentile is deliberately not reported.
-
-Changes from v1.0:
-- Shared module-level httpx.AsyncClient (init_client / close_client for lifespan)
-- compute_percentile accepts an optional concept_cache dict to deduplicate
-  identical concept peer-set fetches across journals in one request
-- _norm_issn removed — uses utils.norm_issn
-"""
-import httpx
-import asyncio
-import re
-import os
-import time
-import logging
-from collections import OrderedDict
-from datetime import datetime, timezone
-from typing import Optional
-from utils import norm_issn
-
-UA = "ResearchBee/1.8.2 (mailto:library@ku.ac.ae)"
-logger = logging.getLogger(__name__)
-_last_success = None
-_last_error = None
-_source_cache = OrderedDict()
-BASE = "https://api.openalex.org"
-
-_http_client: Optional[httpx.AsyncClient] = None
-
-
-async def _record_response(response) -> None:
-    global _last_success, _last_error
-    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
-    if response.is_success:
-        _last_success = now
-        _last_error = None
-    else:
-        _last_error = {"status": response.status_code, "at": now}
-
-
-def status() -> dict:
-    return {"api_key_configured": bool((os.getenv("OPENALEX_API_KEY") or "").strip()),
-            "last_success": _last_success, "last_error": _last_error,
-            "availability": "not_checked" if _last_success is None and _last_error is None else ("degraded" if _last_error else "available")}
-
-
-async def _authorize_request(request) -> None:
-    """Enforce the provider boundary before attaching its credential."""
-    url = request.url
-    if url.scheme != "https" or url.host != "api.openalex.org" or url.port not in (None, 443):
-        request.headers.pop("Authorization", None)
-        raise ValueError("OpenAlex client refuses non-provider origins")
-    key = (os.getenv("OPENALEX_API_KEY") or "").strip()
-    if key:
-        request.headers["Authorization"] = "Bearer " + key
-    else:
-        request.headers.pop("Authorization", None)
-
-
-def _record_failure(exc: Exception) -> None:
-    global _last_error
-    _last_error = {"type": type(exc).__name__,
-                   "at": datetime.now(timezone.utc).isoformat(timespec="seconds")}
-    status_code = getattr(exc, "status_code", None)
-    if status_code is not None:
-        _last_error["status"] = status_code
-
-
-class OpenAlexUnavailable(RuntimeError):
-    def __init__(self, status_code=None):
-        self.status_code = status_code
-        super().__init__("OpenAlex is temporarily unavailable")
-
-
-def init_client() -> None:
-    global _http_client
-    _http_client = httpx.AsyncClient(
-        timeout=httpx.Timeout(10.0, connect=5.0),
-        headers={"User-Agent": UA}, follow_redirects=False,
-        event_hooks={"request": [_authorize_request], "response": [_record_response]},
-    )
-
-
-async def close_client() -> None:
-    global _http_client
-    if _http_client and not _http_client.is_closed:
-        await _http_client.aclose()
-    _http_client = None
-
-
-def _get_client() -> httpx.AsyncClient:
-    if _http_client is None or _http_client.is_closed:
-        init_client()
-    return _http_client
-
-
-async def enrich_journal(name: str, issn: Optional[str]) -> Optional[dict]:
-    cache_key = (norm_issn(issn), name.strip().casefold())
-    cached = _source_cache.get(cache_key)
-    if cached and time.monotonic() - cached[0] < 3600:
-        _source_cache.move_to_end(cache_key)
-        return dict(cached[1])
-    client = _get_client()
-    try:
-        if issn:
-            clean = norm_issn(issn) or issn.strip()
-            r = await client.get(f"{BASE}/sources/issn:{clean}")
-        else:
-            r = await client.get(
-                f"{BASE}/sources", params={"search": name, "per_page": 1}
-            )
-        if not r.is_success:
-            return None
-        d = r.json()
-        s = d if issn else (d.get("results") or [None])[0]
-        if not s:
-            return None
-        # An ISSN request must be confirmed by the response, not merely the URL.
-        source_issns = {norm_issn(v) for v in [s.get("issn_l"), *(s.get("issn") or [])] if v}
-        if issn and norm_issn(issn) not in source_issns:
-            return None
-        if not issn:
-            from researchbee_journals import JournalIndex
-            if JournalIndex.normalise_identity_title(name) != JournalIndex.normalise_identity_title(s.get("display_name", "")):
-                return None
-        result = {
-            "source_updated_at": s.get("updated_date"),
-            "retrieved_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
-            "openalex_id":           s.get("id"),
-            "display_name":          s.get("display_name"),
-            "issn_l":                s.get("issn_l"),
-            "issns":                 s.get("issn", []),
-            "host_organization":     s.get("host_organization_name"),
-            "is_oa":                 s.get("is_oa"),
-            "is_in_doaj":            s.get("is_in_doaj"),
-            "works_count":           s.get("works_count"),
-            "cited_by_count":        s.get("cited_by_count"),
-            "h_index":               (s.get("summary_stats") or {}).get("h_index"),
-            "two_yr_mean_citedness": (s.get("summary_stats") or {}).get("2yr_mean_citedness"),
-            "homepage_url":          s.get("homepage_url"),
-            "top_concept":           ((s.get("x_concepts") or [{}])[0]).get("display_name"),
-            "top_concept_id":        ((s.get("x_concepts") or [{}])[0]).get("id"),
-        }
-        _source_cache[cache_key] = (time.monotonic(), result)
-        while len(_source_cache) > 512:
-            _source_cache.popitem(last=False)
-        return dict(result)
-    except Exception as e:
-        _record_failure(e)
-        logger.warning("OpenAlex source lookup failed (%s)", type(e).__name__)
-        return None
-
-
-async def compute_percentile(openalex: dict, concept_cache: Optional[dict] = None) -> Optional[dict]:
-    """A truncated high-citation peer sample cannot establish a field quartile."""
-    return None
-
-
-async def enrich_all(journals: list) -> list:
-    """Enrich a list of journal dicts in parallel, deduplicating concept peer-sets."""
-    concept_cache: dict = {}
-
-    async def _enrich_one(j: dict) -> dict:
-        oa = await enrich_journal(j.get("name", ""), j.get("issn"))
-        pct = await compute_percentile(oa, concept_cache) if oa else None
-        j["openalex"] = oa
-        j["oa_percentile"] = pct
-        if not j.get("issn") and oa:
-            j["issn"] = oa.get("issn_l") or (oa.get("issns") or [None])[0]
-        return j
-
-    return list(await asyncio.gather(*[_enrich_one(j) for j in journals]))
-
-
-_EVIDENCE_STOP = {
-    "this", "that", "with", "from", "into", "using", "used", "study",
-    "research", "paper", "article", "results", "method", "methods", "based",
-    "analysis", "approach", "between", "among", "their", "these", "those",
-}
-
-
-def _evidence_terms(text: str) -> set[str]:
-    words = re.findall(r"[A-Za-z][A-Za-z\-]{3,}", (text or "").lower())
-    return {w.strip("-") for w in words if w not in _EVIDENCE_STOP}
-
-
-def _title_overlap(query: str, titles: list[str]) -> int:
-    q = _evidence_terms(query)
-    if not q or not titles:
-        return 0
-    scores = []
-    for title in titles:
-        t = _evidence_terms(title)
-        if not t:
-            continue
-        overlap = len(q & t)
-        # Dice coefficient, converted to 0..100. OpenAlex already orders the
-        # search by relevance; this score is only a transparent local signal.
-        scores.append((2 * overlap / max(len(q) + len(t), 1)) * 100)
-    return round(max(scores, default=0.0))
-
-
-async def similar_works_for_journal(
-    name: str,
-    issn: str = "",
-    query: str = "",
-    *,
-    per_page: int = 3,
-) -> dict:
-    """Return representative recent works as evidence for journal fit.
-
-    Failure is explicitly labelled unavailable, separately from no matches. Journal
-    identity and ranking still come from the local verified database, so an
-    OpenAlex outage never removes or invents a recommendation.
-    """
-    source = await enrich_journal(name, issn or None)
-    if not source or not source.get("openalex_id"):
-        return {"source": source, "works": [], "similarity_score": 0, "status": "unavailable"}
-
-    source_id = str(source["openalex_id"]).rsplit("/", 1)[-1]
-    if not re.fullmatch(r"S\d+", source_id):
-        return {"source": source, "works": [], "similarity_score": 0, "status": "unavailable"}
-
-    current_year = datetime.now(timezone.utc).year
-    query_text = " ".join((query or "").split())[:500]
-    client = _get_client()
-
-    async def _fetch(use_search: bool) -> list[dict]:
-        params = {
-            "filter": (
-                f"primary_location.source.id:{source_id},is_retracted:false,"
-                f"from_publication_date:{current_year - 5}-01-01"
-            ),
-            "per_page": min(max(per_page, 1), 5),
-            "select": "id,title,publication_year,doi,cited_by_count,authorships",
-        }
-        if use_search and query_text:
-            params["search"] = query_text
-            params["sort"] = "relevance_score:desc"
-        else:
-            params["sort"] = "cited_by_count:desc"
-        response = await client.get(f"{BASE}/works", params=params, timeout=6.0)
-        if not response.is_success:
-            raise OpenAlexUnavailable(response.status_code)
-        works = []
-        for work in response.json().get("results", []):
-            title = " ".join(str(work.get("title") or "").split())
-            if not title:
-                continue
-            authors = [
-                (a.get("author") or {}).get("display_name", "")
-                for a in (work.get("authorships") or [])[:3]
-            ]
-            works.append({
-                "title": title,
-                "year": work.get("publication_year"),
-                "doi": work.get("doi", ""),
-                "cited_by": work.get("cited_by_count", 0),
-                "authors": [a for a in authors if a],
-                "openalex_url": work.get("id", ""),
-            })
-        return works
-
-    try:
-        works = await _fetch(bool(query_text))
-        if not works:
-            works = await _fetch(False)
-        score = _title_overlap(query_text, [w["title"] for w in works])
-        return {
-            "source": source,
-            "works": works,
-            "similarity_score": score,
-            "status": "available" if works else "no_recent_matches",
-        }
-    except Exception as exc:
-        _record_failure(exc)
-        logger.warning("OpenAlex evidence unavailable (%s)", type(exc).__name__)
-        return {"source": source, "works": [], "similarity_score": 0, "status": "unavailable"}
-
-
-async def evidence_for_journals(
-    journals: list[dict],
-    query: str,
-    *,
-    limit: int = 12,
-    per_page: int = 3,
-    concurrency: int = 5,
-) -> dict[str, dict]:
-    """Fetch representative works for the strongest verified candidates."""
-    semaphore = asyncio.Semaphore(max(1, concurrency))
-
-    async def _one(journal: dict) -> tuple[str, dict]:
-        key = str(journal.get("id") or journal.get("scopus_id") or journal.get("issn") or journal.get("name"))
-        async with semaphore:
-            evidence = await similar_works_for_journal(
-                journal.get("name", ""),
-                journal.get("issn", ""),
-                query,
-                per_page=per_page,
-            )
-        return key, evidence
-
-    selected = journals[: max(0, limit)]
-    pairs = await asyncio.gather(*[_one(j) for j in selected]) if selected else []
-    return {key: evidence for key, evidence in pairs}
+"""
+OpenAlex enrichment — journal metadata + peer percentile calculation.
+Optional server-side API key raises the available request budget.
+The obsolete sample-derived percentile is deliberately not reported.
+
+Changes from v1.0:
+- Shared module-level httpx.AsyncClient (init_client / close_client for lifespan)
+- compute_percentile accepts an optional concept_cache dict to deduplicate
+  identical concept peer-set fetches across journals in one request
+- _norm_issn removed — uses utils.norm_issn
+"""
+import httpx
+import asyncio
+import re
+import os
+import time
+import logging
+from collections import OrderedDict
+from datetime import datetime, timezone
+from typing import Optional
+from utils import norm_issn
+
+UA = "ResearchBee/1.8.2 (mailto:library@ku.ac.ae)"
+logger = logging.getLogger(__name__)
+_last_success = None
+_last_error = None
+_source_cache = OrderedDict()
+BASE = "https://api.openalex.org"
+
+_http_client: Optional[httpx.AsyncClient] = None
+
+
+async def _record_response(response) -> None:
+    global _last_success, _last_error
+    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+    if response.is_success:
+        _last_success = now
+        _last_error = None
+    else:
+        _last_error = {"status": response.status_code, "at": now}
+
+
+def status() -> dict:
+    return {"api_key_configured": bool((os.getenv("OPENALEX_API_KEY") or "").strip()),
+            "last_success": _last_success, "last_error": _last_error,
+            "availability": "not_checked" if _last_success is None and _last_error is None else ("degraded" if _last_error else "available")}
+
+
+async def _authorize_request(request) -> None:
+    """Enforce the provider boundary before attaching its credential."""
+    url = request.url
+    if url.scheme != "https" or url.host != "api.openalex.org" or url.port not in (None, 443):
+        request.headers.pop("Authorization", None)
+        raise ValueError("OpenAlex client refuses non-provider origins")
+    key = (os.getenv("OPENALEX_API_KEY") or "").strip()
+    if key:
+        request.headers["Authorization"] = "Bearer " + key
+    else:
+        request.headers.pop("Authorization", None)
+
+
+def _record_failure(exc: Exception) -> None:
+    global _last_error
+    _last_error = {"type": type(exc).__name__,
+                   "at": datetime.now(timezone.utc).isoformat(timespec="seconds")}
+    status_code = getattr(exc, "status_code", None)
+    if status_code is not None:
+        _last_error["status"] = status_code
+
+
+class OpenAlexUnavailable(RuntimeError):
+    def __init__(self, status_code=None):
+        self.status_code = status_code
+        super().__init__("OpenAlex is temporarily unavailable")
+
+
+def init_client() -> None:
+    global _http_client
+    _http_client = httpx.AsyncClient(
+        timeout=httpx.Timeout(10.0, connect=5.0),
+        headers={"User-Agent": UA}, follow_redirects=False,
+        event_hooks={"request": [_authorize_request], "response": [_record_response]},
+    )
+
+
+async def close_client() -> None:
+    global _http_client
+    if _http_client and not _http_client.is_closed:
+        await _http_client.aclose()
+    _http_client = None
+
+
+def _get_client() -> httpx.AsyncClient:
+    if _http_client is None or _http_client.is_closed:
+        init_client()
+    return _http_client
+
+
+async def enrich_journal(name: str, issn: Optional[str]) -> Optional[dict]:
+    cache_key = (norm_issn(issn), name.strip().casefold())
+    cached = _source_cache.get(cache_key)
+    if cached and time.monotonic() - cached[0] < 3600:
+        _source_cache.move_to_end(cache_key)
+        return dict(cached[1])
+    client = _get_client()
+    try:
+        if issn:
+            clean = norm_issn(issn) or issn.strip()
+            r = await client.get(f"{BASE}/sources/issn:{clean}")
+        else:
+            r = await client.get(
+                f"{BASE}/sources", params={"search": name, "per_page": 1}
+            )
+        if not r.is_success:
+            return None
+        d = r.json()
+        s = d if issn else (d.get("results") or [None])[0]
+        if not s:
+            return None
+        # An ISSN request must be confirmed by the response, not merely the URL.
+        source_issns = {norm_issn(v) for v in [s.get("issn_l"), *(s.get("issn") or [])] if v}
+        if issn and norm_issn(issn) not in source_issns:
+            return None
+        if not issn:
+            from researchbee_journals import JournalIndex
+            if JournalIndex.normalise_identity_title(name) != JournalIndex.normalise_identity_title(s.get("display_name", "")):
+                return None
+        result = {
+            "source_updated_at": s.get("updated_date"),
+            "retrieved_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+            "openalex_id":           s.get("id"),
+            "display_name":          s.get("display_name"),
+            "issn_l":                s.get("issn_l"),
+            "issns":                 s.get("issn", []),
+            "host_organization":     s.get("host_organization_name"),
+            "is_oa":                 s.get("is_oa"),
+            "is_in_doaj":            s.get("is_in_doaj"),
+            "works_count":           s.get("works_count"),
+            "cited_by_count":        s.get("cited_by_count"),
+            "h_index":               (s.get("summary_stats") or {}).get("h_index"),
+            "two_yr_mean_citedness": (s.get("summary_stats") or {}).get("2yr_mean_citedness"),
+            "homepage_url":          s.get("homepage_url"),
+            "top_concept":           ((s.get("x_concepts") or [{}])[0]).get("display_name"),
+            "top_concept_id":        ((s.get("x_concepts") or [{}])[0]).get("id"),
+        }
+        _source_cache[cache_key] = (time.monotonic(), result)
+        while len(_source_cache) > 512:
+            _source_cache.popitem(last=False)
+        return dict(result)
+    except Exception as e:
+        _record_failure(e)
+        logger.warning("OpenAlex source lookup failed (%s)", type(e).__name__)
+        return None
+
+
+async def compute_percentile(openalex: dict, concept_cache: Optional[dict] = None) -> Optional[dict]:
+    """A truncated high-citation peer sample cannot establish a field quartile."""
+    return None
+
+
+async def enrich_all(journals: list) -> list:
+    """Enrich a list of journal dicts in parallel, deduplicating concept peer-sets."""
+    concept_cache: dict = {}
+
+    async def _enrich_one(j: dict) -> dict:
+        oa = await enrich_journal(j.get("name", ""), j.get("issn"))
+        pct = await compute_percentile(oa, concept_cache) if oa else None
+        j["openalex"] = oa
+        j["oa_percentile"] = pct
+        if not j.get("issn") and oa:
+            j["issn"] = oa.get("issn_l") or (oa.get("issns") or [None])[0]
+        return j
+
+    return list(await asyncio.gather(*[_enrich_one(j) for j in journals]))
+
+
+_EVIDENCE_STOP = {
+    "this", "that", "with", "from", "into", "using", "used", "study",
+    "research", "paper", "article", "results", "method", "methods", "based",
+    "analysis", "approach", "between", "among", "their", "these", "those",
+}
+
+
+def _evidence_terms(text: str) -> set[str]:
+    words = re.findall(r"[A-Za-z][A-Za-z\-]{3,}", (text or "").lower())
+    return {w.strip("-") for w in words if w not in _EVIDENCE_STOP}
+
+
+def _title_overlap(query: str, titles: list[str]) -> int:
+    q = _evidence_terms(query)
+    if not q or not titles:
+        return 0
+    scores = []
+    for title in titles:
+        t = _evidence_terms(title)
+        if not t:
+            continue
+        overlap = len(q & t)
+        # Dice coefficient, converted to 0..100. OpenAlex already orders the
+        # search by relevance; this score is only a transparent local signal.
+        scores.append((2 * overlap / max(len(q) + len(t), 1)) * 100)
+    return round(max(scores, default=0.0))
+
+
+async def similar_works_for_journal(
+    name: str,
+    issn: str = "",
+    query: str = "",
+    *,
+    per_page: int = 3,
+) -> dict:
+    """Return representative recent works as evidence for journal fit.
+
+    Failure is explicitly labelled unavailable, separately from no matches. Journal
+    identity and ranking still come from the local verified database, so an
+    OpenAlex outage never removes or invents a recommendation.
+    """
+    source = await enrich_journal(name, issn or None)
+    if not source or not source.get("openalex_id"):
+        return {"source": source, "works": [], "similarity_score": 0, "status": "unavailable"}
+
+    source_id = str(source["openalex_id"]).rsplit("/", 1)[-1]
+    if not re.fullmatch(r"S\d+", source_id):
+        return {"source": source, "works": [], "similarity_score": 0, "status": "unavailable"}
+
+    current_year = datetime.now(timezone.utc).year
+    query_text = " ".join((query or "").split())[:500]
+    client = _get_client()
+
+    async def _fetch(use_search: bool) -> list[dict]:
+        params = {
+            "filter": (
+                f"primary_location.source.id:{source_id},is_retracted:false,"
+                f"from_publication_date:{current_year - 5}-01-01"
+            ),
+            "per_page": min(max(per_page, 1), 5),
+            "select": "id,title,publication_year,doi,cited_by_count,authorships",
+        }
+        if use_search and query_text:
+            params["search"] = query_text
+            params["sort"] = "relevance_score:desc"
+        else:
+            params["sort"] = "cited_by_count:desc"
+        response = await client.get(f"{BASE}/works", params=params, timeout=6.0)
+        if not response.is_success:
+            raise OpenAlexUnavailable(response.status_code)
+        works = []
+        for work in response.json().get("results", []):
+            title = " ".join(str(work.get("title") or "").split())
+            if not title:
+                continue
+            authors = [
+                (a.get("author") or {}).get("display_name", "")
+                for a in (work.get("authorships") or [])[:3]
+            ]
+            works.append({
+                "title": title,
+                "year": work.get("publication_year"),
+                "doi": work.get("doi", ""),
+                "cited_by": work.get("cited_by_count", 0),
+                "authors": [a for a in authors if a],
+                "openalex_url": work.get("id", ""),
+            })
+        return works
+
+    try:
+        works = await _fetch(bool(query_text))
+        if not works:
+            works = await _fetch(False)
+        score = _title_overlap(query_text, [w["title"] for w in works])
+        return {
+            "source": source,
+            "works": works,
+            "similarity_score": score,
+            "status": "available" if works else "no_recent_matches",
+        }
+    except Exception as exc:
+        _record_failure(exc)
+        logger.warning("OpenAlex evidence unavailable (%s)", type(exc).__name__)
+        return {"source": source, "works": [], "similarity_score": 0, "status": "unavailable"}
+
+
+async def evidence_for_journals(
+    journals: list[dict],
+    query: str,
+    *,
+    limit: int = 12,
+    per_page: int = 3,
+    concurrency: int = 5,
+) -> dict[str, dict]:
+    """Fetch representative works for the strongest verified candidates."""
+    semaphore = asyncio.Semaphore(max(1, concurrency))
+
+    async def _one(journal: dict) -> tuple[str, dict]:
+        key = str(journal.get("id") or journal.get("scopus_id") or journal.get("issn") or journal.get("name"))
+        async with semaphore:
+            evidence = await similar_works_for_journal(
+                journal.get("name", ""),
+                journal.get("issn", ""),
+                query,
+                per_page=per_page,
+            )
+        return key, evidence
+
+    selected = journals[: max(0, limit)]
+    pairs = await asyncio.gather(*[_one(j) for j in selected]) if selected else []
+    return {key: evidence for key, evidence in pairs}

re3data.py

@@ -0,0 +1,589 @@
+"""
+re3data.py — supplementary repository discovery from the re3data.org registry.
+
+WHY THIS EXISTS
+---------------
+The curated KU Library registry (repositories.py) stays the primary source:
+the model selects from it by id and every displayed fact comes from a
+reviewed record. That registry is intentionally small, so a dataset in a
+narrower field may only receive generalist suggestions.
+
+This module adds a clearly separated "more options" list from re3data.org,
+the Registry of Research Data Repositories (DataCite). It never changes the
+curated recommendations, the deposit strategy, or Khazna's position.
+
+EVIDENCE RULES
+--------------
+* Facts shown for these entries come only from the re3data record, labelled
+  as such, with the record's own last-update date. They are not reviewed by
+  KU Library and are never merged into curated entries.
+* Repositories already in the curated registry, repositories with an end
+  date (closed), repositories whose data upload is closed, and purely
+  institutional repositories (usually restricted to their own staff) are
+  excluded.
+* Only the researcher-entered *discipline* is sent to re3data, never the
+  dataset title, description or notes, which may be unpublished.
+
+FAILURE MODEL
+-------------
+Every network, parsing or validation failure returns an "unavailable"
+status and an empty list. Nothing here may raise into the route.
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import re
+import time
+from collections import OrderedDict
+from datetime import datetime, timezone
+from typing import Any, Dict, Iterable, List, Optional, Tuple
+from urllib.parse import urlparse
+import xml.etree.ElementTree as ET
+
+import httpx
+
+logger = logging.getLogger(__name__)
+
+API_HOST = "www.re3data.org"
+SEARCH_URL = f"https://{API_HOST}/api/beta/repositories"
+DETAIL_URL = f"https://{API_HOST}/api/v1/repository/{{rid}}"
+RECORD_URL = f"https://{API_HOST}/repository/{{rid}}"
+UA = "ResearchBee/1.8.3 (mailto:library@ku.ac.ae)"
+
+ID_RE = re.compile(r"^r3d\d{9}$")
+MAX_RESPONSE_BYTES = 2_000_000
+MAX_DETAIL_FETCHES = 20
+DETAIL_CONCURRENCY = 6
+SEARCH_TTL = 24 * 3600
+FAILURE_TTL = 10 * 60
+DETAIL_TTL = 7 * 24 * 3600
+CACHE_MAX = 256
+DETAIL_CACHE_MAX = 1024
+
+_STOPWORDS = {
+    "and", "of", "the", "in", "for", "or", "a", "an", "to", "with", "on", "data",
+    "dataset", "datasets", "research", "science", "sciences", "studies", "study",
+    "general", "applied", "other", "misc", "miscellaneous",
+}
+
+_http_client: Optional[httpx.AsyncClient] = None
+_search_cache: "OrderedDict[str, Tuple[float, Any]]" = OrderedDict()
+_detail_cache: "OrderedDict[str, Tuple[float, Optional[dict]]]" = OrderedDict()
+_inflight: Dict[str, asyncio.Future] = {}
+_last_success: Optional[str] = None
+_last_error: Optional[dict] = None
+
+
+def enabled() -> bool:
+    return (os.getenv("RE3DATA_ENABLED", "true").strip().lower() not in {"0", "false", "no", "off"})
+
+
+def _now_iso() -> str:
+    return datetime.now(timezone.utc).isoformat(timespec="seconds")
+
+
+def status() -> dict:
+    return {
+        "enabled": enabled(),
+        "last_success": _last_success,
+        "last_error": _last_error,
+        "availability": (
+            "disabled" if not enabled()
+            else "not_checked" if _last_success is None and _last_error is None
+            else "degraded" if _last_error else "available"
+        ),
+    }
+
+
+# ── HTTP client lifecycle ─────────────────────────────────────────────────
+
+async def _enforce_origin(request: httpx.Request) -> None:
+    url = request.url
+    if url.scheme != "https" or url.host != API_HOST or url.port not in (None, 443):
+        raise ValueError("re3data client refuses non-provider origins")
+
+
+def init_client() -> None:
+    global _http_client
+    if _http_client is None:
+        _http_client = httpx.AsyncClient(
+            timeout=httpx.Timeout(6.0, connect=4.0),
+            headers={"User-Agent": UA, "Accept": "application/xml, text/xml"},
+            follow_redirects=False,
+            event_hooks={"request": [_enforce_origin]},
+        )
+
+
+async def close_client() -> None:
+    global _http_client
+    if _http_client is not None:
+        await _http_client.aclose()
+        _http_client = None
+
+
+def _client() -> httpx.AsyncClient:
+    if _http_client is None:
+        init_client()
+    return _http_client  # type: ignore[return-value]
+
+
+# ── Bounded caches ────────────────────────────────────────────────────────
+
+def _cache_get(cache: OrderedDict, key: str):
+    item = cache.get(key)
+    if not item:
+        return None
+    expires, value = item
+    if expires < time.monotonic():
+        cache.pop(key, None)
+        return None
+    cache.move_to_end(key)
+    return item
+
+
+def _cache_put(cache: OrderedDict, key: str, value: Any, ttl: float, limit: int) -> None:
+    cache[key] = (time.monotonic() + ttl, value)
+    cache.move_to_end(key)
+    while len(cache) > limit:
+        cache.popitem(last=False)
+
+
+def clear_caches() -> None:
+    _search_cache.clear()
+    _detail_cache.clear()
+    _inflight.clear()
+
+
+# ── XML helpers (namespace-agnostic; the schema namespace has changed) ────
+
+def _local(tag: Any) -> str:
+    return tag.rsplit("}", 1)[-1] if isinstance(tag, str) else ""
+
+
+def _safe_parse(content: bytes) -> Optional[ET.Element]:
+    if not content or len(content) > MAX_RESPONSE_BYTES:
+        return None
+    head = content[:4096].upper()
+    # re3data never needs a DTD; refusing one blocks entity-expansion tricks.
+    if b"<!DOCTYPE" in head or b"<!ENTITY" in content.upper():
+        return None
+    try:
+        return ET.fromstring(content)
+    except ET.ParseError:
+        return None
+
+
+def _children(node: ET.Element, name: str) -> List[ET.Element]:
+    return [child for child in list(node) if _local(child.tag) == name]
+
+
+def _text(node: Optional[ET.Element]) -> str:
+    if node is None:
+        return ""
+    return " ".join("".join(node.itertext()).split())
+
+
+def _first_text(node: ET.Element, name: str) -> str:
+    for child in _children(node, name):
+        value = _text(child)
+        if value:
+            return value
+    return ""
+
+
+def _all_text(node: ET.Element, name: str, limit: int = 20, width: int = 200) -> List[str]:
+    out: List[str] = []
+    for child in node.iter():
+        if _local(child.tag) == name:
+            value = _text(child)[:width]
+            if value and value not in out:
+                out.append(value)
+            if len(out) >= limit:
+                break
+    return out
+
+
+def _https_url(value: str) -> str:
+    value = (value or "").strip()
+    try:
+        parsed = urlparse(value)
+    except ValueError:
+        return ""
+    if parsed.scheme != "https" or not parsed.hostname or "." not in parsed.hostname:
+        return ""
+    if parsed.username or parsed.password or (parsed.port not in (None, 443)):
+        return ""
+    return value[:500]
+
+
+def _host(value: str) -> str:
+    try:
+        host = (urlparse(value).hostname or "").lower()
+    except ValueError:
+        return ""
+    return host[4:] if host.startswith("www.") else host
+
+
+def parse_search(content: bytes) -> List[dict]:
+    """Parse a search/list response into [{"id", "name", "doi"}]."""
+    root = _safe_parse(content)
+    if root is None:
+        return []
+    out: List[dict] = []
+    seen = set()
+    for node in root.iter():
+        if _local(node.tag) != "repository":
+            continue
+        rid = _first_text(node, "id")
+        if not ID_RE.match(rid) or rid in seen:
+            continue
+        seen.add(rid)
+        out.append({
+            "id": rid,
+            "name": _first_text(node, "name")[:300],
+            "doi": _first_text(node, "doi")[:200],
+        })
+    return out
+
+
+def _yes_no_unknown(value: str) -> str:
+    value = (value or "").strip().lower()
+    if value == "yes":
+        return "Yes"
+    if value == "no":
+        return "No"
+    return "Unknown"
+
+
+_SUBJECT_CODE = re.compile(r"^\s*\d+\s+")
+
+
+def parse_detail(content: bytes) -> Optional[dict]:
+    """Parse one re3data repository record. Returns None if unusable."""
+    root = _safe_parse(content)
+    if root is None:
+        return None
+    repo = root if _local(root.tag) == "repository" else next(
+        (node for node in root.iter() if _local(node.tag) == "repository"), None
+    )
+    if repo is None:
+        return None
+    rid = _first_text(repo, "re3data.orgIdentifier")
+    name = _first_text(repo, "repositoryName")
+    if not ID_RE.match(rid) or not name:
+        return None
+
+    subjects: List[str] = []
+    for value in _all_text(repo, "subject", limit=30):
+        clean = _SUBJECT_CODE.sub("", value).strip()
+        if clean and clean not in subjects:
+            subjects.append(clean)
+
+    pid_systems = [p for p in _all_text(repo, "pidSystem", limit=8, width=40) if p.lower() != "none"]
+    upload_types = [v.lower() for v in _all_text(repo, "dataUploadType", limit=5, width=30)]
+    access_types = [v.lower() for v in _all_text(repo, "dataAccessType", limit=5, width=30)]
+    description = _first_text(repo, "description")
+    if len(description) > 600:
+        description = description[:597].rsplit(" ", 1)[0] + "…"
+
+    return {
+        "id": rid,
+        "name": name[:300],
+        "url": _https_url(_first_text(repo, "repositoryURL")),
+        "description": description,
+        "types": [t.lower() for t in _all_text(repo, "type", limit=5, width=40)],
+        "subjects": subjects[:10],
+        "keywords": _all_text(repo, "keyword", limit=15, width=80),
+        "content_types": [_SUBJECT_CODE.sub("", c) for c in _all_text(repo, "contentType", limit=10, width=120)],
+        "pid_systems": pid_systems,
+        "versioning": _yes_no_unknown(_first_text(repo, "versioning")),
+        "data_upload": upload_types,
+        "data_access": access_types,
+        "certificates": _all_text(repo, "certificate", limit=5, width=80),
+        "countries": _all_text(repo, "institutionCountry", limit=5, width=10),
+        "end_date": _first_text(repo, "endDate")[:30],
+        "last_update": _first_text(repo, "lastUpdate")[:30],
+    }
+
+
+# ── Network operations ────────────────────────────────────────────────────
+
+class Re3dataUnavailable(RuntimeError):
+    pass
+
+
+async def _get(url: str, params: Optional[dict] = None) -> bytes:
+    global _last_success, _last_error
+    try:
+        response = await _client().get(url, params=params)
+    except (httpx.HTTPError, ValueError) as exc:
+        _last_error = {"type": type(exc).__name__, "at": _now_iso()}
+        raise Re3dataUnavailable() from None
+    if response.status_code != 200:
+        _last_error = {"status": response.status_code, "at": _now_iso()}
+        raise Re3dataUnavailable()
+    content = response.content
+    if len(content) > MAX_RESPONSE_BYTES:
+        _last_error = {"type": "ResponseTooLarge", "at": _now_iso()}
+        raise Re3dataUnavailable()
+    _last_success = _now_iso()
+    _last_error = None
+    return content
+
+
+async def search(query: str) -> List[dict]:
+    """Search re3data; cached. Raises Re3dataUnavailable on failure."""
+    key = " ".join(query.lower().split())[:120]
+    cached = _cache_get(_search_cache, key)
+    if cached is not None:
+        value = cached[1]
+        if isinstance(value, Exception):
+            raise Re3dataUnavailable()
+        return value
+    pending = _inflight.get("s:" + key)
+    if pending is not None:
+        return await asyncio.shield(pending)
+
+    loop = asyncio.get_running_loop()
+    future: asyncio.Future = loop.create_future()
+    _inflight["s:" + key] = future
+    try:
+        content = await _get(SEARCH_URL, params={"query": key})
+        hits = parse_search(content)
+        _cache_put(_search_cache, key, hits, SEARCH_TTL, CACHE_MAX)
+        future.set_result(hits)
+        return hits
+    except Exception as exc:
+        _cache_put(_search_cache, key, Re3dataUnavailable(), FAILURE_TTL, CACHE_MAX)
+        future.set_exception(Re3dataUnavailable())
+        future.exception()  # mark retrieved; concurrent waiters still see it
+        raise Re3dataUnavailable() from None
+    finally:
+        _inflight.pop("s:" + key, None)
+
+
+async def detail(rid: str) -> Optional[dict]:
+    """Fetch one record; cached. Returns None when unavailable."""
+    if not ID_RE.match(rid or ""):
+        return None
+    cached = _cache_get(_detail_cache, rid)
+    if cached is not None:
+        return cached[1]
+    try:
+        content = await _get(DETAIL_URL.format(rid=rid))
+    except Re3dataUnavailable:
+        _cache_put(_detail_cache, rid, None, FAILURE_TTL, DETAIL_CACHE_MAX)
+        return None
+    record = parse_detail(content)
+    _cache_put(_detail_cache, rid, record, DETAIL_TTL if record else FAILURE_TTL, DETAIL_CACHE_MAX)
+    return record
+
+
+# ── Selection ─────────────────────────────────────────────────────────────
+
+def _tokens(text: str) -> List[str]:
+    out: List[str] = []
+    for token in re.findall(r"[a-z0-9]+", (text or "").lower()):
+        if len(token) > 2 and token not in _STOPWORDS and token not in out:
+            out.append(token)
+    return out
+
+
+def _stem(token: str) -> str:
+    for suffix in ("ies", "es", "s"):
+        if token.endswith(suffix) and len(token) - len(suffix) >= 4:
+            return token[: -len(suffix)] + ("y" if suffix == "ies" else "")
+    return token
+
+
+def _matches(tokens: Iterable[str], text: str) -> List[str]:
+    words = {_stem(w) for w in re.findall(r"[a-z0-9]+", (text or "").lower())}
+    return [t for t in tokens if _stem(t) in words]
+
+
+def build_query(discipline: str) -> str:
+    """Only the user's discipline field is sent; bounded and normalised."""
+    text = " ".join(str(discipline or "").split())
+    text = re.sub(r"[^\w\s\-&,/()]", " ", text, flags=re.UNICODE)
+    return " ".join(text.split())[:100]
+
+
+def _is_sensitive(constraints: dict) -> bool:
+    sensitivity = str(constraints.get("sensitivity") or "").strip().lower()
+    personal = str(constraints.get("contains_personal_data") or "").strip().lower()
+    return (bool(sensitivity) and sensitivity not in {"open", "none", "non-sensitive", "not sensitive"}) or (
+        bool(personal) and personal != "no"
+    )
+
+
+def _excluded_by_registry(record: dict, curated: Iterable[dict]) -> bool:
+    name = " ".join(record.get("name", "").lower().split())
+    host = _host(record.get("url", ""))
+    for entry in curated:
+        entry_name = " ".join(str(entry.get("name", "")).lower().split())
+        if name and (name == entry_name or (len(entry_name) > 4 and entry_name in name)):
+            return True
+        entry_host = _host(str(entry.get("url", "")))
+        if host and entry_host and (host == entry_host or host.endswith("." + entry_host)):
+            return True
+    return False
+
+
+def score_record(record: dict, tokens: List[str]) -> Tuple[int, List[str]]:
+    """Relevance from subject/keyword/name evidence plus deposit-readiness."""
+    subject_text = " ".join(record.get("subjects", []) + record.get("keywords", []) + [record.get("name", "")])
+    strong = _matches(tokens, subject_text)
+    weak = [t for t in _matches(tokens, record.get("description", "")) if t not in strong]
+    if not strong:
+        return 0, []
+    score = 10 * len(strong) + 3 * len(weak)
+    pids = {p.lower() for p in record.get("pid_systems", [])}
+    if "doi" in pids:
+        score += 4
+    if record.get("certificates"):
+        score += 3
+    if "open" in record.get("data_upload", []):
+        score += 2
+    if record.get("versioning") == "Yes":
+        score += 1
+    return score, strong + weak
+
+
+def _public_entry(record: dict, matched: List[str], sensitive: bool) -> dict:
+    pids = record.get("pid_systems", [])
+    upload = record.get("data_upload", [])
+    access = record.get("data_access", [])
+    notes = [
+        "Listed in re3data.org; not reviewed by KU Library. Confirm deposit eligibility, costs, "
+        "licences and file limits on the repository's own site before uploading."
+    ]
+    if upload and "open" not in upload:
+        notes.append("re3data lists data upload as restricted; depositor eligibility must be confirmed.")
+    if sensitive:
+        notes.append(
+            "Your dataset is marked sensitive or personal: KU Library and the responsible ethics/data-protection "
+            "team must approve any repository and access conditions before upload."
+        )
+    last_update = record.get("last_update", "")
+    return {
+        "id": record["id"],
+        "name": record["name"],
+        "url": record.get("url", ""),
+        "re3data_url": RECORD_URL.format(rid=record["id"]),
+        "description": record.get("description", ""),
+        "repository_types": record.get("types", []),
+        "subjects": record.get("subjects", [])[:6],
+        "content_types": record.get("content_types", [])[:6],
+        "persistent_identifier": ", ".join(pids) if pids else "Not listed",
+        "doi_assignment": "Yes" if any(p.lower() == "doi" for p in pids) else "Not listed",
+        "versioning": record.get("versioning", "Unknown"),
+        "data_upload": ", ".join(u.capitalize() for u in upload) if upload else "Not listed",
+        "data_access": ", ".join(a.capitalize() for a in access) if access else "Not listed",
+        "certificates": record.get("certificates", []),
+        "countries": record.get("countries", []),
+        "last_update": last_update,
+        "matched_terms": matched[:6],
+        "source": "re3data",
+        "source_note": (
+            "Details as recorded by re3data.org"
+            + (f" (record last updated {last_update[:10]})" if last_update else "")
+            + "."
+        ),
+        "review_notes": notes,
+        "file_deposit_recommended": False,
+    }
+
+
+async def find_supplementary(
+    discipline: str,
+    constraints: Optional[dict] = None,
+    curated: Iterable[dict] = (),
+    limit: int = 5,
+) -> dict:
+    """Return {"status", "query", "repositories", "message", "retrieved_at"}.
+
+    Never raises. ``curated`` is the KU registry, used only for de-duplication.
+    """
+    constraints = constraints if isinstance(constraints, dict) else {}
+    query = build_query(discipline)
+    base = {"status": "skipped", "query": query, "repositories": [], "message": "",
+            "source": "re3data.org", "source_url": "https://www.re3data.org",
+            "retrieved_at": _now_iso()}
+    if not enabled():
+        return {**base, "status": "disabled",
+                "message": "re3data search is switched off for this service."}
+    tokens = _tokens(query)
+    if not tokens:
+        return {**base, "message": "Enter a discipline to search re3data for additional repositories."}
+
+    curated = list(curated)
+    try:
+        hits = await search(query)
+        if len(hits) < 3 and len(tokens) > 1:
+            # A multi-word discipline can be too narrow for the registry's
+            # search; retry once with the most distinctive (longest) word.
+            extra = await search(max(tokens, key=len))
+            seen = {h["id"] for h in hits}
+            hits += [h for h in extra if h["id"] not in seen]
+    except Re3dataUnavailable:
+        return {**base, "status": "unavailable",
+                "message": "re3data.org could not be reached. The curated KU Library recommendations are unaffected."}
+    except Exception as exc:  # defensive: never break the repository finder
+        logger.warning("[re3data] search failed (%s)", type(exc).__name__)
+        return {**base, "status": "unavailable",
+                "message": "re3data.org could not be searched. The curated KU Library recommendations are unaffected."}
+
+    if not hits:
+        return {**base, "status": "no_matches",
+                "message": f'No additional re3data repositories were found for "{query}".'}
+
+    # Fetch the hits most likely to be relevant first: name matches lead.
+    ordered = sorted(enumerate(hits), key=lambda item: (-len(_matches(tokens, item[1].get("name", ""))), item[0]))
+    candidates = [hit for _i, hit in ordered[:MAX_DETAIL_FETCHES]]
+    semaphore = asyncio.Semaphore(DETAIL_CONCURRENCY)
+
+    async def fetch(rid: str) -> Optional[dict]:
+        async with semaphore:
+            try:
+                return await detail(rid)
+            except Exception as exc:  # defensive
+                logger.warning("[re3data] detail failed (%s)", type(exc).__name__)
+                return None
+
+    records = await asyncio.gather(*(fetch(hit["id"]) for hit in candidates))
+    sensitive = _is_sensitive(constraints)
+    scored: List[Tuple[int, str, dict, List[str]]] = []
+    for record in records:
+        if not record:
+            continue
+        if record.get("end_date"):
+            continue  # closed repository
+        upload = record.get("data_upload", [])
+        if upload and all(u == "closed" for u in upload):
+            continue  # does not accept deposits
+        types = set(record.get("types", []))
+        if types and types <= {"institutional"}:
+            continue  # usually limited to that institution's own members
+        if _excluded_by_registry(record, curated):
+            continue
+        if sensitive:
+            access = set(record.get("data_access", []))
+            if not access & {"restricted", "embargoed", "closed"}:
+                continue  # no controlled-access route recorded
+        score, matched = score_record(record, tokens)
+        if score <= 0:
+            continue
+        scored.append((score, record["name"].lower(), record, matched))
+
+    scored.sort(key=lambda item: (-item[0], item[1]))
+    picked = [_public_entry(record, matched, sensitive) for _s, _n, record, matched in scored[: max(0, limit)]]
+    if not picked:
+        return {**base, "status": "no_matches",
+                "message": f're3data returned results for "{query}", but none passed the subject and deposit checks.'}
+    return {**base, "status": "ok", "repositories": picked,
+            "message": (
+                f'{len(picked)} additional repositor{"y" if len(picked) == 1 else "ies"} from re3data.org matched "{query}". '
+                "These are registry listings, not KU Library recommendations."
+            )}

repositories.py

@@ -1,708 +1,708 @@
-"""
-repositories.py — curated registry of research data repositories.
-
-WHY THIS EXISTS
----------------
-The previous /api/find-repository asked the LLM to produce repository facts:
-max file size, certification, cost, persistent identifier type. Those are
-exactly the kind of specific, checkable claims a model gets subtly wrong, and
-a wrong "50 GB limit" or "CoreTrustSeal certified" in a library tool is a
-credibility problem.
-
-Here the LLM only SELECTS entries by id and writes a fit_reason. Every factual
-field below is served from this file and overwrites whatever the model says.
-
-MAINTENANCE
------------
-Each entry carries `source_url` and `verified` (YYYY-MM). Re-check quarterly.
-Fields you are unsure of should be left as "" rather than guessed — the UI
-renders empty fields as "Not specified", which is honest.
-
-Selection is deliberately KU-flavoured: engineering, energy, materials,
-petroleum, AI/CS and biomedical are weighted because that is KU's output
-profile.
-"""
-
-from __future__ import annotations
-
-from typing import Any
-import json
-import re
-
-# ── Registry ───────────────────────────────────────────────────────────────
-# NOTE: max_file_size / certification are left blank where not personally
-# verified. Fill them in from source_url and bump `verified`.
-
-REGISTRY: dict[str, dict[str, Any]] = {
-    # ---- Institutional ----------------------------------------------------
-    "khazna": {
-        "id": "khazna",
-        "name": "Khazna",
-        "type": "Institutional",
-        "url": "https://khazna.ku.ac.ae",
-        "scope": "Khalifa University research outputs, datasets and theses across all disciplines.",
-        "data_types_accepted": "Confirm file-deposit support with KU Library",
-        "persistent_identifier": "",
-        "versioning": "",
-        "embargo_support": "",
-        "cost": "Free for KU researchers",
-        "curation": "Mediated — library review before publication",
-        "source_url": "https://khazna.ku.ac.ae",
-        "verified": "2026-07",
-        "is_khazna": True,
-        "disciplines": ["all"],
-    },
-
-    # ---- Generalist -------------------------------------------------------
-    "zenodo": {
-        "id": "zenodo", "name": "Zenodo", "type": "Generalist",
-        "url": "https://zenodo.org",
-        "scope": "Multidisciplinary repository operated by CERN; accepts data, software, publications.",
-        "data_types_accepted": "All types", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free",
-        "curation": "Unmediated",
-        "source_url": "https://help.zenodo.org", "verified": "2026-07",
-        "disciplines": ["all"],
-    },
-    "figshare": {
-        "id": "figshare", "name": "Figshare", "type": "Generalist",
-        "url": "https://figshare.com",
-        "scope": "Multidisciplinary repository for figures, datasets, media and preprints.",
-        "data_types_accepted": "All types", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free tier available",
-        "curation": "Unmediated",
-        "source_url": "https://help.figshare.com", "verified": "2026-07",
-        "disciplines": ["all"],
-    },
-    "dryad": {
-        "id": "dryad", "name": "Dryad", "type": "Generalist",
-        "url": "https://datadryad.org",
-        "scope": "Curated repository for data underlying published research, strong in life and environmental sciences.",
-        "data_types_accepted": "Research data (not software)", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Fee-based; free where the institution is a member",
-        "curation": "Mediated — curator review",
-        "source_url": "https://datadryad.org/stash/faq", "verified": "2026-07",
-        "disciplines": ["life sciences", "environmental", "ecology", "biology"],
-    },
-    "dataverse": {
-        "id": "dataverse", "name": "Harvard Dataverse", "type": "Generalist",
-        "url": "https://dataverse.harvard.edu",
-        "scope": "Multidisciplinary repository with strong social science and statistical tooling.",
-        "data_types_accepted": "All types", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free",
-        "curation": "Unmediated",
-        "source_url": "https://support.dataverse.harvard.edu", "verified": "2026-07",
-        "disciplines": ["social sciences", "economics", "political science", "all"],
-    },
-    "osf": {
-        "id": "osf", "name": "Open Science Framework (OSF)", "type": "Generalist",
-        "url": "https://osf.io",
-        "scope": "Project-based platform combining data, preprints, preregistration and materials.",
-        "data_types_accepted": "All types", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free",
-        "curation": "Unmediated",
-        "source_url": "https://help.osf.io", "verified": "2026-07",
-        "disciplines": ["psychology", "social sciences", "all"],
-    },
-    "mendeley_data": {
-        "id": "mendeley_data", "name": "Mendeley Data", "type": "Generalist",
-        "url": "https://data.mendeley.com",
-        "scope": "Multidisciplinary data repository operated by Elsevier.",
-        "data_types_accepted": "All types", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free tier available",
-        "curation": "Unmediated",
-        "source_url": "https://data.mendeley.com", "verified": "2026-07",
-        "disciplines": ["all"],
-    },
-
-    # ---- Engineering / energy / materials — KU core ------------------------
-    "materials_cloud": {
-        "id": "materials_cloud", "name": "Materials Cloud", "type": "Domain-specific",
-        "url": "https://www.materialscloud.org",
-        "scope": "Computational materials science data, workflows and simulations.",
-        "data_types_accepted": "Simulation data, structures, workflows",
-        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://www.materialscloud.org/home", "verified": "2026-07",
-        "disciplines": ["materials", "chemistry", "physics", "computational"],
-    },
-    "nomad": {
-        "id": "nomad", "name": "NOMAD", "type": "Domain-specific",
-        "url": "https://nomad-lab.eu",
-        "scope": "Repository for computational materials science, especially DFT calculations.",
-        "data_types_accepted": "Electronic-structure calculation outputs",
-        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Unmediated",
-        "source_url": "https://nomad-lab.eu", "verified": "2026-07",
-        "disciplines": ["materials", "physics", "chemistry", "computational"],
-    },
-    "oedi": {
-        "id": "oedi", "name": "Open Energy Data Initiative (OEDI)", "type": "Domain-specific",
-        "url": "https://data.openei.org",
-        "scope": "Energy systems, renewables, grid and buildings data (US DOE).",
-        "data_types_accepted": "Energy datasets, time series, geospatial",
-        "persistent_identifier": "DOI", "versioning": "", "embargo_support": "",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://data.openei.org", "verified": "2026-07",
-        "disciplines": ["energy", "renewables", "engineering", "sustainability"],
-    },
-    "ieee_dataport": {
-        "id": "ieee_dataport", "name": "IEEE DataPort", "type": "Domain-specific",
-        "url": "https://ieee-dataport.org",
-        "scope": "Datasets across electrical engineering, computing, communications and signal processing.",
-        "data_types_accepted": "All types", "persistent_identifier": "DOI",
-        "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free for open datasets; subscription for some features",
-        "curation": "Unmediated",
-        "source_url": "https://ieee-dataport.org", "verified": "2026-07",
-        "disciplines": ["engineering", "electrical", "computing", "communications", "signal processing"],
-    },
-
-    # ---- Earth / environment / geospatial ---------------------------------
-    "pangaea": {
-        "id": "pangaea", "name": "PANGAEA", "type": "Domain-specific",
-        "url": "https://www.pangaea.de",
-        "scope": "Earth and environmental science data; georeferenced observations.",
-        "data_types_accepted": "Georeferenced environmental data",
-        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated — editorial review",
-        "source_url": "https://www.pangaea.de/submit/", "verified": "2026-07",
-        "disciplines": ["earth science", "environmental", "climate", "oceanography", "geoscience"],
-    },
-    "esa_earth": {
-        "id": "esa_earth", "name": "ESA Earth Online", "type": "Domain-specific",
-        "url": "https://earth.esa.int",
-        "scope": "Earth observation and satellite remote-sensing datasets.",
-        "data_types_accepted": "Satellite imagery, EO products",
-        "persistent_identifier": "", "versioning": "", "embargo_support": "",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://earth.esa.int", "verified": "2026-07",
-        "disciplines": ["remote sensing", "earth observation", "geospatial", "climate"],
-    },
-
-    # ---- Life sciences / biomedical ---------------------------------------
-    "ena": {
-        "id": "ena", "name": "European Nucleotide Archive (ENA)", "type": "Domain-specific",
-        "url": "https://www.ebi.ac.uk/ena",
-        "scope": "Nucleotide sequencing data — raw reads, assemblies, annotations.",
-        "data_types_accepted": "Sequence data (FASTQ, BAM, assemblies)",
-        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated — format validation",
-        "source_url": "https://www.ebi.ac.uk/ena/browser/submit", "verified": "2026-07",
-        "disciplines": ["genomics", "biology", "life sciences", "sequencing", "microbiology"],
-    },
-    "geo": {
-        "id": "geo", "name": "NCBI GEO", "type": "Domain-specific",
-        "url": "https://www.ncbi.nlm.nih.gov/geo",
-        "scope": "Functional genomics data — gene expression, microarray, high-throughput sequencing.",
-        "data_types_accepted": "Expression and functional genomics data",
-        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://www.ncbi.nlm.nih.gov/geo/info/submission.html", "verified": "2026-07",
-        "disciplines": ["genomics", "transcriptomics", "biology", "life sciences"],
-    },
-    "pride": {
-        "id": "pride", "name": "PRIDE", "type": "Domain-specific",
-        "url": "https://www.ebi.ac.uk/pride",
-        "scope": "Mass-spectrometry proteomics data.",
-        "data_types_accepted": "Proteomics MS data",
-        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://www.ebi.ac.uk/pride/markdownpage/submitdatapage", "verified": "2026-07",
-        "disciplines": ["proteomics", "biochemistry", "life sciences"],
-    },
-    "pdb": {
-        "id": "pdb", "name": "Protein Data Bank (PDB)", "type": "Domain-specific",
-        "url": "https://www.rcsb.org",
-        "scope": "Three-dimensional structural data for biological macromolecules.",
-        "data_types_accepted": "Macromolecular structures",
-        "persistent_identifier": "PDB ID", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated — biocuration",
-        "source_url": "https://www.rcsb.org/pages/policy", "verified": "2026-07",
-        "disciplines": ["structural biology", "biochemistry", "chemistry", "life sciences"],
-    },
-    "metabolights": {
-        "id": "metabolights", "name": "MetaboLights", "type": "Domain-specific",
-        "url": "https://www.ebi.ac.uk/metabolights",
-        "scope": "Metabolomics experiments and derived information.",
-        "data_types_accepted": "Metabolomics data",
-        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://www.ebi.ac.uk/metabolights", "verified": "2026-07",
-        "disciplines": ["metabolomics", "biochemistry", "life sciences"],
-    },
-
-    # ---- Chemistry --------------------------------------------------------
-    "chemrxiv_data": {
-        "id": "chemrxiv_data", "name": "Cambridge Structural Database (CCDC)",
-        "type": "Domain-specific", "url": "https://www.ccdc.cam.ac.uk",
-        "scope": "Small-molecule organic and metal-organic crystal structures.",
-        "data_types_accepted": "Crystallographic data (CIF)",
-        "persistent_identifier": "CCDC number", "versioning": "", "embargo_support": "Yes",
-        "cost": "Free deposition", "curation": "Mediated",
-        "source_url": "https://www.ccdc.cam.ac.uk/structures/", "verified": "2026-07",
-        "disciplines": ["chemistry", "crystallography", "materials"],
-    },
-
-    # ---- Computing / AI ---------------------------------------------------
-    "huggingface_datasets": {
-        "id": "huggingface_datasets", "name": "Hugging Face Datasets",
-        "type": "Domain-specific", "url": "https://huggingface.co/datasets",
-        "scope": "Machine learning datasets and models with versioned git-based storage.",
-        "data_types_accepted": "ML datasets, models",
-        "persistent_identifier": "", "versioning": "Yes", "embargo_support": "",
-        "cost": "Free tier available", "curation": "Unmediated",
-        "source_url": "https://huggingface.co/docs/hub/datasets", "verified": "2026-07",
-        "disciplines": ["machine learning", "ai", "nlp", "computer vision", "computing"],
-    },
-    "openml": {
-        "id": "openml", "name": "OpenML", "type": "Domain-specific",
-        "url": "https://www.openml.org",
-        "scope": "Machine learning datasets, tasks, flows and benchmark runs.",
-        "data_types_accepted": "Tabular ML datasets",
-        "persistent_identifier": "", "versioning": "Yes", "embargo_support": "",
-        "cost": "Free", "curation": "Unmediated",
-        "source_url": "https://docs.openml.org", "verified": "2026-07",
-        "disciplines": ["machine learning", "ai", "data science", "computing"],
-    },
-
-    # ---- Social science / humanities --------------------------------------
-    "ukds": {
-        "id": "ukds", "name": "UK Data Service ReShare", "type": "Domain-specific",
-        "url": "https://reshare.ukdataservice.ac.uk",
-        "scope": "Social and economic research data, including sensitive data under controlled access.",
-        "data_types_accepted": "Social science data",
-        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated",
-        "source_url": "https://reshare.ukdataservice.ac.uk", "verified": "2026-07",
-        "disciplines": ["social sciences", "economics", "sociology", "survey"],
-    },
-    "icpsr": {
-        "id": "icpsr", "name": "ICPSR", "type": "Domain-specific",
-        "url": "https://www.icpsr.umich.edu",
-        "scope": "Social and behavioural science data with strong curation and restricted-access options.",
-        "data_types_accepted": "Social science data",
-        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Varies by membership", "curation": "Mediated — extensive curation",
-        "source_url": "https://www.icpsr.umich.edu/web/pages/deposit/", "verified": "2026-07",
-        "disciplines": ["social sciences", "behavioural", "health", "survey"],
-    },
-
-    # ---- Sensitive / controlled access -------------------------------------
-    "ega": {
-        "id": "ega", "name": "European Genome-phenome Archive (EGA)",
-        "type": "Domain-specific", "url": "https://ega-archive.org",
-        "scope": "Controlled-access repository for personally identifiable genetic and phenotypic data.",
-        "data_types_accepted": "Human genomic and phenotypic data",
-        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
-        "cost": "Free", "curation": "Mediated — DAC-controlled access",
-        "source_url": "https://ega-archive.org/submission", "verified": "2026-07",
-        "disciplines": ["human genomics", "clinical", "biomedical", "sensitive"],
-    },
-}
-
-
-# ── Helpers ────────────────────────────────────────────────────────────────
-# The model may author ONLY analytical fit fields. Every other repository
-# attribute is supplied from REGISTRY, including the id itself after lookup.
-MODEL_ALLOWED = ("fit_reason", "fit_confidence")
-
-
-def _clean_text(value: Any, limit: int = 1000) -> str:
-    return " ".join(str(value or "").split())[:limit]
-
-
-def _clean_list(value: Any, limit: int = 12, item_limit: int = 500) -> list[str]:
-    if not isinstance(value, list):
-        return []
-    out: list[str] = []
-    for item in value[:limit]:
-        cleaned = _clean_text(item, item_limit)
-        if cleaned:
-            out.append(cleaned)
-    return out
-
-
-def _yes_no(value: Any) -> str:
-    text = _clean_text(value, 100).casefold()
-    if text in {"yes", "true"} or text.startswith("yes ") or text.startswith("yes—"):
-        return "Yes"
-    if text in {"no", "false"}:
-        return "No"
-    return "Unknown"
-
-
-def repository_capabilities(entry: dict) -> dict:
-    """Expose only capabilities supported by existing curated fields.
-
-    A generic homepage and a broad access description do not establish approval
-    for a particular sensitive dataset, size limit or accepted file format.
-    """
-    identifier = _clean_text(entry.get("persistent_identifier"), 100)
-    description = (str(entry.get("scope", "")) + " " + str(entry.get("curation", ""))).casefold()
-    controlled = any(term in description for term in ("controlled access", "controlled-access", "restricted-access", "restricted access"))
-    return {
-        "doi": "Yes" if identifier.casefold() == "doi" else "No" if identifier else "Unknown",
-        "persistent_id_type": identifier or "Unknown",
-        "versioning": _yes_no(entry.get("versioning")),
-        "embargo": _yes_no(entry.get("embargo_support")),
-        "restricted_access": "Yes" if controlled and not entry.get("is_khazna") else "Unknown",
-        "approved_sensitive_data": "Unknown",
-        "accepts_external_deposits": "Unknown",
-        "accepted_file_formats": "Unknown",
-        "maximum_dataset_size": "Unknown",
-        "source_url": entry.get("source_url", ""),
-        "reviewed": entry.get("verified", ""),
-    }
-
-
-def evaluate_repository(entry: dict, constraints: dict | None = None) -> dict:
-    """Check stated requirements against curated facts; unknown is not Yes."""
-    constraints = constraints if isinstance(constraints, dict) else {}
-    capabilities = repository_capabilities(entry)
-    unresolved, conflicts = [], []
-    sensitivity = _clean_text(constraints.get("sensitivity"), 100).casefold()
-    personal = _clean_text(constraints.get("contains_personal_data"), 100).casefold()
-    sensitive = (bool(sensitivity) and sensitivity not in {"open", "none", "non-sensitive", "not sensitive"}) or (bool(personal) and personal != "no")
-    for requirement, capability, label in (
-        ("needs_doi", "doi", "DOI assignment"),
-        ("needs_versioning", "versioning", "versioning"),
-        ("embargo_required", "embargo", "an embargo"),
-    ):
-        if _yes_no(constraints.get(requirement)) != "Yes":
-            continue
-        if capabilities[capability] == "No":
-            conflicts.append(f"The curated record does not support the requested {label}.")
-        elif capabilities[capability] != "Yes":
-            unresolved.append(f"Confirm support for {label} with the repository.")
-    if sensitive:
-        if capabilities["restricted_access"] != "Yes":
-            conflicts.append("Controlled/restricted access suitable for the submitted sensitive or personal data is not established.")
-        unresolved.append("KU Library and the responsible ethics/data-protection team must approve the repository and access conditions for this dataset before any file upload.")
-    if _clean_text(constraints.get("file_formats")):
-        unresolved.append("Confirm acceptance of the submitted file formats: " + _clean_text(constraints["file_formats"], 300))
-    if _clean_text(constraints.get("approx_size")):
-        unresolved.append("Confirm the size/quota and any associated cost for: " + _clean_text(constraints["approx_size"], 300))
-    if _clean_text(constraints.get("funder")):
-        unresolved.append("Confirm the funder's repository, licence and retention requirements; the registry does not establish mandate compliance.")
-    if _clean_text(constraints.get("licence_intent")):
-        unresolved.append("Confirm that the repository supports the selected licence and that you hold the necessary rights.")
-    institution = _clean_text(constraints.get("institution"))
-    country = _clean_text(constraints.get("country"))
-    if institution or country:
-        unresolved.append("Confirm depositor eligibility and any institution/country restrictions.")
-    if _yes_no(constraints.get("embargo_required")) == "Yes":
-        duration = constraints.get("embargo_months")
-        if duration is not None:
-            unresolved.append(f"Confirm support for the requested {duration}-month embargo.")
-        elif constraints.get("embargo_duration"):
-            unresolved.append("Confirm support for the requested embargo duration: " + _clean_text(constraints["embargo_duration"], 100))
-    if entry.get("id") == "dryad" and "software" in _clean_text(constraints.get("data_types")).casefold():
-        conflicts.append("The curated record lists research data, not software, as accepted content.")
-    metadata_only = bool(entry.get("is_khazna"))
-    if metadata_only:
-        unresolved.insert(0, "Ask KU Library about metadata registration; dataset file acceptance, DOI assignment and access controls require confirmation.")
-        # Institutional registration advice is not a file-deposit destination.
-        conflicts = []
-    return {
-        "state": "metadata_only" if metadata_only else "does_not_meet" if conflicts else "conditional" if unresolved else "meets",
-        "unresolved_constraints": list(dict.fromkeys(unresolved)),
-        "constraint_conflicts": list(dict.fromkeys(conflicts)),
-        "sensitive_data_review_required": sensitive,
-        "file_deposit_recommended": not metadata_only and not conflicts and not unresolved,
-        "strategy_eligible": not metadata_only and not conflicts and not sensitive,
-        "capabilities": capabilities,
-    }
-
-
-def registry_for_prompt(limit: int = 0) -> str:
-    """Compact registry listing for the model to select from."""
-    lines = []
-    for r in REGISTRY.values():
-        lines.append(
-            f'- id="{r["id"]}" | {r["name"]} | {r["type"]} | '
-            f'disciplines: {", ".join(r["disciplines"])} | {r["scope"]} | '
-            f'capabilities: {json.dumps(repository_capabilities(r), ensure_ascii=False)} | '
-            f'accepted content: {r.get("data_types_accepted", "Unknown")} | cost: {r.get("cost", "Unknown")}' 
-        )
-        if limit and len(lines) >= limit:
-            break
-    return "\n".join(lines)
-
-
-def hydrate_selection(selected: list[dict], constraints: dict | None = None) -> list[dict]:
-    """
-    Hydrate exact registry ids and discard every model-authored factual field.
-
-    Only `fit_reason` and `fit_confidence` survive from the model. Unknown ids,
-    duplicate ids, non-object entries, and malformed confidence values are
-    dropped or normalised rather than displayed as repository facts.
-    """
-    out: list[dict] = []
-    seen: set[str] = set()
-    if not isinstance(selected, list):
-        return out
-
-    for item in selected:
-        if not isinstance(item, dict):
-            continue
-        rid = str(item.get("id", "")).strip().lower()
-        entry = REGISTRY.get(rid)
-        if not entry or rid in seen:
-            continue
-        seen.add(rid)
-        assessment = evaluate_repository(entry, constraints)
-        if constraints is not None and assessment["state"] == "does_not_meet":
-            continue
-
-        fit_reason = _clean_text(item.get("fit_reason"), 1200)
-        confidence = _clean_text(item.get("fit_confidence"), 20).title()
-        if confidence not in {"High", "Medium", "Low"}:
-            confidence = ""
-
-        merged = dict(entry)
-        merged["fit_reason"] = fit_reason
-        merged["fit_confidence"] = confidence
-        # Backwards-compatible alias for older frontends; it is still an
-        # analytical fit score, never a repository-fact confidence score.
-        merged["confidence"] = confidence
-        merged["verification_status"] = "Curated KU Library registry record"
-        merged["verified_note"] = (
-            f"Repository details verified {entry['verified']} from {entry['source_url']}"
-        )
-        merged["capabilities"] = assessment["capabilities"]
-        merged["constraint_assessment"] = assessment
-        merged["file_deposit_recommended"] = assessment["file_deposit_recommended"]
-        merged["metadata_registration_only"] = bool(entry.get("is_khazna"))
-        if constraints is not None and assessment["state"] == "conditional":
-            merged["fit_reason"] = ("Candidate for review. " + fit_reason + " " + " ".join(assessment["unresolved_constraints"]))[:3000]
-            merged["fit_confidence"] = "Low" if assessment["sensitive_data_review_required"] else "Medium"
-            merged["confidence"] = merged["fit_confidence"]
-        out.append(merged)
-    return out
-
-
-def selection_exclusions(selected: list[dict], constraints: dict | None = None) -> list[dict]:
-    """Explain deterministic exclusions without trusting model-authored facts."""
-    excluded, seen = [], set()
-    for item in selected if isinstance(selected, list) else []:
-        if not isinstance(item, dict):
-            continue
-        rid = _clean_text(item.get("id"), 100).lower()
-        entry = REGISTRY.get(rid)
-        if not entry or rid in seen:
-            continue
-        seen.add(rid)
-        result = evaluate_repository(entry, constraints)
-        if result["state"] == "does_not_meet":
-            excluded.append({"id": rid, "name": entry["name"], "reasons": result["constraint_conflicts"],
-                             "unresolved_constraints": result["unresolved_constraints"]})
-    return excluded
-
-
-def hydrate_deposit_strategy(raw: Any, hydrated: list[dict], constraints: dict | None = None) -> dict:
-    """Resolve model-selected ids to server-owned names and bound free text."""
-    raw = raw if isinstance(raw, dict) else {}
-    by_id = {str(r.get("id", "")): r for r in hydrated if r.get("id")}
-    eligible = {rid for rid, entry in by_id.items()
-                if not entry.get("is_khazna") and entry.get("constraint_assessment", {}).get("strategy_eligible", True)}
-    ranked_ids = [
-        str(r.get("id")) for r in hydrated
-        if r.get("id") in eligible
-    ]
-
-    def selected_id(key: str, fallback_index: int) -> str:
-        candidate = str(raw.get(key, "")).strip().lower()
-        if candidate in eligible:
-            return candidate
-        return ranked_ids[fallback_index] if len(ranked_ids) > fallback_index else ""
-
-    primary_id = selected_id("primary_repository_id", 0)
-    secondary_id = selected_id("secondary_repository_id", 1)
-    if secondary_id == primary_id:
-        secondary_id = ranked_ids[1] if len(ranked_ids) > 1 else ""
-
-    checks = _clean_list(raw.get("manual_checks_required"), limit=10)
-    for entry in hydrated:
-        checks.extend(entry.get("constraint_assessment", {}).get("unresolved_constraints", []))
-    if not primary_id:
-        checks.insert(0, "No file-deposit destination has been established for these requirements; contact KU Library before uploading data.")
-    workflow = _clean_list(raw.get("deposit_workflow"), limit=10)
-    if constraints is not None:
-        # Model prose must not instruct immediate upload while deterministic
-        # requirements remain unresolved, especially for sensitive datasets.
-        workflow = (["Review the candidate repository's source guidance and all unresolved requirements.",
-                     "Confirm rights, consent, access conditions and depositor eligibility.",
-                     "Prepare metadata and supporting documentation; upload files only after all checks are complete."]
-                    if primary_id else ["Contact KU Library to identify an approved deposit or metadata-registration route.",
-                                        "Do not upload dataset files until a suitable destination and access conditions are confirmed."])
-    return {
-        "primary_repository_id": primary_id,
-        "primary_url": REGISTRY.get(primary_id, {}).get("url"),
-        "secondary_url": REGISTRY.get(secondary_id, {}).get("url"),
-        "secondary_repository_id": secondary_id,
-        "primary_choice": by_id.get(primary_id, {}).get("name", "Not determined"),
-        "secondary_choice": by_id.get(secondary_id, {}).get("name", "Not determined"),
-        "metadata_standard": _clean_text(
-            raw.get("metadata_standard") or "Check the selected repository's current metadata guidance.",
-            500,
-        ),
-        "deposit_workflow": workflow,
-        "manual_checks_required": list(dict.fromkeys(checks)),
-        "selection_status": "candidates_require_review" if checks and primary_id else "no_verified_destination" if not primary_id else "requirements_matched",
-        "khazna_metadata_note": (
-            "Ask KU Library about registering dataset metadata in Khazna when the files are held in an appropriate external repository."
-        ),
-    }
-
-
-def khazna_entry(constraints: dict | None = None) -> dict:
-    e = dict(REGISTRY["khazna"])
-    e["fit_reason"] = ("Contact KU Library about registering dataset metadata in Khazna for institutional visibility. "
-                       "File-deposit eligibility, DOI assignment and access controls must be confirmed separately.")
-    e["metadata_registration_only"] = True
-    e["file_deposit_recommended"] = False
-    e["constraint_assessment"] = evaluate_repository(e, constraints)
-    e["capabilities"] = e["constraint_assessment"]["capabilities"]
-    e["fit_confidence"] = "High"
-    e["confidence"] = "High"
-    e["verification_status"] = "Curated KU Library registry record"
-    e["verified_note"] = (f"Repository details verified {e['verified']} "
-                          f"from {e['source_url']}")
-    return e
-
-
-# ══════════════════════════════════════════════════════════════════════════
-# SUBJECT REPOSITORIES — for ARTICLES, not datasets
-# ══════════════════════════════════════════════════════════════════════════
-# OA.Works returns permitted locations such as "Non-commercial Subject
-# Repository". That string alone is not actionable: the author still has to
-# know WHICH one. These entries are matched against the journal's SCImago
-# subject areas/categories so the answer is specific.
-#
-# Deliberately distinct from REGISTRY above, which holds DATA repositories.
-# Depositing a manuscript in Dryad or a dataset in arXiv are both wrong.
-
-SUBJECT_REPOSITORIES: dict[str, dict] = {
-    "arxiv": {
-        "id": "arxiv", "name": "arXiv", "url": "https://arxiv.org",
-        "scope": "Physics, mathematics, computer science, quantitative biology, "
-                 "statistics, electrical engineering and economics.",
-        "versions": "Preprint and accepted manuscript",
-        "moderation": "Moderated; endorsement required for first submission",
-        "source_url": "https://arxiv.org/help/submit", "verified": "2026-07",
-        "match": ["physics", "astronomy", "mathematics", "computer science",
-                  "engineering", "electrical", "statistics", "economics",
-                  "materials science", "energy"],
-    },
-    "pmc": {
-        "id": "pmc", "name": "Europe PMC / PubMed Central",
-        "url": "https://europepmc.org",
-        "scope": "Biomedical and life sciences literature. Required by many "
-                 "health funders.",
-        "versions": "Accepted manuscript and published version where licensed",
-        "moderation": "Mediated; funder-linked deposit routes available",
-        "source_url": "https://europepmc.org/Help", "verified": "2026-07",
-        "match": ["medicine", "biochemistry", "immunology", "microbiology",
-                  "neuroscience", "pharmacology", "health", "nursing",
-                  "veterinary", "dentistry", "biology"],
-    },
-    "biorxiv": {
-        "id": "biorxiv", "name": "bioRxiv", "url": "https://www.biorxiv.org",
-        "scope": "Preprints across the biological sciences.",
-        "versions": "Preprint (submitted version) only",
-        "moderation": "Screened, not peer reviewed",
-        "source_url": "https://www.biorxiv.org/about/FAQ", "verified": "2026-07",
-        "match": ["biology", "biochemistry", "genetics", "neuroscience",
-                  "immunology", "microbiology", "agricultural"],
-    },
-    "chemrxiv": {
-        "id": "chemrxiv", "name": "ChemRxiv", "url": "https://chemrxiv.org",
-        "scope": "Preprints in chemistry and related areas.",
-        "versions": "Preprint (submitted version) only",
-        "moderation": "Screened, not peer reviewed",
-        "source_url": "https://chemrxiv.org/engage/chemrxiv/public-dashboard",
-        "verified": "2026-07",
-        "match": ["chemistry", "chemical engineering", "materials science"],
-    },
-    "engrxiv": {
-        "id": "engrxiv", "name": "engrXiv", "url": "https://engrxiv.org",
-        "scope": "Preprints across all engineering disciplines.",
-        "versions": "Preprint and accepted manuscript",
-        "moderation": "Screened, not peer reviewed",
-        "source_url": "https://engrxiv.org", "verified": "2026-07",
-        "match": ["engineering", "civil", "mechanical", "aerospace",
-                  "industrial", "environmental engineering"],
-    },
-    "ssrn": {
-        "id": "ssrn", "name": "SSRN", "url": "https://www.ssrn.com",
-        "scope": "Social sciences, economics, management, law and humanities.",
-        "versions": "Preprint and accepted manuscript",
-        "moderation": "Screened",
-        "source_url": "https://www.ssrn.com/index.cfm/en/", "verified": "2026-07",
-        "match": ["social sciences", "economics", "business", "management",
-                  "accounting", "finance", "law", "psychology", "decision sciences"],
-    },
-    "repec": {
-        "id": "repec", "name": "RePEc", "url": "http://repec.org",
-        "scope": "Working papers and articles in economics and finance.",
-        "versions": "Working paper and accepted manuscript",
-        "moderation": "Institution-mediated archives",
-        "source_url": "http://repec.org/", "verified": "2026-07",
-        "match": ["economics", "econometrics", "finance"],
-    },
-    "eartharxiv": {
-        "id": "eartharxiv", "name": "EarthArXiv",
-        "url": "https://eartharxiv.org",
-        "scope": "Preprints in earth, planetary and environmental sciences.",
-        "versions": "Preprint and accepted manuscript",
-        "moderation": "Screened, not peer reviewed",
-        "source_url": "https://eartharxiv.org", "verified": "2026-07",
-        "match": ["earth", "planetary", "environmental science", "geology",
-                  "atmospheric", "oceanography"],
-    },
-    "medrxiv": {
-        "id": "medrxiv", "name": "medRxiv", "url": "https://www.medrxiv.org",
-        "scope": "Preprints in clinical and health sciences.",
-        "versions": "Preprint (submitted version) only",
-        "moderation": "Screened; clinical claims reviewed",
-        "source_url": "https://www.medrxiv.org/about/FAQ", "verified": "2026-07",
-        "match": ["medicine", "health", "clinical", "epidemiology", "public health"],
-    },
-}
-
-# Always offered when a subject repository is permitted but nothing matched.
-SUBJECT_FALLBACK = ["arxiv", "ssrn"]
-
-
-def suggest_subject_repositories(areas: str = "", categories: str = "",
-                                 limit: int = 3) -> list[dict]:
-    """
-    Match a journal's SCImago subject areas/categories to subject repositories.
-
-    Deterministic keyword scoring — no LLM. Returns [] rather than guessing
-    when nothing matches well and no fallback is warranted.
-    """
-    haystack = f"{areas} {categories}".lower()
-    if not haystack.strip():
-        return []
-
-    scored: list[tuple[int, dict]] = []
-    for entry in SUBJECT_REPOSITORIES.values():
-        score = sum(1 for kw in entry["match"] if kw in haystack)
-        if score:
-            scored.append((score, entry))
-
-    scored.sort(key=lambda x: (-x[0], x[1]["name"]))
-    picked = [dict(e) for _, e in scored[:limit]]
-
-    # An unrelated fallback is not a subject match. Return no suggestions if
-    # the supplied discipline does not match a curated repository.
-    if not picked:
-        return []
-
-    for p in picked:
-        p.pop("match", None)
-        p["verified_note"] = f"Verified {p['verified']} from {p['source_url']}"
-    return picked
+"""
+repositories.py — curated registry of research data repositories.
+
+WHY THIS EXISTS
+---------------
+The previous /api/find-repository asked the LLM to produce repository facts:
+max file size, certification, cost, persistent identifier type. Those are
+exactly the kind of specific, checkable claims a model gets subtly wrong, and
+a wrong "50 GB limit" or "CoreTrustSeal certified" in a library tool is a
+credibility problem.
+
+Here the LLM only SELECTS entries by id and writes a fit_reason. Every factual
+field below is served from this file and overwrites whatever the model says.
+
+MAINTENANCE
+-----------
+Each entry carries `source_url` and `verified` (YYYY-MM). Re-check quarterly.
+Fields you are unsure of should be left as "" rather than guessed — the UI
+renders empty fields as "Not specified", which is honest.
+
+Selection is deliberately KU-flavoured: engineering, energy, materials,
+petroleum, AI/CS and biomedical are weighted because that is KU's output
+profile.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+import json
+import re
+
+# ── Registry ───────────────────────────────────────────────────────────────
+# NOTE: max_file_size / certification are left blank where not personally
+# verified. Fill them in from source_url and bump `verified`.
+
+REGISTRY: dict[str, dict[str, Any]] = {
+    # ---- Institutional ----------------------------------------------------
+    "khazna": {
+        "id": "khazna",
+        "name": "Khazna",
+        "type": "Institutional",
+        "url": "https://khazna.ku.ac.ae",
+        "scope": "Khalifa University research outputs, datasets and theses across all disciplines.",
+        "data_types_accepted": "Confirm file-deposit support with KU Library",
+        "persistent_identifier": "",
+        "versioning": "",
+        "embargo_support": "",
+        "cost": "Free for KU researchers",
+        "curation": "Mediated — library review before publication",
+        "source_url": "https://khazna.ku.ac.ae",
+        "verified": "2026-07",
+        "is_khazna": True,
+        "disciplines": ["all"],
+    },
+
+    # ---- Generalist -------------------------------------------------------
+    "zenodo": {
+        "id": "zenodo", "name": "Zenodo", "type": "Generalist",
+        "url": "https://zenodo.org",
+        "scope": "Multidisciplinary repository operated by CERN; accepts data, software, publications.",
+        "data_types_accepted": "All types", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free",
+        "curation": "Unmediated",
+        "source_url": "https://help.zenodo.org", "verified": "2026-07",
+        "disciplines": ["all"],
+    },
+    "figshare": {
+        "id": "figshare", "name": "Figshare", "type": "Generalist",
+        "url": "https://figshare.com",
+        "scope": "Multidisciplinary repository for figures, datasets, media and preprints.",
+        "data_types_accepted": "All types", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free tier available",
+        "curation": "Unmediated",
+        "source_url": "https://help.figshare.com", "verified": "2026-07",
+        "disciplines": ["all"],
+    },
+    "dryad": {
+        "id": "dryad", "name": "Dryad", "type": "Generalist",
+        "url": "https://datadryad.org",
+        "scope": "Curated repository for data underlying published research, strong in life and environmental sciences.",
+        "data_types_accepted": "Research data (not software)", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Fee-based; free where the institution is a member",
+        "curation": "Mediated — curator review",
+        "source_url": "https://datadryad.org/stash/faq", "verified": "2026-07",
+        "disciplines": ["life sciences", "environmental", "ecology", "biology"],
+    },
+    "dataverse": {
+        "id": "dataverse", "name": "Harvard Dataverse", "type": "Generalist",
+        "url": "https://dataverse.harvard.edu",
+        "scope": "Multidisciplinary repository with strong social science and statistical tooling.",
+        "data_types_accepted": "All types", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free",
+        "curation": "Unmediated",
+        "source_url": "https://support.dataverse.harvard.edu", "verified": "2026-07",
+        "disciplines": ["social sciences", "economics", "political science", "all"],
+    },
+    "osf": {
+        "id": "osf", "name": "Open Science Framework (OSF)", "type": "Generalist",
+        "url": "https://osf.io",
+        "scope": "Project-based platform combining data, preprints, preregistration and materials.",
+        "data_types_accepted": "All types", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free",
+        "curation": "Unmediated",
+        "source_url": "https://help.osf.io", "verified": "2026-07",
+        "disciplines": ["psychology", "social sciences", "all"],
+    },
+    "mendeley_data": {
+        "id": "mendeley_data", "name": "Mendeley Data", "type": "Generalist",
+        "url": "https://data.mendeley.com",
+        "scope": "Multidisciplinary data repository operated by Elsevier.",
+        "data_types_accepted": "All types", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes", "cost": "Free tier available",
+        "curation": "Unmediated",
+        "source_url": "https://data.mendeley.com", "verified": "2026-07",
+        "disciplines": ["all"],
+    },
+
+    # ---- Engineering / energy / materials — KU core ------------------------
+    "materials_cloud": {
+        "id": "materials_cloud", "name": "Materials Cloud", "type": "Domain-specific",
+        "url": "https://www.materialscloud.org",
+        "scope": "Computational materials science data, workflows and simulations.",
+        "data_types_accepted": "Simulation data, structures, workflows",
+        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://www.materialscloud.org/home", "verified": "2026-07",
+        "disciplines": ["materials", "chemistry", "physics", "computational"],
+    },
+    "nomad": {
+        "id": "nomad", "name": "NOMAD", "type": "Domain-specific",
+        "url": "https://nomad-lab.eu",
+        "scope": "Repository for computational materials science, especially DFT calculations.",
+        "data_types_accepted": "Electronic-structure calculation outputs",
+        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Unmediated",
+        "source_url": "https://nomad-lab.eu", "verified": "2026-07",
+        "disciplines": ["materials", "physics", "chemistry", "computational"],
+    },
+    "oedi": {
+        "id": "oedi", "name": "Open Energy Data Initiative (OEDI)", "type": "Domain-specific",
+        "url": "https://data.openei.org",
+        "scope": "Energy systems, renewables, grid and buildings data (US DOE).",
+        "data_types_accepted": "Energy datasets, time series, geospatial",
+        "persistent_identifier": "DOI", "versioning": "", "embargo_support": "",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://data.openei.org", "verified": "2026-07",
+        "disciplines": ["energy", "renewables", "engineering", "sustainability"],
+    },
+    "ieee_dataport": {
+        "id": "ieee_dataport", "name": "IEEE DataPort", "type": "Domain-specific",
+        "url": "https://ieee-dataport.org",
+        "scope": "Datasets across electrical engineering, computing, communications and signal processing.",
+        "data_types_accepted": "All types", "persistent_identifier": "DOI",
+        "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free for open datasets; subscription for some features",
+        "curation": "Unmediated",
+        "source_url": "https://ieee-dataport.org", "verified": "2026-07",
+        "disciplines": ["engineering", "electrical", "computing", "communications", "signal processing"],
+    },
+
+    # ---- Earth / environment / geospatial ---------------------------------
+    "pangaea": {
+        "id": "pangaea", "name": "PANGAEA", "type": "Domain-specific",
+        "url": "https://www.pangaea.de",
+        "scope": "Earth and environmental science data; georeferenced observations.",
+        "data_types_accepted": "Georeferenced environmental data",
+        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated — editorial review",
+        "source_url": "https://www.pangaea.de/submit/", "verified": "2026-07",
+        "disciplines": ["earth science", "environmental", "climate", "oceanography", "geoscience"],
+    },
+    "esa_earth": {
+        "id": "esa_earth", "name": "ESA Earth Online", "type": "Domain-specific",
+        "url": "https://earth.esa.int",
+        "scope": "Earth observation and satellite remote-sensing datasets.",
+        "data_types_accepted": "Satellite imagery, EO products",
+        "persistent_identifier": "", "versioning": "", "embargo_support": "",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://earth.esa.int", "verified": "2026-07",
+        "disciplines": ["remote sensing", "earth observation", "geospatial", "climate"],
+    },
+
+    # ---- Life sciences / biomedical ---------------------------------------
+    "ena": {
+        "id": "ena", "name": "European Nucleotide Archive (ENA)", "type": "Domain-specific",
+        "url": "https://www.ebi.ac.uk/ena",
+        "scope": "Nucleotide sequencing data — raw reads, assemblies, annotations.",
+        "data_types_accepted": "Sequence data (FASTQ, BAM, assemblies)",
+        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated — format validation",
+        "source_url": "https://www.ebi.ac.uk/ena/browser/submit", "verified": "2026-07",
+        "disciplines": ["genomics", "biology", "life sciences", "sequencing", "microbiology"],
+    },
+    "geo": {
+        "id": "geo", "name": "NCBI GEO", "type": "Domain-specific",
+        "url": "https://www.ncbi.nlm.nih.gov/geo",
+        "scope": "Functional genomics data — gene expression, microarray, high-throughput sequencing.",
+        "data_types_accepted": "Expression and functional genomics data",
+        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://www.ncbi.nlm.nih.gov/geo/info/submission.html", "verified": "2026-07",
+        "disciplines": ["genomics", "transcriptomics", "biology", "life sciences"],
+    },
+    "pride": {
+        "id": "pride", "name": "PRIDE", "type": "Domain-specific",
+        "url": "https://www.ebi.ac.uk/pride",
+        "scope": "Mass-spectrometry proteomics data.",
+        "data_types_accepted": "Proteomics MS data",
+        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://www.ebi.ac.uk/pride/markdownpage/submitdatapage", "verified": "2026-07",
+        "disciplines": ["proteomics", "biochemistry", "life sciences"],
+    },
+    "pdb": {
+        "id": "pdb", "name": "Protein Data Bank (PDB)", "type": "Domain-specific",
+        "url": "https://www.rcsb.org",
+        "scope": "Three-dimensional structural data for biological macromolecules.",
+        "data_types_accepted": "Macromolecular structures",
+        "persistent_identifier": "PDB ID", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated — biocuration",
+        "source_url": "https://www.rcsb.org/pages/policy", "verified": "2026-07",
+        "disciplines": ["structural biology", "biochemistry", "chemistry", "life sciences"],
+    },
+    "metabolights": {
+        "id": "metabolights", "name": "MetaboLights", "type": "Domain-specific",
+        "url": "https://www.ebi.ac.uk/metabolights",
+        "scope": "Metabolomics experiments and derived information.",
+        "data_types_accepted": "Metabolomics data",
+        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://www.ebi.ac.uk/metabolights", "verified": "2026-07",
+        "disciplines": ["metabolomics", "biochemistry", "life sciences"],
+    },
+
+    # ---- Chemistry --------------------------------------------------------
+    "chemrxiv_data": {
+        "id": "chemrxiv_data", "name": "Cambridge Structural Database (CCDC)",
+        "type": "Domain-specific", "url": "https://www.ccdc.cam.ac.uk",
+        "scope": "Small-molecule organic and metal-organic crystal structures.",
+        "data_types_accepted": "Crystallographic data (CIF)",
+        "persistent_identifier": "CCDC number", "versioning": "", "embargo_support": "Yes",
+        "cost": "Free deposition", "curation": "Mediated",
+        "source_url": "https://www.ccdc.cam.ac.uk/structures/", "verified": "2026-07",
+        "disciplines": ["chemistry", "crystallography", "materials"],
+    },
+
+    # ---- Computing / AI ---------------------------------------------------
+    "huggingface_datasets": {
+        "id": "huggingface_datasets", "name": "Hugging Face Datasets",
+        "type": "Domain-specific", "url": "https://huggingface.co/datasets",
+        "scope": "Machine learning datasets and models with versioned git-based storage.",
+        "data_types_accepted": "ML datasets, models",
+        "persistent_identifier": "", "versioning": "Yes", "embargo_support": "",
+        "cost": "Free tier available", "curation": "Unmediated",
+        "source_url": "https://huggingface.co/docs/hub/datasets", "verified": "2026-07",
+        "disciplines": ["machine learning", "ai", "nlp", "computer vision", "computing"],
+    },
+    "openml": {
+        "id": "openml", "name": "OpenML", "type": "Domain-specific",
+        "url": "https://www.openml.org",
+        "scope": "Machine learning datasets, tasks, flows and benchmark runs.",
+        "data_types_accepted": "Tabular ML datasets",
+        "persistent_identifier": "", "versioning": "Yes", "embargo_support": "",
+        "cost": "Free", "curation": "Unmediated",
+        "source_url": "https://docs.openml.org", "verified": "2026-07",
+        "disciplines": ["machine learning", "ai", "data science", "computing"],
+    },
+
+    # ---- Social science / humanities --------------------------------------
+    "ukds": {
+        "id": "ukds", "name": "UK Data Service ReShare", "type": "Domain-specific",
+        "url": "https://reshare.ukdataservice.ac.uk",
+        "scope": "Social and economic research data, including sensitive data under controlled access.",
+        "data_types_accepted": "Social science data",
+        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated",
+        "source_url": "https://reshare.ukdataservice.ac.uk", "verified": "2026-07",
+        "disciplines": ["social sciences", "economics", "sociology", "survey"],
+    },
+    "icpsr": {
+        "id": "icpsr", "name": "ICPSR", "type": "Domain-specific",
+        "url": "https://www.icpsr.umich.edu",
+        "scope": "Social and behavioural science data with strong curation and restricted-access options.",
+        "data_types_accepted": "Social science data",
+        "persistent_identifier": "DOI", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Varies by membership", "curation": "Mediated — extensive curation",
+        "source_url": "https://www.icpsr.umich.edu/web/pages/deposit/", "verified": "2026-07",
+        "disciplines": ["social sciences", "behavioural", "health", "survey"],
+    },
+
+    # ---- Sensitive / controlled access -------------------------------------
+    "ega": {
+        "id": "ega", "name": "European Genome-phenome Archive (EGA)",
+        "type": "Domain-specific", "url": "https://ega-archive.org",
+        "scope": "Controlled-access repository for personally identifiable genetic and phenotypic data.",
+        "data_types_accepted": "Human genomic and phenotypic data",
+        "persistent_identifier": "Accession", "versioning": "Yes", "embargo_support": "Yes",
+        "cost": "Free", "curation": "Mediated — DAC-controlled access",
+        "source_url": "https://ega-archive.org/submission", "verified": "2026-07",
+        "disciplines": ["human genomics", "clinical", "biomedical", "sensitive"],
+    },
+}
+
+
+# ── Helpers ────────────────────────────────────────────────────────────────
+# The model may author ONLY analytical fit fields. Every other repository
+# attribute is supplied from REGISTRY, including the id itself after lookup.
+MODEL_ALLOWED = ("fit_reason", "fit_confidence")
+
+
+def _clean_text(value: Any, limit: int = 1000) -> str:
+    return " ".join(str(value or "").split())[:limit]
+
+
+def _clean_list(value: Any, limit: int = 12, item_limit: int = 500) -> list[str]:
+    if not isinstance(value, list):
+        return []
+    out: list[str] = []
+    for item in value[:limit]:
+        cleaned = _clean_text(item, item_limit)
+        if cleaned:
+            out.append(cleaned)
+    return out
+
+
+def _yes_no(value: Any) -> str:
+    text = _clean_text(value, 100).casefold()
+    if text in {"yes", "true"} or text.startswith("yes ") or text.startswith("yes—"):
+        return "Yes"
+    if text in {"no", "false"}:
+        return "No"
+    return "Unknown"
+
+
+def repository_capabilities(entry: dict) -> dict:
+    """Expose only capabilities supported by existing curated fields.
+
+    A generic homepage and a broad access description do not establish approval
+    for a particular sensitive dataset, size limit or accepted file format.
+    """
+    identifier = _clean_text(entry.get("persistent_identifier"), 100)
+    description = (str(entry.get("scope", "")) + " " + str(entry.get("curation", ""))).casefold()
+    controlled = any(term in description for term in ("controlled access", "controlled-access", "restricted-access", "restricted access"))
+    return {
+        "doi": "Yes" if identifier.casefold() == "doi" else "No" if identifier else "Unknown",
+        "persistent_id_type": identifier or "Unknown",
+        "versioning": _yes_no(entry.get("versioning")),
+        "embargo": _yes_no(entry.get("embargo_support")),
+        "restricted_access": "Yes" if controlled and not entry.get("is_khazna") else "Unknown",
+        "approved_sensitive_data": "Unknown",
+        "accepts_external_deposits": "Unknown",
+        "accepted_file_formats": "Unknown",
+        "maximum_dataset_size": "Unknown",
+        "source_url": entry.get("source_url", ""),
+        "reviewed": entry.get("verified", ""),
+    }
+
+
+def evaluate_repository(entry: dict, constraints: dict | None = None) -> dict:
+    """Check stated requirements against curated facts; unknown is not Yes."""
+    constraints = constraints if isinstance(constraints, dict) else {}
+    capabilities = repository_capabilities(entry)
+    unresolved, conflicts = [], []
+    sensitivity = _clean_text(constraints.get("sensitivity"), 100).casefold()
+    personal = _clean_text(constraints.get("contains_personal_data"), 100).casefold()
+    sensitive = (bool(sensitivity) and sensitivity not in {"open", "none", "non-sensitive", "not sensitive"}) or (bool(personal) and personal != "no")
+    for requirement, capability, label in (
+        ("needs_doi", "doi", "DOI assignment"),
+        ("needs_versioning", "versioning", "versioning"),
+        ("embargo_required", "embargo", "an embargo"),
+    ):
+        if _yes_no(constraints.get(requirement)) != "Yes":
+            continue
+        if capabilities[capability] == "No":
+            conflicts.append(f"The curated record does not support the requested {label}.")
+        elif capabilities[capability] != "Yes":
+            unresolved.append(f"Confirm support for {label} with the repository.")
+    if sensitive:
+        if capabilities["restricted_access"] != "Yes":
+            conflicts.append("Controlled/restricted access suitable for the submitted sensitive or personal data is not established.")
+        unresolved.append("KU Library and the responsible ethics/data-protection team must approve the repository and access conditions for this dataset before any file upload.")
+    if _clean_text(constraints.get("file_formats")):
+        unresolved.append("Confirm acceptance of the submitted file formats: " + _clean_text(constraints["file_formats"], 300))
+    if _clean_text(constraints.get("approx_size")):
+        unresolved.append("Confirm the size/quota and any associated cost for: " + _clean_text(constraints["approx_size"], 300))
+    if _clean_text(constraints.get("funder")):
+        unresolved.append("Confirm the funder's repository, licence and retention requirements; the registry does not establish mandate compliance.")
+    if _clean_text(constraints.get("licence_intent")):
+        unresolved.append("Confirm that the repository supports the selected licence and that you hold the necessary rights.")
+    institution = _clean_text(constraints.get("institution"))
+    country = _clean_text(constraints.get("country"))
+    if institution or country:
+        unresolved.append("Confirm depositor eligibility and any institution/country restrictions.")
+    if _yes_no(constraints.get("embargo_required")) == "Yes":
+        duration = constraints.get("embargo_months")
+        if duration is not None:
+            unresolved.append(f"Confirm support for the requested {duration}-month embargo.")
+        elif constraints.get("embargo_duration"):
+            unresolved.append("Confirm support for the requested embargo duration: " + _clean_text(constraints["embargo_duration"], 100))
+    if entry.get("id") == "dryad" and "software" in _clean_text(constraints.get("data_types")).casefold():
+        conflicts.append("The curated record lists research data, not software, as accepted content.")
+    metadata_only = bool(entry.get("is_khazna"))
+    if metadata_only:
+        unresolved.insert(0, "Ask KU Library about metadata registration; dataset file acceptance, DOI assignment and access controls require confirmation.")
+        # Institutional registration advice is not a file-deposit destination.
+        conflicts = []
+    return {
+        "state": "metadata_only" if metadata_only else "does_not_meet" if conflicts else "conditional" if unresolved else "meets",
+        "unresolved_constraints": list(dict.fromkeys(unresolved)),
+        "constraint_conflicts": list(dict.fromkeys(conflicts)),
+        "sensitive_data_review_required": sensitive,
+        "file_deposit_recommended": not metadata_only and not conflicts and not unresolved,
+        "strategy_eligible": not metadata_only and not conflicts and not sensitive,
+        "capabilities": capabilities,
+    }
+
+
+def registry_for_prompt(limit: int = 0) -> str:
+    """Compact registry listing for the model to select from."""
+    lines = []
+    for r in REGISTRY.values():
+        lines.append(
+            f'- id="{r["id"]}" | {r["name"]} | {r["type"]} | '
+            f'disciplines: {", ".join(r["disciplines"])} | {r["scope"]} | '
+            f'capabilities: {json.dumps(repository_capabilities(r), ensure_ascii=False)} | '
+            f'accepted content: {r.get("data_types_accepted", "Unknown")} | cost: {r.get("cost", "Unknown")}' 
+        )
+        if limit and len(lines) >= limit:
+            break
+    return "\n".join(lines)
+
+
+def hydrate_selection(selected: list[dict], constraints: dict | None = None) -> list[dict]:
+    """
+    Hydrate exact registry ids and discard every model-authored factual field.
+
+    Only `fit_reason` and `fit_confidence` survive from the model. Unknown ids,
+    duplicate ids, non-object entries, and malformed confidence values are
+    dropped or normalised rather than displayed as repository facts.
+    """
+    out: list[dict] = []
+    seen: set[str] = set()
+    if not isinstance(selected, list):
+        return out
+
+    for item in selected:
+        if not isinstance(item, dict):
+            continue
+        rid = str(item.get("id", "")).strip().lower()
+        entry = REGISTRY.get(rid)
+        if not entry or rid in seen:
+            continue
+        seen.add(rid)
+        assessment = evaluate_repository(entry, constraints)
+        if constraints is not None and assessment["state"] == "does_not_meet":
+            continue
+
+        fit_reason = _clean_text(item.get("fit_reason"), 1200)
+        confidence = _clean_text(item.get("fit_confidence"), 20).title()
+        if confidence not in {"High", "Medium", "Low"}:
+            confidence = ""
+
+        merged = dict(entry)
+        merged["fit_reason"] = fit_reason
+        merged["fit_confidence"] = confidence
+        # Backwards-compatible alias for older frontends; it is still an
+        # analytical fit score, never a repository-fact confidence score.
+        merged["confidence"] = confidence
+        merged["verification_status"] = "Curated KU Library registry record"
+        merged["verified_note"] = (
+            f"Repository details verified {entry['verified']} from {entry['source_url']}"
+        )
+        merged["capabilities"] = assessment["capabilities"]
+        merged["constraint_assessment"] = assessment
+        merged["file_deposit_recommended"] = assessment["file_deposit_recommended"]
+        merged["metadata_registration_only"] = bool(entry.get("is_khazna"))
+        if constraints is not None and assessment["state"] == "conditional":
+            merged["fit_reason"] = ("Candidate for review. " + fit_reason + " " + " ".join(assessment["unresolved_constraints"]))[:3000]
+            merged["fit_confidence"] = "Low" if assessment["sensitive_data_review_required"] else "Medium"
+            merged["confidence"] = merged["fit_confidence"]
+        out.append(merged)
+    return out
+
+
+def selection_exclusions(selected: list[dict], constraints: dict | None = None) -> list[dict]:
+    """Explain deterministic exclusions without trusting model-authored facts."""
+    excluded, seen = [], set()
+    for item in selected if isinstance(selected, list) else []:
+        if not isinstance(item, dict):
+            continue
+        rid = _clean_text(item.get("id"), 100).lower()
+        entry = REGISTRY.get(rid)
+        if not entry or rid in seen:
+            continue
+        seen.add(rid)
+        result = evaluate_repository(entry, constraints)
+        if result["state"] == "does_not_meet":
+            excluded.append({"id": rid, "name": entry["name"], "reasons": result["constraint_conflicts"],
+                             "unresolved_constraints": result["unresolved_constraints"]})
+    return excluded
+
+
+def hydrate_deposit_strategy(raw: Any, hydrated: list[dict], constraints: dict | None = None) -> dict:
+    """Resolve model-selected ids to server-owned names and bound free text."""
+    raw = raw if isinstance(raw, dict) else {}
+    by_id = {str(r.get("id", "")): r for r in hydrated if r.get("id")}
+    eligible = {rid for rid, entry in by_id.items()
+                if not entry.get("is_khazna") and entry.get("constraint_assessment", {}).get("strategy_eligible", True)}
+    ranked_ids = [
+        str(r.get("id")) for r in hydrated
+        if r.get("id") in eligible
+    ]
+
+    def selected_id(key: str, fallback_index: int) -> str:
+        candidate = str(raw.get(key, "")).strip().lower()
+        if candidate in eligible:
+            return candidate
+        return ranked_ids[fallback_index] if len(ranked_ids) > fallback_index else ""
+
+    primary_id = selected_id("primary_repository_id", 0)
+    secondary_id = selected_id("secondary_repository_id", 1)
+    if secondary_id == primary_id:
+        secondary_id = ranked_ids[1] if len(ranked_ids) > 1 else ""
+
+    checks = _clean_list(raw.get("manual_checks_required"), limit=10)
+    for entry in hydrated:
+        checks.extend(entry.get("constraint_assessment", {}).get("unresolved_constraints", []))
+    if not primary_id:
+        checks.insert(0, "No file-deposit destination has been established for these requirements; contact KU Library before uploading data.")
+    workflow = _clean_list(raw.get("deposit_workflow"), limit=10)
+    if constraints is not None:
+        # Model prose must not instruct immediate upload while deterministic
+        # requirements remain unresolved, especially for sensitive datasets.
+        workflow = (["Review the candidate repository's source guidance and all unresolved requirements.",
+                     "Confirm rights, consent, access conditions and depositor eligibility.",
+                     "Prepare metadata and supporting documentation; upload files only after all checks are complete."]
+                    if primary_id else ["Contact KU Library to identify an approved deposit or metadata-registration route.",
+                                        "Do not upload dataset files until a suitable destination and access conditions are confirmed."])
+    return {
+        "primary_repository_id": primary_id,
+        "primary_url": REGISTRY.get(primary_id, {}).get("url"),
+        "secondary_url": REGISTRY.get(secondary_id, {}).get("url"),
+        "secondary_repository_id": secondary_id,
+        "primary_choice": by_id.get(primary_id, {}).get("name", "Not determined"),
+        "secondary_choice": by_id.get(secondary_id, {}).get("name", "Not determined"),
+        "metadata_standard": _clean_text(
+            raw.get("metadata_standard") or "Check the selected repository's current metadata guidance.",
+            500,
+        ),
+        "deposit_workflow": workflow,
+        "manual_checks_required": list(dict.fromkeys(checks)),
+        "selection_status": "candidates_require_review" if checks and primary_id else "no_verified_destination" if not primary_id else "requirements_matched",
+        "khazna_metadata_note": (
+            "Ask KU Library about registering dataset metadata in Khazna when the files are held in an appropriate external repository."
+        ),
+    }
+
+
+def khazna_entry(constraints: dict | None = None) -> dict:
+    e = dict(REGISTRY["khazna"])
+    e["fit_reason"] = ("Contact KU Library about registering dataset metadata in Khazna for institutional visibility. "
+                       "File-deposit eligibility, DOI assignment and access controls must be confirmed separately.")
+    e["metadata_registration_only"] = True
+    e["file_deposit_recommended"] = False
+    e["constraint_assessment"] = evaluate_repository(e, constraints)
+    e["capabilities"] = e["constraint_assessment"]["capabilities"]
+    e["fit_confidence"] = "High"
+    e["confidence"] = "High"
+    e["verification_status"] = "Curated KU Library registry record"
+    e["verified_note"] = (f"Repository details verified {e['verified']} "
+                          f"from {e['source_url']}")
+    return e
+
+
+# ══════════════════════════════════════════════════════════════════════════
+# SUBJECT REPOSITORIES — for ARTICLES, not datasets
+# ══════════════════════════════════════════════════════════════════════════
+# OA.Works returns permitted locations such as "Non-commercial Subject
+# Repository". That string alone is not actionable: the author still has to
+# know WHICH one. These entries are matched against the journal's SCImago
+# subject areas/categories so the answer is specific.
+#
+# Deliberately distinct from REGISTRY above, which holds DATA repositories.
+# Depositing a manuscript in Dryad or a dataset in arXiv are both wrong.
+
+SUBJECT_REPOSITORIES: dict[str, dict] = {
+    "arxiv": {
+        "id": "arxiv", "name": "arXiv", "url": "https://arxiv.org",
+        "scope": "Physics, mathematics, computer science, quantitative biology, "
+                 "statistics, electrical engineering and economics.",
+        "versions": "Preprint and accepted manuscript",
+        "moderation": "Moderated; endorsement required for first submission",
+        "source_url": "https://arxiv.org/help/submit", "verified": "2026-07",
+        "match": ["physics", "astronomy", "mathematics", "computer science",
+                  "engineering", "electrical", "statistics", "economics",
+                  "materials science", "energy"],
+    },
+    "pmc": {
+        "id": "pmc", "name": "Europe PMC / PubMed Central",
+        "url": "https://europepmc.org",
+        "scope": "Biomedical and life sciences literature. Required by many "
+                 "health funders.",
+        "versions": "Accepted manuscript and published version where licensed",
+        "moderation": "Mediated; funder-linked deposit routes available",
+        "source_url": "https://europepmc.org/Help", "verified": "2026-07",
+        "match": ["medicine", "biochemistry", "immunology", "microbiology",
+                  "neuroscience", "pharmacology", "health", "nursing",
+                  "veterinary", "dentistry", "biology"],
+    },
+    "biorxiv": {
+        "id": "biorxiv", "name": "bioRxiv", "url": "https://www.biorxiv.org",
+        "scope": "Preprints across the biological sciences.",
+        "versions": "Preprint (submitted version) only",
+        "moderation": "Screened, not peer reviewed",
+        "source_url": "https://www.biorxiv.org/about/FAQ", "verified": "2026-07",
+        "match": ["biology", "biochemistry", "genetics", "neuroscience",
+                  "immunology", "microbiology", "agricultural"],
+    },
+    "chemrxiv": {
+        "id": "chemrxiv", "name": "ChemRxiv", "url": "https://chemrxiv.org",
+        "scope": "Preprints in chemistry and related areas.",
+        "versions": "Preprint (submitted version) only",
+        "moderation": "Screened, not peer reviewed",
+        "source_url": "https://chemrxiv.org/engage/chemrxiv/public-dashboard",
+        "verified": "2026-07",
+        "match": ["chemistry", "chemical engineering", "materials science"],
+    },
+    "engrxiv": {
+        "id": "engrxiv", "name": "engrXiv", "url": "https://engrxiv.org",
+        "scope": "Preprints across all engineering disciplines.",
+        "versions": "Preprint and accepted manuscript",
+        "moderation": "Screened, not peer reviewed",
+        "source_url": "https://engrxiv.org", "verified": "2026-07",
+        "match": ["engineering", "civil", "mechanical", "aerospace",
+                  "industrial", "environmental engineering"],
+    },
+    "ssrn": {
+        "id": "ssrn", "name": "SSRN", "url": "https://www.ssrn.com",
+        "scope": "Social sciences, economics, management, law and humanities.",
+        "versions": "Preprint and accepted manuscript",
+        "moderation": "Screened",
+        "source_url": "https://www.ssrn.com/index.cfm/en/", "verified": "2026-07",
+        "match": ["social sciences", "economics", "business", "management",
+                  "accounting", "finance", "law", "psychology", "decision sciences"],
+    },
+    "repec": {
+        "id": "repec", "name": "RePEc", "url": "http://repec.org",
+        "scope": "Working papers and articles in economics and finance.",
+        "versions": "Working paper and accepted manuscript",
+        "moderation": "Institution-mediated archives",
+        "source_url": "http://repec.org/", "verified": "2026-07",
+        "match": ["economics", "econometrics", "finance"],
+    },
+    "eartharxiv": {
+        "id": "eartharxiv", "name": "EarthArXiv",
+        "url": "https://eartharxiv.org",
+        "scope": "Preprints in earth, planetary and environmental sciences.",
+        "versions": "Preprint and accepted manuscript",
+        "moderation": "Screened, not peer reviewed",
+        "source_url": "https://eartharxiv.org", "verified": "2026-07",
+        "match": ["earth", "planetary", "environmental science", "geology",
+                  "atmospheric", "oceanography"],
+    },
+    "medrxiv": {
+        "id": "medrxiv", "name": "medRxiv", "url": "https://www.medrxiv.org",
+        "scope": "Preprints in clinical and health sciences.",
+        "versions": "Preprint (submitted version) only",
+        "moderation": "Screened; clinical claims reviewed",
+        "source_url": "https://www.medrxiv.org/about/FAQ", "verified": "2026-07",
+        "match": ["medicine", "health", "clinical", "epidemiology", "public health"],
+    },
+}
+
+# Always offered when a subject repository is permitted but nothing matched.
+SUBJECT_FALLBACK = ["arxiv", "ssrn"]
+
+
+def suggest_subject_repositories(areas: str = "", categories: str = "",
+                                 limit: int = 3) -> list[dict]:
+    """
+    Match a journal's SCImago subject areas/categories to subject repositories.
+
+    Deterministic keyword scoring — no LLM. Returns [] rather than guessing
+    when nothing matches well and no fallback is warranted.
+    """
+    haystack = f"{areas} {categories}".lower()
+    if not haystack.strip():
+        return []
+
+    scored: list[tuple[int, dict]] = []
+    for entry in SUBJECT_REPOSITORIES.values():
+        score = sum(1 for kw in entry["match"] if kw in haystack)
+        if score:
+            scored.append((score, entry))
+
+    scored.sort(key=lambda x: (-x[0], x[1]["name"]))
+    picked = [dict(e) for _, e in scored[:limit]]
+
+    # An unrelated fallback is not a subject match. Return no suggestions if
+    # the supplied discipline does not match a curated repository.
+    if not picked:
+        return []
+
+    for p in picked:
+        p.pop("match", None)
+        p["verified_note"] = f"Verified {p['verified']} from {p['source_url']}"
+    return picked

scimago_index.py

@@ -183,6 +183,330 @@ def closest_subject_term(text: str, cutoff: float = 0.90) -> Optional[Dict[str,
     }
 
 
+# ── Deterministic nearest-subject resolution ──────────────────────────────
+#
+# Researchers type everyday discipline names ("physics", "maths", "machine
+# learning") that are not literal SCImago/ASJC names. Previously such input
+# depended entirely on the LLM classifier; when it was unavailable or answered
+# "not_found", the user saw no journals at all.
+#
+# This resolver is deliberately conservative and auditable:
+#   1. A reviewed alias table maps common discipline names onto real
+#      vocabulary terms. Every target is validated against the live
+#      vocabulary at call time, so a renamed category is skipped, never
+#      invented (a test also fails if any target disappears).
+#   2. Whole-word containment: every significant query word must appear in
+#      the vocabulary name ("physics" -> area "physics and astronomy").
+#   3. (Only after the LLM has failed) a partial match on a distinctive word.
+# Character-level similarity is NOT used here; that is the spelling repair in
+# closest_subject_term, which rejects "criminology" -> "immunology".
+
+# kind: "area" or "category". match: "synonym" for true equivalents, otherwise
+# "nearest". Alternatives are optional extra verified categories to offer.
+_SUBJECT_ALIASES: Dict[str, Dict[str, Any]] = {}
+
+
+def _alias(keys: List[str], kind: str, name: str, match: str = "nearest",
+           alternatives: Tuple[str, ...] = ()) -> None:
+    for key in keys:
+        _SUBJECT_ALIASES[_normalise_subject_match_text(key)] = {
+            "kind": kind, "name": name, "match": match,
+            "alternatives": list(alternatives),
+        }
+
+
+# Mathematics, statistics, physics
+_alias(["math", "maths", "mathematic"], "area", "mathematics", "synonym",
+       ("applied mathematics", "mathematics (miscellaneous)", "statistics and probability"))
+_alias(["stats"], "category", "statistics and probability", "synonym")
+_alias(["quantum physics", "quantum mechanics", "theoretical physics"], "area", "physics and astronomy",
+       alternatives=("atomic and molecular physics, and optics", "mathematical physics",
+                     "statistical and nonlinear physics", "nuclear and high energy physics"))
+_alias(["photonics", "optics and photonics", "lasers"], "category", "atomic and molecular physics, and optics",
+       alternatives=("electronic, optical and magnetic materials", "electrical and electronic engineering"))
+_alias(["semiconductors", "semiconductor", "microelectronics"], "category", "electronic, optical and magnetic materials",
+       alternatives=("electrical and electronic engineering", "condensed matter physics"))
+# Computing and AI
+_alias(["ai", "a i", "machine learning", "deep learning", "neural networks", "neural network",
+        "natural language processing", "nlp", "reinforcement learning", "generative ai",
+        "large language models", "llm", "llms"], "category", "artificial intelligence",
+       alternatives=("computer vision and pattern recognition", "computer science applications",
+                     "information systems"))
+_alias(["computer vision", "image processing", "pattern recognition"], "category",
+       "computer vision and pattern recognition",
+       alternatives=("artificial intelligence", "signal processing", "software"))
+_alias(["cs", "comp sci", "computing", "informatics", "computer engineering"], "area", "computer science",
+       alternatives=("computer science applications", "hardware and architecture", "software"))
+_alias(["data science", "big data", "data analytics", "data mining"], "area", "computer science",
+       alternatives=("information systems", "artificial intelligence", "statistics and probability"))
+_alias(["software engineering", "programming"], "category", "software",
+       alternatives=("computer science applications", "information systems"))
+_alias(["cybersecurity", "cyber security", "computer security", "information security",
+        "network security", "cryptography"], "category", "computer networks and communications",
+       alternatives=("information systems", "safety, risk, reliability and quality", "hardware and architecture"))
+_alias(["networking", "internet of things", "iot", "telecommunications", "telecommunication",
+        "wireless communications", "wireless communication", "communications engineering"],
+       "category", "computer networks and communications",
+       alternatives=("electrical and electronic engineering", "signal processing", "hardware and architecture"))
+_alias(["hci", "user experience", "ux"], "category", "human-computer interaction",
+       alternatives=("human factors and ergonomics", "computer science applications"))
+_alias(["robotics", "robots", "autonomous systems", "mechatronics"], "category", "control and systems engineering",
+       alternatives=("artificial intelligence", "mechanical engineering", "electrical and electronic engineering"))
+# Engineering, energy, materials
+_alias(["electronics", "electronic engineering"], "category", "electrical and electronic engineering", "synonym")
+_alias(["power systems", "power engineering", "smart grid", "smart grids", "energy storage",
+        "batteries", "battery"], "category", "energy engineering and power technology",
+       alternatives=("electrical and electronic engineering", "renewable energy, sustainability and the environment",
+                     "electrochemistry"))
+_alias(["solar energy", "wind energy", "clean energy", "green energy"], "category",
+       "renewable energy, sustainability and the environment",
+       alternatives=("energy engineering and power technology", "energy (miscellaneous)"))
+_alias(["hydrogen", "hydrogen energy", "fuel cells", "fuel cell"], "category", "fuel technology",
+       alternatives=("energy engineering and power technology", "renewable energy, sustainability and the environment",
+                     "electrochemistry"))
+_alias(["petroleum", "petroleum engineering", "oil and gas", "reservoir engineering", "petroleum geoscience"],
+       "category", "energy engineering and power technology",
+       alternatives=("fuel technology", "geotechnical engineering and engineering geology", "geochemistry and petrology"))
+_alias(["materials engineering", "material science", "materials"], "area", "materials science", "synonym",
+       alternatives=("materials chemistry", "mechanics of materials", "electronic, optical and magnetic materials"))
+_alias(["nanotech"], "category", "nanoscience and nanotechnology", "synonym")
+_alias(["biotech"], "category", "biotechnology", "synonym")
+_alias(["aeronautics", "aeronautical engineering", "aviation", "astronautics"], "category", "aerospace engineering")
+_alias(["heat transfer", "fluid mechanics", "fluid dynamics", "cfd"], "category", "fluid flow and transfer processes",
+       alternatives=("mechanical engineering", "computational mechanics"))
+_alias(["desalination", "water resources", "hydrology", "water treatment"], "category", "water science and technology",
+       alternatives=("filtration and separation", "environmental engineering"))
+_alias(["supply chain", "supply chain management", "logistics"], "category", "management science and operations research",
+       alternatives=("transportation", "strategy and management", "industrial and manufacturing engineering"))
+_alias(["transport", "transportation engineering", "traffic engineering"], "category", "transportation",
+       alternatives=("civil and structural engineering", "urban studies"))
+_alias(["urban planning", "urban design", "city planning", "regional planning"], "category", "urban studies",
+       alternatives=("geography, planning and development", "architecture"))
+_alias(["remote sensing", "gis", "geographic information systems", "geoinformatics"], "category",
+       "computers in earth sciences",
+       alternatives=("earth-surface processes", "geography, planning and development"))
+# Earth and environment
+_alias(["geoscience", "geosciences", "earth science", "earth sciences"], "area", "earth and planetary sciences", "synonym",
+       alternatives=("geology", "geophysics", "geochemistry and petrology"))
+_alias(["climate", "climate change", "climate science"], "category", "global and planetary change",
+       alternatives=("atmospheric science", "environmental science (miscellaneous)"))
+_alias(["meteorology"], "category", "atmospheric science")
+_alias(["environment", "environmental studies", "environmental sciences"], "area", "environmental science", "synonym",
+       alternatives=("environmental engineering", "pollution", "ecology"))
+_alias(["marine science", "marine sciences", "marine biology"], "category", "aquatic science",
+       alternatives=("oceanography", "ecology"))
+_alias(["agriculture", "agricultural sciences"], "area", "agricultural and biological sciences", "synonym",
+       alternatives=("agronomy and crop science", "soil science", "food science"))
+# Life and health sciences
+_alias(["life sciences", "life science", "biological sciences", "biosciences", "bioscience"], "area",
+       "biochemistry, genetics and molecular biology",
+       alternatives=("agricultural and biological sciences (miscellaneous)", "molecular biology", "cell biology", "ecology"))
+_alias(["bioinformatics", "computational biology"], "category", "molecular biology",
+       alternatives=("computational theory and mathematics", "computer science applications",
+                     "statistics and probability", "genetics"))
+_alias(["genomics", "genome"], "category", "genetics",
+       alternatives=("molecular biology", "biotechnology"))
+_alias(["biochem"], "category", "biochemistry", "synonym")
+_alias(["chem"], "area", "chemistry", "synonym")
+_alias(["neurobiology"], "area", "neuroscience",
+       alternatives=("cellular and molecular neuroscience", "cognitive neuroscience"))
+_alias(["biomedical", "biomedical science", "biomedical sciences", "biomedicine", "medical",
+        "medical sciences", "clinical", "clinical medicine", "health sciences", "healthcare", "health care"],
+       "area", "medicine",
+       alternatives=("molecular medicine", "biomedical engineering", "health policy",
+                     "public health, environmental and occupational health"))
+_alias(["health", "global health", "population health"], "category",
+       "public health, environmental and occupational health",
+       alternatives=("health policy", "epidemiology", "medicine (miscellaneous)"))
+_alias(["medical imaging"], "category", "radiology, nuclear medicine and imaging")
+_alias(["dental", "dental medicine", "oral health"], "area", "dentistry", "synonym",
+       alternatives=("oral surgery", "orthodontics", "periodontics"))
+_alias(["veterinary medicine", "veterinary science"], "area", "veterinary", "synonym")
+_alias(["covid", "covid 19", "covid19", "infectious disease"], "category", "infectious diseases",
+       alternatives=("epidemiology", "virology", "public health, environmental and occupational health"))
+_alias(["cognitive science"], "category", "experimental and cognitive psychology",
+       alternatives=("cognitive neuroscience", "artificial intelligence"))
+# Social sciences, business, humanities
+_alias(["politics", "political science", "international politics"], "category",
+       "political science and international relations", "synonym",
+       alternatives=("sociology and political science", "public administration"))
+_alias(["criminology", "criminal justice", "crime", "crime studies"], "category", "sociology and political science",
+       alternatives=("law", "pathology and forensic medicine", "social psychology"))
+_alias(["public policy", "governance"], "category", "public administration",
+       alternatives=("political science and international relations", "health policy"))
+_alias(["entrepreneurship"], "category", "business and international management",
+       alternatives=("strategy and management", "management of technology and innovation"))
+_alias(["hr", "hrm"], "category", "organizational behavior and human resource management", "synonym")
+_alias(["fintech"], "category", "finance",
+       alternatives=("information systems", "economics and econometrics"))
+_alias(["teaching", "pedagogy", "higher education", "engineering education", "medical education"],
+       "category", "education",
+       alternatives=("e-learning", "developmental and educational psychology"))
+_alias(["library science", "librarianship", "libraries", "library and information science",
+        "information science", "information studies"], "category", "library and information sciences", "synonym")
+_alias(["legal studies", "jurisprudence"], "category", "law", "synonym")
+_alias(["media", "media studies", "journalism", "mass communication"], "category", "communication",
+       alternatives=("cultural studies", "media technology"))
+_alias(["religion", "theology", "islamic studies"], "category", "religious studies",
+       alternatives=("philosophy", "history"))
+_alias(["ethics"], "category", "philosophy",
+       alternatives=("issues, ethics and legal aspects", "health policy"))
+_alias(["business", "business studies"], "area", "business, management and accounting", "synonym",
+       alternatives=("business and international management", "strategy and management", "marketing"))
+_alias(["economy", "banking"], "area", "economics, econometrics and finance",
+       alternatives=("economics and econometrics", "finance"))
+_alias(["animal", "animals", "zoology", "animal science"], "category", "animal science and zoology", "synonym")
+_alias(["ocean", "oceans", "ocean science", "ocean sciences"], "category", "oceanography", "synonym",
+       alternatives=("ocean engineering", "aquatic science"))
+_alias(["marine"], "category", "aquatic science", alternatives=("oceanography", "ocean engineering"))
+_alias(["film", "films", "cinema", "film studies", "theatre", "theater", "performing arts"], "category",
+       "visual arts and performing arts", alternatives=("communication", "cultural studies"))
+_alias(["nano"], "category", "nanoscience and nanotechnology", "synonym")
+_alias(["solar", "wind", "wind power", "solar power"], "category", "renewable energy, sustainability and the environment",
+       alternatives=("energy engineering and power technology",))
+_alias(["english literature", "literary studies", "literature studies"], "category", "literature and literary theory",
+       alternatives=("linguistics and language", "cultural studies"))
+
+# Words that never decide a nearest match on their own.
+_GENERIC_SUBJECT_TOKENS = frozenset({
+    "science", "sciences", "scientific", "study", "studies", "research", "engineering",
+    "technology", "technologies", "theory", "applied", "general", "miscellaneous",
+    "system", "systems", "learning", "process", "processes", "analysis", "development",
+    "international", "policy", "care", "methods", "method", "advanced", "modern",
+    "introduction", "principles", "topics", "issues", "aspects", "new",
+    # Ambiguous on their own; left to the semantic classifier.
+    "planning", "behavior", "behaviour", "family", "design", "film", "films",
+})
+
+
+def _token_matches(query_token: str, name_token: str) -> bool:
+    if query_token == name_token:
+        return True
+    for a, b in ((query_token, name_token), (name_token, query_token)):
+        if a == b + "s" or a == b + "es":
+            return True
+        if a.endswith("ies") and a[:-3] + "y" == b:
+            return True
+    return False
+
+
+def _covered(query_tokens: Set[str], name: str) -> Set[str]:
+    # Parenthetical qualifiers ("archeology (arts and humanities)") describe
+    # scope; they must not make a category look like a match for its parent.
+    name_tokens = _tokens(re.sub(r"\([^)]*\)", " ", name))
+    return {q for q in query_tokens if any(_token_matches(q, n) for n in name_tokens)}
+
+
+def _is_misc(name: str) -> bool:
+    return name.endswith("(miscellaneous)")
+
+
+def nearest_subject_term(text: str, allow_partial: bool = False) -> Optional[Dict[str, Any]]:
+    """Resolve free text to the nearest real area/category, or None.
+
+    Returns {"name", "kind" ("area"|"category"), "method", "match"
+    ("synonym"|"nearest"), "alternatives": [real category names]}.
+    ``method`` is "alias", "token_match" or "partial_token_match".
+    Exact names are not handled here (the caller checks them first).
+    """
+    vocab = subject_vocabulary()
+    cats, areas = vocab["categories"], vocab["areas"]
+
+    def real(kind: str, name: str) -> bool:
+        return name in (areas if kind == "area" else cats)
+
+    def result(name: str, kind: str, method: str, match: str, extra: List[str]) -> Dict[str, Any]:
+        alternatives: List[str] = []
+        for alt in extra:
+            if alt in cats and alt != name and alt not in alternatives:
+                alternatives.append(alt)
+        return {"name": name, "kind": kind, "method": method, "match": match,
+                "alternatives": alternatives[:6]}
+
+    key = _normalise_subject_match_text(text)
+    if not key:
+        return None
+
+    def related_suggestions() -> List[str]:
+        # Lexical suggestions that share a distinctive word with the input;
+        # a shared generic word ("science") is not a relationship.
+        words = {t for t in _tokens(key) if t not in _GENERIC_SUBJECT_TOKENS}
+        return [c for c in suggest_categories(text, limit=8) if _covered(words, c)]
+
+    alias = _SUBJECT_ALIASES.get(key)
+    if alias and real(alias["kind"], alias["name"]):
+        return result(alias["name"], alias["kind"], "alias", alias["match"],
+                      list(alias["alternatives"]) + related_suggestions())
+
+    query = _tokens(key)
+    distinctive = {t for t in query if t not in _GENERIC_SUBJECT_TOKENS and len(t) > 1}
+    if not distinctive:
+        return None
+
+    # Full containment: every significant query word appears in the name.
+    cat_hits = [c for c in cats if _covered(query, c) == query]
+    area_hits = [a for a in areas if _covered(query, a) == query]
+    specific = [c for c in cat_hits if not _is_misc(c)]
+
+    def best_category(candidates: List[str]) -> str:
+        # Most precise name first (fewest extra words), then most journals.
+        return sorted(candidates, key=lambda c: (len(_tokens(c)), -cats.get(c, 0), c))[0]
+
+    def extra_words(name: str) -> int:
+        return len(_tokens(re.sub(r"\([^)]*\)", " ", name)) - query)
+
+    chosen: Optional[Tuple[str, str]] = None
+    if len(specific) == 1 and len(area_hits) == 1:
+        # One specific category and one area: take the tighter name, so
+        # "astronomy" -> "astronomy and astrophysics" but "chemical" ->
+        # area "chemical engineering", not "chemical health and safety".
+        if extra_words(specific[0]) <= extra_words(area_hits[0]):
+            chosen = (specific[0], "category")
+        else:
+            chosen = (area_hits[0], "area")
+    elif len(specific) == 1:
+        chosen = (specific[0], "category")
+    elif len(area_hits) == 1:
+        chosen = (area_hits[0], "area")
+    elif specific:
+        chosen = (best_category(specific), "category")
+    elif cat_hits:
+        chosen = (best_category(cat_hits), "category")
+    if chosen:
+        name, kind = chosen
+        extra = specific + [c for c in cat_hits if c not in specific] + related_suggestions()
+        return result(name, kind, "token_match", "nearest", extra)
+
+    if not allow_partial:
+        return None
+
+    # Partial: the best candidate sharing at least one distinctive word,
+    # weighted by how much of the query it covers. Used only as a last
+    # resort when the semantic classifier is unavailable or gave up.
+    weights = _subject_token_weights()
+    total = sum(weights.get(t, 1.0) for t in distinctive) or 1.0
+    ranked: List[Tuple[float, int, str, str]] = []
+    for kind, names in (("area", areas), ("category", cats)):
+        for name in names:
+            hit = _covered(distinctive, name)
+            if not hit:
+                continue
+            coverage = sum(weights.get(t, 1.0) for t in hit) / total
+            extra_words = len(_tokens(name) - set(hit))
+            # Prefer areas for a broad single-word match and specific
+            # categories otherwise; tie-break on fewer extra words.
+            ranked.append((-coverage, extra_words + (0 if kind == "area" else 1), name, kind))
+    if not ranked:
+        return None
+    ranked.sort()
+    best_cov = -ranked[0][0]
+    if best_cov < 0.5:
+        return None
+    _cov, _extra, name, kind = ranked[0]
+    extra = [n for _c, _e, n, k in ranked[1:12] if k == "category"] + related_suggestions()
+    return result(name, kind, "partial_token_match", "nearest", extra)
+
+
 _subject_token_df_cache: Optional[Dict[str, float]] = None
 
 

trusted_links.py

@@ -1,283 +1,283 @@
-"""Evidence-backed hyperlinks. No model links, paid search, or client credentials."""
-from __future__ import annotations
-import asyncio
-import csv
-import functools
-import json
-import http.client
-import ipaddress
-import re
-import socket
-import ssl
-import time
-from collections import OrderedDict
-from datetime import datetime, timezone
-from html.parser import HTMLParser
-from pathlib import Path
-from urllib.parse import urlsplit, urljoin
-
-ROOT = Path(__file__).parent
-# Publisher identity AND destination must match. Unknown publishers fail closed.
-PUBLISHERS = {
-    'elsevier': ('elsevier.com', 'sciencedirect.com'),
-    'springer': ('springer.com', 'springernature.com', 'nature.com', 'biomedcentral.com'),
-    'nature': ('nature.com', 'springernature.com'),
-    'wiley': ('wiley.com',), 'taylor': ('tandfonline.com', 'taylorandfrancis.com'),
-    'sage': ('sagepub.com',), 'ieee': ('ieee.org',), 'mdpi': ('mdpi.com',),
-    'frontiers': ('frontiersin.org',), 'plos': ('plos.org',),
-    'oxford': ('oup.com',), 'cambridge': ('cambridge.org',),
-    'american chemical': ('acs.org',), 'royal society of chemistry': ('rsc.org',),
-    'iop': ('iop.org',), 'acm': ('acm.org',), 'american physical': ('aps.org',),
-}
-
-def safe_url(value):
-    try:
-        if not isinstance(value, str) or len(value) > 2048 or re.search(r'[\s\\\x00-\x1f]', value): return None
-        p = urlsplit(value)
-        if p.scheme != 'https' or not p.hostname or p.username or p.password or p.port not in (None,443): return None
-        h = p.hostname.lower()
-        if '.' not in h or h.endswith(('.local', '.internal', '.localhost')): return None
-        try:
-            if not ipaddress.ip_address(h).is_global: return None
-        except ValueError: pass
-        return value
-    except ValueError: return None
-
-def allowed(url, domains):
-    return bool(safe_url(url) and any(urlsplit(url).hostname == d or urlsplit(url).hostname.endswith('.'+d) for d in domains))
-
-def stamp(): return datetime.now(timezone.utc).isoformat(timespec='seconds')
-def unavailable(): return {'url': None, 'status':'unavailable', 'source':None, 'reviewed_at':None}
-
-def canonical(j):
-    from journal_lookup import get_index
-    from researchbee_journals import norm_issn
-    idx = get_index()
-    if not idx: return None
-    ident = j.get('identity_verification') or {}
-    sid = j.get('scopus_id') or ident.get('scopus_id')
-    issn = j.get('issn')
-    rec = idx.lookup_exact(scopus_id=str(sid)) if sid else idx.lookup_exact(issn=issn) if issn else idx.lookup_exact(title=j.get('name',''))
-    if rec and issn and norm_issn(issn) not in {norm_issn(rec.issn_e),norm_issn(rec.issn_print)}: return None
-    if rec and j.get('name'):
-        if idx.normalise_identity_title(j['name']) != idx.normalise_identity_title(rec.title): return None
-    return rec
-
-@functools.lru_cache(maxsize=1)
-def metric_data():
-    from journal_lookup import get_index
-    from scimago_index import parse_category_quartiles
-    idx = get_index()
-    names = {}
-    if idx:
-        for row in idx.con.execute('SELECT code, name FROM asjc'):
-            names.setdefault(row['name'].casefold(),set()).add(str(row['code']))
-    records = {}
-    with (ROOT/'scimagojr_2025.csv').open(encoding='latin-1',newline='') as f:
-        reader=csv.DictReader(f)
-        years=[re.fullmatch(r'Total Docs\. \((\d{4})\)',h) for h in reader.fieldnames]
-        year=next((m.group(1) for m in years if m),None)
-        for row in reader:
-            fields=[]
-            sid=row.get('Sourceid','')
-            profile=f'https://www.scimagojr.com/journalsearch.php?q={sid}&tip=sid' if sid.isdigit() else None
-            parsed=parse_category_quartiles(row.get('Categories',''))
-            labels={}
-            for part in row.get('Categories','').split(';'):
-                label=re.sub(r'\s*\(Q[1-4]\)\s*$','',part.strip(),flags=re.I)
-                if label: labels[' '.join(label.split()).casefold()]=label
-            for name,label in labels.items():
-                q=parsed.get(name)
-                codes=names.get(name.casefold(),set())
-                code=next(iter(codes)) if len(codes)==1 else None
-                fields.append({'field':label,'quartile':q,'code':code,'source':'SCImago SJR','year':year,
-                    'category_url':f'https://www.scimagojr.com/journalrank.php?category={code}&year={year}' if code and year else None,
-                    'profile_url':profile})
-            records[sid]={'fields':fields,'year':year,'profile_url':profile,'issns':set(re.findall(r'[0-9X]{8}',row.get('Issn','')))}
-    return records
-
-@functools.lru_cache(maxsize=1)
-def curated_records():
-    try:
-        return json.loads((ROOT/'journal_links_curated.json').read_text())
-    except (OSError, ValueError): return {}
-
-def curated_guidance(rec):
-    entry = curated_records().get(str(rec.scopus_id))
-    if not isinstance(entry,dict): return None
-    norm=lambda v: re.sub('[^0-9X]','',str(v).upper())
-    if not ({norm(rec.issn_e),norm(rec.issn_print)}-{''}) & {norm(v) for v in entry.get('issns',[])}: return None
-    if not safe_url(entry.get('url')) or not safe_url(entry.get('source')): return None
-    domains=tuple(d for key,ds in PUBLISHERS.items() if key in ((rec.publisher_group or '')+' '+(rec.publisher or '')).lower() for d in ds)
-    if not allowed(entry['url'],domains) or not allowed(entry['source'],domains): return None
-    if entry.get('scope') not in ('journal-specific','general-publisher'): return None
-    # Dates are retained exactly, never populated with this software release date.
-    return {**entry,'status':'curated'}
-
-_cache=OrderedDict()
-_metadata_cache=OrderedDict()
-_locks={}
-_gate=asyncio.Semaphore(4)
-
-def cached(key):
-    item=_cache.get(key)
-    if item and time.monotonic()<item[0]:
-        _cache.move_to_end(key); return item[1]
-    if item: del _cache[key]
-    return None
-
-def cache_put(key,value):
-    _cache[key]=(time.monotonic()+(86400 if value['status']=='checked' else 3600),value)
-    while len(_cache)>512: _cache.popitem(last=False)
-
-async def journal_links(j, fetch_metadata=True):
-    import openalex
-    rec=canonical(j)
-    result={'homepage':unavailable(),'guidelines':unavailable(),'can_discover':bool(rec)}
-    j['journal_links']=result
-    if not rec: return result
-    result['canonical_id']=rec.scopus_id
-    metric=metric_data().get(str(rec.scimago_id)) or {}
-    if not ({re.sub('[^0-9X]','',i or '') for i in (rec.issn_e,rec.issn_print)} & metric.get('issns',set())): metric={}
-    fields=metric.get('fields',[])
-    if not metric:
-        j['quartile']=None
-        if isinstance(j.get('ranking'),dict):
-            j['ranking'].update(quartile=None, verification_status='Not available')
-    j['quartiles_by_field']=fields
-    if isinstance(j.get('ranking'),dict):
-        j['ranking'].update(source='SCImago SJR' if metric else 'Not available',year=metric.get('year'),quartiles_by_field=fields)
-    vl=j.setdefault('verify_links',{})
-    vl['scimago']=metric.get('profile_url')
-    vl['scopus']=f'https://www.scopus.com/sourceid/{rec.scopus_id}' if str(rec.scopus_id).isdigit() else None
-    oa=j.get('openalex') or {}
-    expected={re.sub('[^0-9X]','',i or '') for i in (rec.issn_e,rec.issn_print)}-{''}
-    norm=lambda s:re.sub('[^0-9X]','',str(s).upper())
-    matched=lambda v:bool(expected & {norm(s) for s in [v.get('issn_l'),*(v.get('issns') or [])]})
-    if not matched(oa) and fetch_metadata:
-        item=_metadata_cache.get(rec.scopus_id)
-        if item and time.monotonic()<item[0]:
-            oa=item[1]
-            _metadata_cache.move_to_end(rec.scopus_id)
-        else:
-            oa=await openalex.enrich_journal(rec.title,rec.issn_e or rec.issn_print) or {}
-            _metadata_cache[rec.scopus_id]=(time.monotonic()+3600,oa)
-            while len(_metadata_cache)>512: _metadata_cache.popitem(last=False)
-    if matched(oa) and safe_url(oa.get('homepage_url')):
-        result['homepage']={'url':oa['homepage_url'],'status':'metadata-only','source':oa.get('openalex_id'),'source_updated_at':oa.get('source_updated_at'),'retrieved_at':oa.get('retrieved_at'),'reviewed_at':None}
-    result['guidelines']=curated_guidance(rec) or cached(rec.scopus_id) or unavailable()
-    return result
-
-class Anchors(HTMLParser):
-    def __init__(self): super().__init__(); self.links=[]; self.current=None; self.text=[]
-    def handle_starttag(self,tag,attrs):
-        if tag=='a': self.current=dict(attrs).get('href'); self.text=[]
-    def handle_data(self,data):
-        if self.current: self.text.append(data)
-    def handle_endtag(self,tag):
-        if tag=='a' and self.current:
-            self.links.append((self.current,' '.join(self.text))); self.current=None
-
-class PinnedHTTPS(http.client.HTTPSConnection):
-    def __init__(self,host,ip): super().__init__(host,timeout=3,context=ssl.create_default_context()); self.ip=ip
-    def connect(self):
-        self.sock=self._context.wrap_socket(socket.create_connection((self.ip,443),self.timeout),server_hostname=self.host)
-
-def fetch_page(url,domains):
-    """At most 3 requests, public DNS only, pinned IP + original TLS SNI/Host."""
-    deadline=time.monotonic()+6
-    for redirect in range(3):
-        if time.monotonic()>deadline: raise TimeoutError('Page deadline')
-        if not allowed(url,domains): raise ValueError('Destination not allowed')
-        p=urlsplit(url)
-        ips={x[4][0] for x in socket.getaddrinfo(p.hostname,443,type=socket.SOCK_STREAM)}
-        if not ips or any(not ipaddress.ip_address(ip).is_global for ip in ips): raise ValueError('Unsafe DNS')
-        conn=PinnedHTTPS(p.hostname,sorted(ips)[0])
-        try:
-            conn.request('GET',(p.path or '/')+('?' + p.query if p.query else ''),headers={'User-Agent':'ResearchBee/1.9 (library@ku.ac.ae)','Accept':'text/html','Accept-Encoding':'identity'})
-            response=conn.getresponse()
-            if response.status in (301,302,303,307,308):
-                if redirect==2: raise ValueError('Redirect limit')
-                url=urljoin(url,response.getheader('Location','')); continue
-            if response.status!=200 or 'text/html' not in (response.getheader('Content-Type') or ''): raise ValueError('Page unavailable')
-            chunks=[]; total=0
-            while total<=512000:
-                if time.monotonic()>deadline: raise TimeoutError('Page deadline')
-                chunk=response.read1(min(16384,512001-total))
-                if not chunk: break
-                chunks.append(chunk); total+=len(chunk)
-            data=b''.join(chunks)
-            if len(data)>512000: raise ValueError('Page too large')
-            return url,data.decode('utf-8',errors='replace')
-        finally: conn.close()
-    raise ValueError('Redirect limit')
-
-async def discover(j):
-    rec=canonical(j)
-    if not rec: return {'homepage':unavailable(),'guidelines':unavailable(),'can_discover':False}
-    async with _gate:
-        lock=_locks.setdefault(rec.scopus_id,asyncio.Lock())
-        try:
-            async with lock:
-                links=await journal_links(j)
-                prior=curated_guidance(rec) or cached(rec.scopus_id)
-                if prior: links['guidelines']=prior; return links
-                result=unavailable()
-                result['attempted_at']=stamp()
-                result['message']='Author guidance unavailable; use the known journal homepage if available.'
-                domains=tuple(d for key,ds in PUBLISHERS.items() if key in ((rec.publisher_group or '')+' '+(rec.publisher or '')).lower() for d in ds)
-                home=links['homepage'].get('url')
-                try:
-                    if not allowed(home,domains): raise ValueError('Official publisher not established')
-                    async with asyncio.timeout(15):
-                        page,html=await asyncio.to_thread(fetch_page,home,domains)
-                        # Require journal identity on the publisher page, not just its domain.
-                        from researchbee_journals import JournalIndex
-                        text=JournalIndex.normalise_identity_title(re.sub('<[^>]+>',' ',html))
-                        title=JournalIndex.normalise_identity_title(rec.title)
-                        if title not in text and not any(i and i.replace('-','') in re.sub('[^0-9X]','',html.upper()) for i in (rec.issn_e,rec.issn_print)):
-                            raise ValueError('Journal page identity missing')
-                        parser=Anchors(); parser.feed(html)
-                        candidates=[]
-                        for href,label in parser.links:
-                            if re.search(r'author.{0,25}(guid|instruction|information)|guide for authors|submission guidelines',label,re.I):
-                                target=urljoin(page,href)
-                                if allowed(target,domains): candidates.append(target)
-                        if not candidates: raise ValueError('No actual guidance link')
-                        target,body=await asyncio.to_thread(fetch_page,candidates[0],domains)
-                        headings = ' '.join(re.findall(r'<(?:title|h1)\b[^>]*>(.*?)</(?:title|h1)>', body, flags=re.I|re.S))
-                        home_parts=[p for p in urlsplit(page).path.split('/') if p]
-                        journal_slug=home_parts[-1] if home_parts else ''
-                        target_parts=urlsplit(target).path.split('/')
-                        specific=bool(journal_slug and journal_slug in target_parts and title in JournalIndex.normalise_identity_title(re.sub('<[^>]+>',' ',headings)))
-                        result={'url':target,'status':'checked','scope':'journal-specific' if specific else 'general-publisher',
-                            'source':page,'reviewed_at':stamp(),'message':'Journal-specific author instructions' if specific else 'General publisher guidance; journal-specific instructions were not established.'}
-                except Exception as exc:
-                    result['failure_reason']=type(exc).__name__
-                cache_put(rec.scopus_id,result); links['guidelines']=result
-                return links
-        finally:
-            # Keep locks only while waiters exist; global gate bounds active tasks.
-            if not lock.locked() and not getattr(lock,'_waiters',None): _locks.pop(rec.scopus_id,None)
-
-async def enrich_response(result):
-    jobs=[]
-    def walk(value):
-        if isinstance(value,dict):
-            if value.get('name') and ('ranking' in value or 'quartiles_by_field' in value or 'identity_verification' in value): jobs.append(value)
-            for child in list(value.values()): walk(child)
-        elif isinstance(value,list):
-            for child in value: walk(child)
-    walk(result)
-    semaphore=asyncio.Semaphore(5)
-    async def one(j):
-        async with semaphore:
-            try: await journal_links(j)
-            except Exception: j['journal_links']={'homepage':unavailable(),'guidelines':unavailable(),'can_discover':False}
-    await asyncio.gather(*(one(j) for j in jobs))
-    return result
-
-def with_links(fn):
-    @functools.wraps(fn)
-    async def wrapped(*args,**kwargs): return await enrich_response(await fn(*args,**kwargs))
-    return wrapped
+"""Evidence-backed hyperlinks. No model links, paid search, or client credentials."""
+from __future__ import annotations
+import asyncio
+import csv
+import functools
+import json
+import http.client
+import ipaddress
+import re
+import socket
+import ssl
+import time
+from collections import OrderedDict
+from datetime import datetime, timezone
+from html.parser import HTMLParser
+from pathlib import Path
+from urllib.parse import urlsplit, urljoin
+
+ROOT = Path(__file__).parent
+# Publisher identity AND destination must match. Unknown publishers fail closed.
+PUBLISHERS = {
+    'elsevier': ('elsevier.com', 'sciencedirect.com'),
+    'springer': ('springer.com', 'springernature.com', 'nature.com', 'biomedcentral.com'),
+    'nature': ('nature.com', 'springernature.com'),
+    'wiley': ('wiley.com',), 'taylor': ('tandfonline.com', 'taylorandfrancis.com'),
+    'sage': ('sagepub.com',), 'ieee': ('ieee.org',), 'mdpi': ('mdpi.com',),
+    'frontiers': ('frontiersin.org',), 'plos': ('plos.org',),
+    'oxford': ('oup.com',), 'cambridge': ('cambridge.org',),
+    'american chemical': ('acs.org',), 'royal society of chemistry': ('rsc.org',),
+    'iop': ('iop.org',), 'acm': ('acm.org',), 'american physical': ('aps.org',),
+}
+
+def safe_url(value):
+    try:
+        if not isinstance(value, str) or len(value) > 2048 or re.search(r'[\s\\\x00-\x1f]', value): return None
+        p = urlsplit(value)
+        if p.scheme != 'https' or not p.hostname or p.username or p.password or p.port not in (None,443): return None
+        h = p.hostname.lower()
+        if '.' not in h or h.endswith(('.local', '.internal', '.localhost')): return None
+        try:
+            if not ipaddress.ip_address(h).is_global: return None
+        except ValueError: pass
+        return value
+    except ValueError: return None
+
+def allowed(url, domains):
+    return bool(safe_url(url) and any(urlsplit(url).hostname == d or urlsplit(url).hostname.endswith('.'+d) for d in domains))
+
+def stamp(): return datetime.now(timezone.utc).isoformat(timespec='seconds')
+def unavailable(): return {'url': None, 'status':'unavailable', 'source':None, 'reviewed_at':None}
+
+def canonical(j):
+    from journal_lookup import get_index
+    from researchbee_journals import norm_issn
+    idx = get_index()
+    if not idx: return None
+    ident = j.get('identity_verification') or {}
+    sid = j.get('scopus_id') or ident.get('scopus_id')
+    issn = j.get('issn')
+    rec = idx.lookup_exact(scopus_id=str(sid)) if sid else idx.lookup_exact(issn=issn) if issn else idx.lookup_exact(title=j.get('name',''))
+    if rec and issn and norm_issn(issn) not in {norm_issn(rec.issn_e),norm_issn(rec.issn_print)}: return None
+    if rec and j.get('name'):
+        if idx.normalise_identity_title(j['name']) != idx.normalise_identity_title(rec.title): return None
+    return rec
+
+@functools.lru_cache(maxsize=1)
+def metric_data():
+    from journal_lookup import get_index
+    from scimago_index import parse_category_quartiles
+    idx = get_index()
+    names = {}
+    if idx:
+        for row in idx.con.execute('SELECT code, name FROM asjc'):
+            names.setdefault(row['name'].casefold(),set()).add(str(row['code']))
+    records = {}
+    with (ROOT/'scimagojr_2025.csv').open(encoding='latin-1',newline='') as f:
+        reader=csv.DictReader(f)
+        years=[re.fullmatch(r'Total Docs\. \((\d{4})\)',h) for h in reader.fieldnames]
+        year=next((m.group(1) for m in years if m),None)
+        for row in reader:
+            fields=[]
+            sid=row.get('Sourceid','')
+            profile=f'https://www.scimagojr.com/journalsearch.php?q={sid}&tip=sid' if sid.isdigit() else None
+            parsed=parse_category_quartiles(row.get('Categories',''))
+            labels={}
+            for part in row.get('Categories','').split(';'):
+                label=re.sub(r'\s*\(Q[1-4]\)\s*$','',part.strip(),flags=re.I)
+                if label: labels[' '.join(label.split()).casefold()]=label
+            for name,label in labels.items():
+                q=parsed.get(name)
+                codes=names.get(name.casefold(),set())
+                code=next(iter(codes)) if len(codes)==1 else None
+                fields.append({'field':label,'quartile':q,'code':code,'source':'SCImago SJR','year':year,
+                    'category_url':f'https://www.scimagojr.com/journalrank.php?category={code}&year={year}' if code and year else None,
+                    'profile_url':profile})
+            records[sid]={'fields':fields,'year':year,'profile_url':profile,'issns':set(re.findall(r'[0-9X]{8}',row.get('Issn','')))}
+    return records
+
+@functools.lru_cache(maxsize=1)
+def curated_records():
+    try:
+        return json.loads((ROOT/'journal_links_curated.json').read_text())
+    except (OSError, ValueError): return {}
+
+def curated_guidance(rec):
+    entry = curated_records().get(str(rec.scopus_id))
+    if not isinstance(entry,dict): return None
+    norm=lambda v: re.sub('[^0-9X]','',str(v).upper())
+    if not ({norm(rec.issn_e),norm(rec.issn_print)}-{''}) & {norm(v) for v in entry.get('issns',[])}: return None
+    if not safe_url(entry.get('url')) or not safe_url(entry.get('source')): return None
+    domains=tuple(d for key,ds in PUBLISHERS.items() if key in ((rec.publisher_group or '')+' '+(rec.publisher or '')).lower() for d in ds)
+    if not allowed(entry['url'],domains) or not allowed(entry['source'],domains): return None
+    if entry.get('scope') not in ('journal-specific','general-publisher'): return None
+    # Dates are retained exactly, never populated with this software release date.
+    return {**entry,'status':'curated'}
+
+_cache=OrderedDict()
+_metadata_cache=OrderedDict()
+_locks={}
+_gate=asyncio.Semaphore(4)
+
+def cached(key):
+    item=_cache.get(key)
+    if item and time.monotonic()<item[0]:
+        _cache.move_to_end(key); return item[1]
+    if item: del _cache[key]
+    return None
+
+def cache_put(key,value):
+    _cache[key]=(time.monotonic()+(86400 if value['status']=='checked' else 3600),value)
+    while len(_cache)>512: _cache.popitem(last=False)
+
+async def journal_links(j, fetch_metadata=True):
+    import openalex
+    rec=canonical(j)
+    result={'homepage':unavailable(),'guidelines':unavailable(),'can_discover':bool(rec)}
+    j['journal_links']=result
+    if not rec: return result
+    result['canonical_id']=rec.scopus_id
+    metric=metric_data().get(str(rec.scimago_id)) or {}
+    if not ({re.sub('[^0-9X]','',i or '') for i in (rec.issn_e,rec.issn_print)} & metric.get('issns',set())): metric={}
+    fields=metric.get('fields',[])
+    if not metric:
+        j['quartile']=None
+        if isinstance(j.get('ranking'),dict):
+            j['ranking'].update(quartile=None, verification_status='Not available')
+    j['quartiles_by_field']=fields
+    if isinstance(j.get('ranking'),dict):
+        j['ranking'].update(source='SCImago SJR' if metric else 'Not available',year=metric.get('year'),quartiles_by_field=fields)
+    vl=j.setdefault('verify_links',{})
+    vl['scimago']=metric.get('profile_url')
+    vl['scopus']=f'https://www.scopus.com/sourceid/{rec.scopus_id}' if str(rec.scopus_id).isdigit() else None
+    oa=j.get('openalex') or {}
+    expected={re.sub('[^0-9X]','',i or '') for i in (rec.issn_e,rec.issn_print)}-{''}
+    norm=lambda s:re.sub('[^0-9X]','',str(s).upper())
+    matched=lambda v:bool(expected & {norm(s) for s in [v.get('issn_l'),*(v.get('issns') or [])]})
+    if not matched(oa) and fetch_metadata:
+        item=_metadata_cache.get(rec.scopus_id)
+        if item and time.monotonic()<item[0]:
+            oa=item[1]
+            _metadata_cache.move_to_end(rec.scopus_id)
+        else:
+            oa=await openalex.enrich_journal(rec.title,rec.issn_e or rec.issn_print) or {}
+            _metadata_cache[rec.scopus_id]=(time.monotonic()+3600,oa)
+            while len(_metadata_cache)>512: _metadata_cache.popitem(last=False)
+    if matched(oa) and safe_url(oa.get('homepage_url')):
+        result['homepage']={'url':oa['homepage_url'],'status':'metadata-only','source':oa.get('openalex_id'),'source_updated_at':oa.get('source_updated_at'),'retrieved_at':oa.get('retrieved_at'),'reviewed_at':None}
+    result['guidelines']=curated_guidance(rec) or cached(rec.scopus_id) or unavailable()
+    return result
+
+class Anchors(HTMLParser):
+    def __init__(self): super().__init__(); self.links=[]; self.current=None; self.text=[]
+    def handle_starttag(self,tag,attrs):
+        if tag=='a': self.current=dict(attrs).get('href'); self.text=[]
+    def handle_data(self,data):
+        if self.current: self.text.append(data)
+    def handle_endtag(self,tag):
+        if tag=='a' and self.current:
+            self.links.append((self.current,' '.join(self.text))); self.current=None
+
+class PinnedHTTPS(http.client.HTTPSConnection):
+    def __init__(self,host,ip): super().__init__(host,timeout=3,context=ssl.create_default_context()); self.ip=ip
+    def connect(self):
+        self.sock=self._context.wrap_socket(socket.create_connection((self.ip,443),self.timeout),server_hostname=self.host)
+
+def fetch_page(url,domains):
+    """At most 3 requests, public DNS only, pinned IP + original TLS SNI/Host."""
+    deadline=time.monotonic()+6
+    for redirect in range(3):
+        if time.monotonic()>deadline: raise TimeoutError('Page deadline')
+        if not allowed(url,domains): raise ValueError('Destination not allowed')
+        p=urlsplit(url)
+        ips={x[4][0] for x in socket.getaddrinfo(p.hostname,443,type=socket.SOCK_STREAM)}
+        if not ips or any(not ipaddress.ip_address(ip).is_global for ip in ips): raise ValueError('Unsafe DNS')
+        conn=PinnedHTTPS(p.hostname,sorted(ips)[0])
+        try:
+            conn.request('GET',(p.path or '/')+('?' + p.query if p.query else ''),headers={'User-Agent':'ResearchBee/1.9 (library@ku.ac.ae)','Accept':'text/html','Accept-Encoding':'identity'})
+            response=conn.getresponse()
+            if response.status in (301,302,303,307,308):
+                if redirect==2: raise ValueError('Redirect limit')
+                url=urljoin(url,response.getheader('Location','')); continue
+            if response.status!=200 or 'text/html' not in (response.getheader('Content-Type') or ''): raise ValueError('Page unavailable')
+            chunks=[]; total=0
+            while total<=512000:
+                if time.monotonic()>deadline: raise TimeoutError('Page deadline')
+                chunk=response.read1(min(16384,512001-total))
+                if not chunk: break
+                chunks.append(chunk); total+=len(chunk)
+            data=b''.join(chunks)
+            if len(data)>512000: raise ValueError('Page too large')
+            return url,data.decode('utf-8',errors='replace')
+        finally: conn.close()
+    raise ValueError('Redirect limit')
+
+async def discover(j):
+    rec=canonical(j)
+    if not rec: return {'homepage':unavailable(),'guidelines':unavailable(),'can_discover':False}
+    async with _gate:
+        lock=_locks.setdefault(rec.scopus_id,asyncio.Lock())
+        try:
+            async with lock:
+                links=await journal_links(j)
+                prior=curated_guidance(rec) or cached(rec.scopus_id)
+                if prior: links['guidelines']=prior; return links
+                result=unavailable()
+                result['attempted_at']=stamp()
+                result['message']='Author guidance unavailable; use the known journal homepage if available.'
+                domains=tuple(d for key,ds in PUBLISHERS.items() if key in ((rec.publisher_group or '')+' '+(rec.publisher or '')).lower() for d in ds)
+                home=links['homepage'].get('url')
+                try:
+                    if not allowed(home,domains): raise ValueError('Official publisher not established')
+                    async with asyncio.timeout(15):
+                        page,html=await asyncio.to_thread(fetch_page,home,domains)
+                        # Require journal identity on the publisher page, not just its domain.
+                        from researchbee_journals import JournalIndex
+                        text=JournalIndex.normalise_identity_title(re.sub('<[^>]+>',' ',html))
+                        title=JournalIndex.normalise_identity_title(rec.title)
+                        if title not in text and not any(i and i.replace('-','') in re.sub('[^0-9X]','',html.upper()) for i in (rec.issn_e,rec.issn_print)):
+                            raise ValueError('Journal page identity missing')
+                        parser=Anchors(); parser.feed(html)
+                        candidates=[]
+                        for href,label in parser.links:
+                            if re.search(r'author.{0,25}(guid|instruction|information)|guide for authors|submission guidelines',label,re.I):
+                                target=urljoin(page,href)
+                                if allowed(target,domains): candidates.append(target)
+                        if not candidates: raise ValueError('No actual guidance link')
+                        target,body=await asyncio.to_thread(fetch_page,candidates[0],domains)
+                        headings = ' '.join(re.findall(r'<(?:title|h1)\b[^>]*>(.*?)</(?:title|h1)>', body, flags=re.I|re.S))
+                        home_parts=[p for p in urlsplit(page).path.split('/') if p]
+                        journal_slug=home_parts[-1] if home_parts else ''
+                        target_parts=urlsplit(target).path.split('/')
+                        specific=bool(journal_slug and journal_slug in target_parts and title in JournalIndex.normalise_identity_title(re.sub('<[^>]+>',' ',headings)))
+                        result={'url':target,'status':'checked','scope':'journal-specific' if specific else 'general-publisher',
+                            'source':page,'reviewed_at':stamp(),'message':'Journal-specific author instructions' if specific else 'General publisher guidance; journal-specific instructions were not established.'}
+                except Exception as exc:
+                    result['failure_reason']=type(exc).__name__
+                cache_put(rec.scopus_id,result); links['guidelines']=result
+                return links
+        finally:
+            # Keep locks only while waiters exist; global gate bounds active tasks.
+            if not lock.locked() and not getattr(lock,'_waiters',None): _locks.pop(rec.scopus_id,None)
+
+async def enrich_response(result):
+    jobs=[]
+    def walk(value):
+        if isinstance(value,dict):
+            if value.get('name') and ('ranking' in value or 'quartiles_by_field' in value or 'identity_verification' in value): jobs.append(value)
+            for child in list(value.values()): walk(child)
+        elif isinstance(value,list):
+            for child in value: walk(child)
+    walk(result)
+    semaphore=asyncio.Semaphore(5)
+    async def one(j):
+        async with semaphore:
+            try: await journal_links(j)
+            except Exception: j['journal_links']={'homepage':unavailable(),'guidelines':unavailable(),'can_discover':False}
+    await asyncio.gather(*(one(j) for j in jobs))
+    return result
+
+def with_links(fn):
+    @functools.wraps(fn)
+    async def wrapped(*args,**kwargs): return await enrich_response(await fn(*args,**kwargs))
+    return wrapped