Upload 7 files

README-DEPLOY.md

@@ -1,25 +1,22 @@
-# ResearchBee v1.8.0 — deployment and maintenance guide
+# ResearchBee v1.8.1 — deployment and maintenance guide
 
-This release replaces the two independent journal-suggestion paths with one verified, database-first journal-discovery system. The same filters, policy checks, scoring, evidence, comparison, and export are available in **Browse by subject** and **Analyse manuscript**.
+This patch aligns the LibGuides display with the shared, database-first journal-discovery system introduced in v1.8.0 and strengthens subject correction and nearest-category mapping. The same filters, policy checks, scoring, evidence, comparison, and export remain available in **Browse by subject** and **Analyse manuscript**.
 
-## 1. Files changed or added in v1.8.0
+## 1. Files changed or added in v1.8.1
 
 | File | Action | Purpose |
 |---|---|---|
-| `app.py` | Replace | Unified journal endpoints, bounded policy batch endpoint, verified result schema |
-| `journal_discovery.py` | Add | Shared deterministic candidate generation, filtering, score helpers, policy tri-state |
-| `journal_lookup.py` | Replace | Exposes the verified `JournalIndex` instance |
-| `researchbee_journals.py` | Replace | Active-record recommendations, exact quartile/access/exclusion filters, preferred-title scoring |
-| `openalex.py` | Replace | Recent journal-article evidence and local title-overlap signal |
-| `openpolicyfinder.py` | Replace | v1.8 client identity and shared journal-policy integration |
-| `prompts.py` | Replace | Closed-vocabulary profiling and closed-list manuscript reranking |
-| `README.md` | Replace | v1.8 architecture and evidence boundaries |
+| `app.py` | Replace | Version alignment, closed-vocabulary spelling repair, validated nearest-category fallback, and mapping provenance |
+| `scimago_index.py` | Replace | Conservative spelling matcher and improved deterministic category suggestions |
+| `prompts.py` | Replace | Explicit synonym, spelling-correction, nearest-category, and no-match decisions constrained to the approved vocabulary |
+| `openpolicyfinder.py` | Replace | v1.8.1 client identity |
+| `README.md` | Replace | v1.8.1 behavior and evidence boundaries |
 | `README-DEPLOY.md` | Replace | This deployment guide |
-| `RELEASE_1.8.0.md` | Add | Release notes and migration summary |
-| `ResearchBee_LibGuides.html` | Replace in LibGuides only | Unified interface for both journal-discovery modes |
-| `tests/` | Replace/add for source repository | Regression coverage for both routes and the front end |
+| `RELEASE_1.8.1.md` | Add | Patch release notes and exact migration steps |
+| `ResearchBee_LibGuides.html` | Replace in LibGuides only | Unified comparison/OA rendering, schema-mismatch protection, and transparent subject-mapping notices |
+| `tests/` | Replace/add for source repository | Regression coverage for both journal modes and subject mapping |
 
-Existing data files such as `journals.db` and `scimagojr_2025.csv` are not altered by this release and should remain in the Space.
+Existing database and data files such as `journals.db` and `scimagojr_2025.csv` are not altered and should remain in the Space.
 
 ## 2. Hugging Face Space secrets
 
@@ -46,7 +43,7 @@ OPF_API_MAX_POLICY_HYDRATIONS=6
 
 ## 3. Backend deployment
 
-Upload the changed backend files to the repository root. Allow the files marked **Replace** above to overwrite the existing versions, and add the new `journal_discovery.py` and `RELEASE_1.8.0.md` files.
+Upload the seven changed backend files listed above to the repository root. Allow the six replacement files to overwrite their existing versions and add `RELEASE_1.8.1.md`.
 
 After the Space rebuilds, check:
 
@@ -56,7 +53,7 @@ GET /api/health
 
 Expected indicators:
 
-- `version: 1.8.0`
+- `version: 1.8.1`
 - journal index loaded with an active-record count
 - Open Policy Finder configured when its secret is present
 - knowledge base loaded
@@ -193,13 +190,20 @@ node --check /tmp/researchbee.js
 6. Use **Review or correct inferred details**, amend a field, and resubmit.
 7. Compare and export two to five journals.
 
+### Subject spelling and taxonomy fallback
+
+1. Search `artifical inteligence` and confirm the interface shows the correction to **Artificial Intelligence** before returning verified journals.
+2. Search a meaningful unlisted subject such as `criminology`; confirm the interface states that no exact taxonomy category exists, names the nearest verified category used, and offers alternatives.
+3. Search meaningless input such as `asdfgh`; confirm ResearchBee does not force a category and instead returns verified suggestions or a safe no-match message.
+4. Confirm every returned category exists in the bundled SCImago/ASJC vocabulary.
+
 ### Safety and regression
 
 1. Disconnect or remove the OPF key in a test environment. Policy results must remain unavailable/unknown rather than guessed.
 2. Temporarily make the journal database unavailable. Both journal routes must return a safe error rather than model-created titles.
 3. Confirm the DOI-based licence workflow still uses OA.Works and Khazna independently.
 4. Confirm repository finder, chat, service introduction, language controls, mobile tabs, cover-letter generation, and related works still operate.
-5. Confirm `/api/health` reports `1.8.0`.
+5. Confirm `/api/health` reports `1.8.1`.
 
 ## 9. Known evidence limitations
 

README.md

@@ -10,7 +10,7 @@ license: mit
 short_description: Journal selection, Open Access and research data assistant
 ---
 
-# 🐝 ResearchBee v1.8.0
+# 🐝 ResearchBee v1.8.1
 
 **Publishing, Open Access, and research-data assistant for Khalifa University researchers**
 
@@ -21,6 +21,21 @@ ResearchBee combines three structured tools with a source-aware chat assistant:
 3. **Deposit your data** — match a dataset to curated repository records and preserve factual repository attributes on the server.
 4. **Ask ResearchBee** — get guidance on journal selection, metrics, publishing practice, KU Open Access support, research data management, repository selection, and responsible use of AI.
 
+## What changed in v1.8.1
+
+### Journal Finder display alignment
+
+- The LibGuides interface now checks for the `journal_discovery_v2` backend schema and reports a clear version-mismatch message instead of silently falling back to the legacy journal-card renderer.
+- Both **Browse by subject** and **Analyse manuscript** use the same access-model badge, comparison selector, comparison table, CSV export, and on-demand sharing-policy controls.
+- Journal-level OA source flags are labelled separately from manuscript-sharing policy. An unchecked sharing policy no longer makes an open-access journal appear to have an unverified OA status.
+
+### Subject spelling and nearest-category mapping
+
+- High-confidence spelling mistakes are corrected locally against the closed SCImago/ASJC vocabulary; the original and corrected terms are shown to the user.
+- Meaningful subjects that are absent from the bundled taxonomy may be mapped by the language model to the nearest defensible listed category. The server validates the selected term against the approved vocabulary and explicitly states that a nearest-category mapping was used.
+- Alternative verified categories are displayed as one-click refinements. Meaningless or materially ambiguous input is not forced into a category.
+- Deterministic suggestions now weight distinctive terms more strongly than generic words such as “science”.
+
 ## What changed in v1.8.0
 
 ### One journal-discovery system for both modes
@@ -98,7 +113,7 @@ OPEN_POLICY_FINDER_API_KEY
 
 `OPEN_POLICY_FINDER_API_KEY` is required for journal-level policy checks in both Journal Finder modes and in the separate licence workflow. It must be stored only as a Hugging Face Space secret.
 
-## Main runtime files for v1.8.0
+## Main runtime files for v1.8.1
 
 ```text
 app.py

RELEASE_1.8.1.md

@@ -0,0 +1,54 @@
+# ResearchBee v1.8.1 — Journal Finder display and subject-mapping patch
+
+Released: 2026-09-09
+
+## Summary
+
+ResearchBee v1.8.1 fixes a front-end/back-end alignment problem that could make **Analyse manuscript** use the legacy result renderer. In that state, the new top-level journal access data was ignored, every card could show an old “not verified” policy status, and the unified comparison controls were absent. The patch also makes subject spelling correction and nearest-category fallback explicit and testable.
+
+## Changes
+
+### Both Journal Finder modes
+
+- **Browse by subject** and **Analyse manuscript** now enforce the same `journal_discovery_v2` display path.
+- Both modes expose journal selection, a two-to-five-journal comparison table, CSV export, access-model source flags, and on-demand sharing-policy checks.
+- The interface detects a mismatched backend schema and asks for a refresh after deployment rather than silently displaying legacy cards.
+- Journal access model and manuscript-sharing policy remain separate: an unchecked Open Policy Finder policy does not mean that the journal’s OA status is unverified.
+
+### Subject input
+
+- Added conservative high-confidence spelling correction against the approved local category and area vocabulary.
+- Added a constrained LLM nearest-category route for meaningful subjects not represented exactly in the taxonomy.
+- The server validates every returned category or area before use.
+- The result states the original term, the verified category/area selected, and whether the selection was a spelling correction, synonym, broad-area mapping, or nearest-category fallback.
+- Alternative verified categories are provided as one-click refinements.
+- Nonsense or unsafe ambiguity remains a no-match; ResearchBee does not force a category.
+
+## Files to upload to Hugging Face
+
+Add or replace only:
+
+```text
+app.py
+scimago_index.py
+prompts.py
+openpolicyfinder.py
+README.md
+README-DEPLOY.md
+RELEASE_1.8.1.md
+```
+
+Do not delete or re-upload `journals.db`, `scimagojr_2025.csv`, or other unchanged runtime files.
+
+## LibGuides
+
+Replace the complete existing ResearchBee source fragment with `ResearchBee_LibGuides_v1.8.1.html`. Do not upload the HTML file to the Hugging Face Space.
+
+After publishing the LibGuide, hard-refresh the page or open it in a private window. The footer should show **Interface v1.8.1**.
+
+## Verification
+
+- 49 automated tests passed.
+- Python source compilation passed.
+- Embedded LibGuides JavaScript syntax validation passed.
+- Browser interaction tests covered manuscript OA display, manuscript comparison, subject mapping disclosure, and alternative-category refinement.

app.py

@@ -1,5 +1,11 @@
 """
-ResearchBee - FastAPI backend v1.8.0 (unified verified journal discovery)
+ResearchBee - FastAPI backend v1.8.1 (journal UI alignment and subject correction)
+
+Changes in v1.8.1:
+- Subject browsing applies conservative closed-vocabulary spelling correction before the LLM.
+- Meaningful unlisted subjects may be mapped to the nearest verified category with explicit disclosure.
+- Subject responses include structured mapping provenance and alternative verified categories.
+- The backend version aligns with the unified comparison/OA front end.
 
 Changes in v1.8.0:
 - Browse by subject and manuscript matching now share one database-first journal pipeline.
@@ -77,7 +83,7 @@ import httpx
 from journal_lookup import (
     lookup_any, load_scimago, search_by_subject, load_subject_index,
     subject_vocabulary, is_real_category, is_real_area, suggest_categories,
-    get_index,
+    closest_subject_term, get_index,
 )
 import openalex
 import openpolicyfinder
@@ -446,7 +452,7 @@ async def lifespan(app: FastAPI):
 
 
 # ── App ────────────────────────────────────────────────────────────────────
-app = FastAPI(title="ResearchBee API", version="1.8.0", lifespan=lifespan)
+app = FastAPI(title="ResearchBee API", version="1.8.1", lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -1014,7 +1020,7 @@ async def health():
     return {
         "status": "ok",
         "service": "ResearchBee",
-        "version": "1.8.0",
+        "version": "1.8.1",
         # Restart detection for the uptime monitor. boot_ts is the stable
         # identity of this process; when it changes between two polls the
         # container was replaced, whatever the HTTP status said.
@@ -2353,9 +2359,11 @@ async def browse_subject(req: SubjectRequest, request: Request):
     subject_text = req.subject.strip()
     typed_category = subject_text if is_real_category(subject_text) else ""
     typed_area = subject_text if is_real_area(subject_text) else ""
+    spelling_match = None if (typed_category or typed_area) else closest_subject_term(subject_text)
 
-    # An exact user-entered vocabulary term is stronger evidence than a model
-    # interpretation and needs no external classification call.
+    # Exact vocabulary terms and conservative spelling repairs are resolved
+    # locally. The LLM is reserved for synonym and semantic nearest-category
+    # mapping, and may only return names from the closed vocabulary.
     if typed_category or typed_area:
         parsed = {
             "confidence": "exact",
@@ -2363,15 +2371,33 @@ async def browse_subject(req: SubjectRequest, request: Request):
             "scimago_category": typed_category,
             "scimago_area": typed_area,
             "nearest_categories": [],
+            "mapping_type": "exact",
+            "use_nearest_category": False,
             "user_message": "Matched directly to the verified ASJC/SCImago vocabulary.",
         }
+    elif spelling_match:
+        corrected = str(spelling_match["name"])
+        is_category = spelling_match["kind"] == "category"
+        parsed = {
+            "confidence": "near",
+            "normalised": corrected,
+            "scimago_category": corrected if is_category else "",
+            "scimago_area": corrected if not is_category else "",
+            "nearest_categories": suggest_categories(corrected, limit=4),
+            "mapping_type": "spelling_correction",
+            "use_nearest_category": True,
+            "user_message": (
+                f'Spelling corrected: "{subject_text}" → "{corrected.title()}". '
+                f'Results use this verified {spelling_match["kind"]}.'
+            ),
+        }
     else:
         try:
             parsed = await call_openai(
                 get_client(),
                 subject_prompt_with_vocabulary(),
                 f"{lang_instruction(req.language)}Subject input: {subject_text}",
-                max_tokens=450,
+                max_tokens=520,
                 model=MODEL_LIGHT,
             )
         except (RateLimitError, APIStatusError, HTTPException) as exc:
@@ -2382,6 +2408,8 @@ async def browse_subject(req: SubjectRequest, request: Request):
                 "scimago_category": "",
                 "scimago_area": "",
                 "nearest_categories": [],
+                "mapping_type": "none",
+                "use_nearest_category": False,
                 "user_message": "The subject classifier was unavailable. Choose a verified category suggestion or try again.",
             }
         except Exception as exc:
@@ -2392,15 +2420,32 @@ async def browse_subject(req: SubjectRequest, request: Request):
                 "scimago_category": "",
                 "scimago_area": "",
                 "nearest_categories": [],
+                "mapping_type": "none",
+                "use_nearest_category": False,
                 "user_message": "The subject could not be classified safely. Choose a verified category suggestion or try again.",
             }
 
     confidence = parsed.get("confidence", "not_found")
+    if confidence not in {"exact", "near", "not_found"}:
+        confidence = "not_found"
+    mapping_type = str(parsed.get("mapping_type") or "").strip().lower()
+    if mapping_type not in {"exact", "synonym", "spelling_correction", "nearest_category", "none"}:
+        mapping_type = "nearest_category" if confidence == "near" else "none"
+    use_nearest_category = parsed.get("use_nearest_category") is True
     normalised = str(parsed.get("normalised") or subject_text).strip()
     category = str(parsed.get("scimago_category") or "").strip()
     area = str(parsed.get("scimago_area") or "").strip()
     user_message = " ".join(str(parsed.get("user_message") or "").split())
 
+    nearest_categories: list[str] = []
+    nearest_seen: set[str] = set()
+    for name in parsed.get("nearest_categories") or []:
+        value = str(name).strip()
+        key = value.lower()
+        if key and key not in nearest_seen and is_real_category(value):
+            nearest_seen.add(key)
+            nearest_categories.append(value)
+
     bad_category = bool(category) and not is_real_category(category)
     bad_area = bool(area) and not is_real_area(area)
     if bad_category:
@@ -2410,28 +2455,81 @@ async def browse_subject(req: SubjectRequest, request: Request):
         logger.warning("[BrowseSubject] discarded unlisted area %r", area)
         area = ""
 
-    # Exact typed terms always win over a model's alternative valid category.
+    # Exact typed terms always win over a model's alternative valid term.
     if typed_category:
         category = typed_category
+        area = ""
         confidence = "exact"
+        mapping_type = "exact"
     if typed_area:
         area = typed_area
+        category = ""
         confidence = "exact"
-
-    def suggestions() -> list:
-        out, seen = [], set()
-        for name in parsed.get("nearest_categories") or []:
+        mapping_type = "exact"
+
+    # A model may conservatively return the chosen nearest category only in
+    # nearest_categories. Use it only when the model explicitly says the
+    # fallback is safe, or when it labelled the response "near". A true
+    # not_found with use_nearest_category=false remains a no-result response.
+    used_nearest_fallback = False
+    if not category and not area and nearest_categories and (
+        use_nearest_category or confidence == "near"
+    ):
+        category = nearest_categories[0]
+        confidence = "near"
+        mapping_type = "nearest_category"
+        used_nearest_fallback = True
+        if not normalised or normalised == subject_text:
+            normalised = category
+
+    selected_term = category or area
+    if selected_term:
+        display_term = selected_term.title()
+        if mapping_type == "spelling_correction":
+            disclosure = (
+                f'Spelling corrected: "{subject_text}" → "{display_term}". '
+                f'Results use this verified {"category" if category else "broad area"}.'
+            )
+        elif mapping_type == "nearest_category" or used_nearest_fallback:
+            disclosure = (
+                f'SCImago and Scopus have no exact subject category for "{subject_text}". '
+                f'ResearchBee mapped it to the nearest verified category "{display_term}".'
+            )
+        elif mapping_type == "synonym" and selected_term.lower() != subject_text.lower():
+            disclosure = (
+                f'Interpreted "{subject_text}" as the verified '
+                f'{"category" if category else "area"} "{display_term}".'
+            )
+        elif confidence == "near" and selected_term.lower() != subject_text.lower():
+            disclosure = (
+                f'Interpreted "{subject_text}" using the closest verified '
+                f'{"category" if category else "area"} "{display_term}".'
+            )
+        else:
+            disclosure = ""
+        if disclosure and disclosure.lower() not in user_message.lower():
+            user_message = f"{disclosure} {user_message}".strip()
+
+    def suggestions() -> list[dict]:
+        out: list[str] = []
+        seen: set[str] = set()
+        for name in nearest_categories:
             key = str(name).strip().lower()
             if key and key not in seen and is_real_category(key):
-                seen.add(key); out.append(key)
-        for name in suggest_categories(req.subject.strip(), limit=6):
-            if name not in seen:
-                seen.add(name); out.append(name)
+                seen.add(key)
+                out.append(key)
+        for name in suggest_categories(subject_text, limit=8):
+            key = str(name).strip().lower()
+            if key and key not in seen and is_real_category(key):
+                seen.add(key)
+                out.append(key)
         counts = subject_vocabulary()["categories"]
         return [{"category": name.title(), "n_journals": counts.get(name, 0)} for name in out[:6]]
 
     have_category = bool(category)
-    have_area = bool(area) and bool(typed_area)
+    # Permit a corrected or LLM-resolved broad area when no category is
+    # available. Exact typed areas continue to take precedence.
+    have_area = bool(area) and not have_category
     if confidence == "not_found" or (not have_category and not have_area):
         usage.usage.record_gap(
             "journal_subject",
@@ -2439,9 +2537,9 @@ async def browse_subject(req: SubjectRequest, request: Request):
         )
         request.state.usage_recorded = True
         nearby = suggestions()
-        base = f"SCImago and Scopus have no exact subject category for '{req.subject.strip()}'."
+        base = f"SCImago and Scopus have no exact subject category for '{subject_text}'."
         hint = (
-            " Closest indexed categories: " + ", ".join(item["category"] for item in nearby) + "."
+            " Closest verified categories: " + ", ".join(item["category"] for item in nearby) + "."
             if nearby else ""
         )
         return {"result": {
@@ -2450,6 +2548,16 @@ async def browse_subject(req: SubjectRequest, request: Request):
             "confidence": "not_found",
             "normalised": normalised,
             "user_message": (user_message + " " if user_message else "") + base + hint,
+            "subject_mapping": {
+                "original_input": subject_text,
+                "normalised": normalised,
+                "match_type": "none",
+                "selected_category": "",
+                "selected_area": "",
+                "used_nearest_category": False,
+                "spelling_corrected": False,
+                "alternatives": nearby,
+            },
             "suggestions": nearby,
             "journals": [],
             "extended_list": [],
@@ -2500,13 +2608,18 @@ async def browse_subject(req: SubjectRequest, request: Request):
         unverified.extend(policy_notes)
     if have_area and not have_category:
         area_note = (
-            f"'{req.subject.strip()}' is a broad subject area. Results span the whole of "
-            f"'{area.title()}'; choose a narrower category for a more specific list."
+            f"'{subject_text}' was resolved to the broad subject area "
+            f"'{area.title()}'. Results span that area; choose a narrower category for a more specific list."
         )
         user_message = f"{user_message} {area_note}".strip()
         usage.usage.record_gap("journal_subject", "area_fallback")
         request.state.usage_recorded = True
 
+    all_suggestions = suggestions()
+    alternatives = [
+        item for item in all_suggestions
+        if item["category"].lower() != str(category).lower()
+    ][:4]
     result = _finalise_discovery_result(
         mode="subject",
         ranked=ranked,
@@ -2521,6 +2634,9 @@ async def browse_subject(req: SubjectRequest, request: Request):
             "category": category,
             "area": area,
             "confidence": confidence,
+            "mapping_type": mapping_type,
+            "original_input": subject_text,
+            "alternatives": alternatives,
         },
         classification_note=(
             "This is a broad area result, not a manuscript-specific fit assessment."
@@ -2535,7 +2651,17 @@ async def browse_subject(req: SubjectRequest, request: Request):
         "scimago_area": area,
         "user_message": user_message,
         "asjc_codes": asjc_codes,
-        "suggestions": suggestions() if confidence == "near" else [],
+        "subject_mapping": {
+            "original_input": subject_text,
+            "normalised": normalised,
+            "match_type": mapping_type,
+            "selected_category": category,
+            "selected_area": area,
+            "used_nearest_category": mapping_type == "nearest_category" or used_nearest_fallback,
+            "spelling_corrected": mapping_type == "spelling_correction",
+            "alternatives": alternatives,
+        },
+        "suggestions": alternatives if confidence == "near" else [],
     })
     if not ranked:
         result["filter_no_match"] = True

openpolicyfinder.py

@@ -55,7 +55,7 @@ LICENCE_NAME = "CC BY-NC-ND 4.0"
 LICENCE_URL = "https://creativecommons.org/licenses/by-nc-nd/4.0/"
 CONTACT = "nikesh.narayanan@ku.ac.ae"
 USER_AGENT = (
-    "ResearchBee/1.8.0 (Khalifa University Library; "
+    "ResearchBee/1.8.1 (Khalifa University Library; "
     f"mailto:{CONTACT}; Open Policy Finder API client)"
 )
 

prompts.py

@@ -265,41 +265,58 @@ Rules:
 """
 
 # ── Subject normalisation ─────────────────────────────────────────────────
-SUBJECT_NORMALISE_PROMPT = """You are an academic subject classification expert. Map the user's subject input onto SCImago's classification.
-
-You MUST choose scimago_category from the VALID CATEGORIES list supplied below,
-copied character for character. You MUST choose scimago_area from the VALID
-AREAS list. Do not invent, translate, pluralise, or reword a name. If nothing in
-the lists fits, that is a legitimate answer — return confidence "not_found" and
-say which listed categories come closest. A wrong-but-plausible category name is
-far worse than an honest miss: it silently returns an unrelated journal list.
+SUBJECT_NORMALISE_PROMPT = """You are an academic subject classification expert. Map the user's free-text subject onto the verified SCImago/ASJC vocabulary supplied below.
+
+Closed-vocabulary rules:
+- Copy scimago_category character for character from VALID CATEGORIES, or use "".
+- Copy scimago_area character for character from VALID AREAS, or use "".
+- Never invent, translate, pluralise, or reword a category or area name.
+- A meaningful academic subject that has no exact category should normally be
+  mapped to the single closest defensible listed category. Mark it as "near"
+  and explicitly disclose that the exact subject is absent from the taxonomy.
+- Use "not_found" only when the input is meaningless, is not an academic
+  subject, or is too ambiguous to map responsibly. Do not force a match.
 
 Return ONLY valid JSON:
 
 {
   "confidence": "exact|near|not_found",
-  "normalised": "Normalised subject name",
-  "scimago_category": "A name copied verbatim from VALID CATEGORIES, or \"\" if none fits",
-  "scimago_area": "A name copied verbatim from VALID AREAS, or \"\" if none fits",
+  "normalised": "Corrected or normalised user subject",
+  "scimago_category": "A name copied verbatim from VALID CATEGORIES, or empty",
+  "scimago_area": "A name copied verbatim from VALID AREAS, or empty",
   "nearest_categories": ["Up to 4 names copied verbatim from VALID CATEGORIES"],
-  "user_message": "Optional message if a correction was made or nothing fitted"
+  "mapping_type": "exact|synonym|spelling_correction|nearest_category|none",
+  "use_nearest_category": true,
+  "user_message": "A short transparent explanation for the researcher"
 }
 
-"nearest_categories" is REQUIRED whenever confidence is "near" or "not_found".
-It is the fallback the user actually sees, so it does the real work: for an
-input with no home in the taxonomy, list the categories that publish closest
-to it. For "crime" that is Law; Sociology And Political Science; Pathology And
-Forensic Medicine — not an empty list, and not a subject AREA. Copy the names
-verbatim; any name not on the list is discarded silently.
-
-Guidance:
-- "exact"     the user named a listed category, or an unambiguous synonym of one.
-- "near"      you mapped to the closest listed category; say so in user_message.
-- "not_found" nothing in the list covers it. Name the nearest listed categories
-              in user_message so the user can pick one.
-- If the input is misspelled, give the correct spelling in user_message.
-- An area without a category is a weak answer. Only leave scimago_category empty
-  when no category genuinely fits."""
+Decision rules:
+- Exact listed category or area: confidence "exact", mapping_type "exact".
+- Unambiguous common synonym or abbreviation: choose the listed term,
+  confidence "exact" or "near", mapping_type "synonym", and say what was used.
+- Misspelling: correct it, choose the corrected listed category or area,
+  confidence "near", mapping_type "spelling_correction", and show
+  “original -> corrected” in user_message.
+- Meaningful subject with no exact listed category: choose the closest listed
+  category in scimago_category, confidence "near", mapping_type
+  "nearest_category", use_nearest_category true, and say clearly that there is
+  no exact SCImago/Scopus category and which nearest category is being used.
+- Truly unusable input: leave category and area empty, confidence "not_found",
+  mapping_type "none", use_nearest_category false, and explain why.
+- nearest_categories is required for every "near" or "not_found" response and
+  must be ordered from most to least relevant. Include the selected category
+  first when confidence is "near".
+- For a broad real area such as Engineering or Computer Science, scimago_area
+  may be used without a category, but the message must say results are broad.
+
+Examples of semantic nearest mapping:
+- “criminology” may map to “Sociology and Political Science” with alternatives
+  such as “Law” and “Pathology and Forensic Medicine”.
+- “robotics” may map to the closest defensible engineering or AI category based
+  on the wording, while disclosing that Robotics is not an exact listed category.
+
+A plausible but unlisted category is worse than an honest nearest-category
+mapping. Every returned category and area is validated by the server."""
 
 # ── Cover letter ──────────────────────────────────────────────────────────
 COVER_LETTER_PROMPT = """You are ResearchBee, an expert academic writing assistant. Generate a professional cover letter for journal submission. Return ONLY valid JSON:

scimago_index.py

@@ -10,7 +10,9 @@ import csv
 import re
 import os
 import difflib
-from typing import Optional, List, Dict, Set, Tuple
+import math
+from collections import Counter
+from typing import Optional, List, Dict, Set, Tuple, Any
 from utils import norm_issn
 
 _scimago_cache: dict = {}
@@ -91,14 +93,98 @@ def is_real_area(name: str) -> bool:
     return bool(name) and name.strip().lower() in subject_vocabulary()["areas"]
 
 
-def suggest_categories(text: str, limit: int = 12) -> List[str]:
+def _normalise_subject_match_text(value: str, *, strip_qualifier: bool = False) -> str:
+    """Normalise a subject phrase for conservative spelling comparison."""
+    text = str(value or "").lower()
+    if strip_qualifier:
+        text = re.sub(r"\([^)]*\)", " ", text)
+    return " ".join(re.findall(r"[a-z0-9]+", text))
+
+
+def _subject_similarity(query: str, candidate: str) -> Tuple[float, float]:
+    """Return (score, raw_score), lightly discounting qualifier-stripped matches."""
+    q_raw = _normalise_subject_match_text(query)
+    c_raw = _normalise_subject_match_text(candidate)
+    if not q_raw or not c_raw:
+        return 0.0, 0.0
+    raw = difflib.SequenceMatcher(None, q_raw, c_raw).ratio()
+    q_base = _normalise_subject_match_text(query, strip_qualifier=True)
+    c_base = _normalise_subject_match_text(candidate, strip_qualifier=True)
+    base = difflib.SequenceMatcher(None, q_base, c_base).ratio() if q_base and c_base else 0.0
+    return max(raw, base * 0.985), raw
+
+
+def closest_subject_term(text: str, cutoff: float = 0.90) -> Optional[Dict[str, Any]]:
+    """Return a high-confidence spelling repair from the closed vocabulary.
+
+    This is deliberately conservative. It is for typographical repair only,
+    not semantic guessing: ``criminology`` must not become ``immunology`` just
+    because the strings happen to look similar. Semantic nearest-category
+    mapping remains the LLM's job and is constrained to the same vocabulary.
     """
-    Deterministic shortlist of REAL category names for a free-text subject.
-    Exact -> whole-word containment -> difflib. Never invents a name.
+    q = str(text or "").strip()
+    if len(_normalise_subject_match_text(q)) < 4:
+        return None
+
+    ranked: List[Tuple[float, float, str, str]] = []
+    vocab = subject_vocabulary()
+    for kind in ("categories", "areas"):
+        singular = "category" if kind == "categories" else "area"
+        for name in vocab[kind]:
+            score, raw = _subject_similarity(q, name)
+            if score >= cutoff:
+                ranked.append((score, raw, singular, name))
+    if not ranked:
+        return None
+
+    # Prefer the strongest raw phrase match. This makes a misspelt broad area
+    # such as "compter scince" resolve to the real area "computer science"
+    # instead of the category "computer science (miscellaneous)".
+    ranked.sort(key=lambda item: (-item[0], -item[1], item[2] != "area", len(item[3])))
+    best = ranked[0]
+    if len(ranked) > 1:
+        second = ranked[1]
+        best_base = _normalise_subject_match_text(best[3], strip_qualifier=True)
+        second_base = _normalise_subject_match_text(second[3], strip_qualifier=True)
+        # A near-tie between genuinely different terms is not safe enough for
+        # an automatic spelling correction. Category/area variants sharing the
+        # same base phrase are safe and are resolved by the raw-match rule.
+        if best_base != second_base and best[0] - second[0] < 0.025:
+            return None
+    return {
+        "name": best[3],
+        "kind": best[2],
+        "score": round(best[0], 4),
+        "method": "spelling_correction",
+    }
+
+
+_subject_token_df_cache: Optional[Dict[str, float]] = None
+
+
+def _subject_token_weights() -> Dict[str, float]:
+    global _subject_token_df_cache
+    if _subject_token_df_cache is not None:
+        return _subject_token_df_cache
+    categories = list(subject_vocabulary()["categories"])
+    df: Counter[str] = Counter()
+    for name in categories:
+        df.update(_tokens(name))
+    total = max(len(categories), 1)
+    _subject_token_df_cache = {
+        token: math.log((total + 1) / (count + 1)) + 1.0
+        for token, count in df.items()
+    }
+    return _subject_token_df_cache
 
-    This exists so the LLM chooses from a closed list rather than generating
-    one: "Crime and Society" is a plausible SCImago category name and is not
-    a SCImago category at all.
+
+def suggest_categories(text: str, limit: int = 12) -> List[str]:
+    """Return deterministic, real category suggestions for free text.
+
+    High-confidence spelling repairs come first. Remaining suggestions are
+    ranked by weighted whole-token overlap, so a distinctive term such as
+    ``sustainability`` outweighs a generic term such as ``science``. The
+    function never invents a category and never performs semantic guessing.
     """
     vocab = list(subject_vocabulary()["categories"].keys())
     q = (text or "").strip().lower()
@@ -107,27 +193,45 @@ def suggest_categories(text: str, limit: int = 12) -> List[str]:
     if q in vocab:
         return [q]
 
+    out: List[str] = []
+    # Category-only typo candidates. A broad-area spelling correction is
+    # handled by ``closest_subject_term`` in the route itself.
+    close = []
+    for name in vocab:
+        score, _raw = _subject_similarity(q, name)
+        if score >= 0.90:
+            close.append((score, name))
+    close.sort(key=lambda item: (-item[0], len(item[1])))
+    for _score, name in close:
+        if name not in out:
+            out.append(name)
+        if len(out) >= limit:
+            return out
+
     qt = _tokens(q)
-    ranked: List[Tuple[int, str]] = []
+    if not qt:
+        return out[:limit]
+    weights = _subject_token_weights()
+    q_total = sum(weights.get(token, 1.0) for token in qt) or 1.0
+    ranked: List[Tuple[float, str]] = []
     for name in vocab:
         nt = _tokens(name)
-        if not nt:
+        overlap = qt & nt
+        if not overlap:
             continue
-        if qt and qt <= nt:
-            ranked.append((3, name))
-        elif qt & nt:
-            ranked.append((2, name))
-    ranked.sort(key=lambda x: (-x[0], len(x[1])))
-    out = [n for _, n in ranked][:limit]
-
-    # difflib only as a typo repair, and only at a cutoff high enough to be
-    # about spelling rather than about vibes. At 0.72 it offered "immunology"
-    # for "criminology" (ratio 0.76) — lexically close, semantically absurd.
-    # 0.90 still catches "engineerng" -> "engineering" (0.95).
-    if len(out) < limit:
-        for n in difflib.get_close_matches(q, vocab, n=limit - len(out), cutoff=0.90):
-            if n not in out:
-                out.append(n)
+        matched = sum(weights.get(token, 1.0) for token in overlap)
+        coverage = matched / q_total
+        candidate_total = sum(weights.get(token, 1.0) for token in nt) or 1.0
+        precision = matched / candidate_total
+        char_score, _raw = _subject_similarity(q, name)
+        score = 3.0 * coverage + 1.2 * precision + 0.25 * char_score
+        ranked.append((score, name))
+    ranked.sort(key=lambda item: (-item[0], len(item[1]), item[1]))
+    for _score, name in ranked:
+        if name not in out:
+            out.append(name)
+        if len(out) >= limit:
+            break
     return out[:limit]