Initial commit

.env.example

@@ -0,0 +1,15 @@
+# Copy to .env and fill in. NOVA loads this at import via ui/paths.py.
+# On Hugging Face Spaces / any host, set these as secrets instead — there's no
+# .env in the image.
+
+# --- required ---
+GROQ_API_KEY=
+SECOND_GROQ_API_KEY=          # both graphs read this one; Qa.get_llm() does too
+TAVILY_API_KEY=               # langchain-tavily, used to ground the INTENT graph
+
+# --- optional: raise rate limits / unlock more sources ---
+SEMANTIC_SCHOLAR_API_KEY=
+OPENALEX_API_KEY=
+OPENALEX_CONTENT_API_KEY=
+OPENALEX_MAILTO=
+UNPAYWALL_EMAIL=

.gitignore

@@ -0,0 +1,27 @@
+# --- secrets (NEVER commit) ---
+.env
+.env.*
+!.env.example
+
+# --- virtualenvs ---
+NOVA_venv/
+venv/
+.venv/
+
+# --- runtime caches (ephemeral, regenerated at runtime) ---
+vectorstores/
+downloads/
+logs/
+.gradio/
+
+# --- generated at import by ui/sonic.py ---
+assets/sonic.svg
+assets/user.svg
+
+# --- python cruft ---
+__pycache__/
+*.pyc
+*.pyo
+
+# --- misc ---
+.DS_Store

README.md

@@ -0,0 +1,180 @@
+---
+title: NOVA
+emoji: ✨
+colorFrom: purple
+colorTo: indigo
+sdk: gradio
+sdk_version: 6.20.0
+app_file: nova_app.py
+python_version: "3.12"
+pinned: false
+short_description: Frame a research idea, find the papers, chat with them.
+---
+
+# NOVA — Research, guided by SONIC
+
+A Gradio app that stitches two projects together, unchanged:
+
+- **`app/`** — the research pipeline: an INTENT graph that frames your messy idea into
+  Problem / Objective / Additional Context, then a SEARCH + CLUSTER graph that fetches
+  from arXiv + Semantic Scholar + OpenAlex, reranks with SPECTER, and clusters by approach.
+- **`chatbot_core/`** — single-PDF Q&A: `vectorizeer.py` chunks a paper by section and
+  embeds it, `Qa.py` answers over it with page citations.
+
+NOVA is the product. SONIC is the assistant persona that talks you through it.
+
+```
+your raw idea
+   -> INTENT agent frames it
+   -> you review/edit the framing
+   -> SEARCH agent fetches + reranks + clusters
+   -> paper cards, grouped by approach
+   -> "Chat it out" on any card -> PDF fetched, vectorized, Q&A with citations
+```
+
+## Layout
+
+| Path | What it is |
+| --- | --- |
+| `nova_app.py` | The whole UI: stage machine, event wiring, entrypoint |
+| `ui/` | Presentation + glue — theme, SONIC, paper/PDF helpers, model loading |
+| `app/`, `chatbot_core/` | Backend. Imported and driven, **never modified** |
+
+`ui/paths.py` must be imported first — it does the `sys.path` + `chdir` + `.env` wiring
+that makes `import app...` and `from vectorizeer import ...` resolve.
+
+## Run it
+
+```bash
+cd NOVA
+python -m venv .venv && source .venv/bin/activate
+
+# CPU-only box: install the CPU torch wheel FIRST, or pip resolves the CUDA build
+# and drags in ~2.5 GB of nvidia-*/cuda-* wheels you'll never execute. NOVA pins
+# device="cpu" and never asks for a GPU.
+pip install torch --index-url https://download.pytorch.org/whl/cpu
+
+pip install -r requirements.txt
+cp .env.example .env      # then fill it in
+python nova_app.py        # http://localhost:7860
+```
+
+Verified end-to-end on **Python 3.14** with torch 2.13.0+cpu. Anything ≥3.10 works.
+Doing the CPU-torch line first pulls in **zero** nvidia packages and lands at a 2.0 GB
+venv; skip it and you get roughly double that, for a GPU the app never touches.
+
+### Keys
+
+`GROQ_API_KEY`, `SECOND_GROQ_API_KEY` and `TAVILY_API_KEY` are required — both graphs
+read them at *import* time, so a missing key fails the boot splash, not the first search.
+The rest in `.env.example` are optional and only raise rate limits.
+
+## First boot is slow, once
+
+NOVA loads three models totalling ~2 GB:
+
+| Model | Loaded by | Size |
+| --- | --- | --- |
+| `allenai-specter` | the search graph, at import | 440 MB |
+| `BAAI/bge-base-en-v1.5` | the chatbot's embeddings | 438 MB |
+| `BAAI/bge-reranker-base` | the chatbot's cross-encoder | 1.1 GB |
+
+They download from HuggingFace once and cache in `~/.cache/huggingface`. Set `HF_TOKEN`
+to avoid the anonymous rate limit on that first pull.
+
+Only the search graph loads **synchronously** behind the splash — the two chatbot models
+warm on a background thread, since you can't click "Chat it out" until you've framed an
+intent and run a search anyway. If the thread hasn't finished by then, the click simply
+blocks on the same lock rather than loading twice.
+
+## Deploying to Hugging Face Spaces
+
+The YAML front-matter at the top of this file *is* the Space config — `app_file` points
+at `nova_app.py`, so nothing needs renaming. The free CPU tier (16 GB RAM) is the target.
+
+**1. Create the Space.** [huggingface.co/new-space](https://huggingface.co/new-space) →
+SDK **Gradio**, hardware **CPU basic (free)**. Don't initialize it with anything.
+
+**2. Push this folder as the Space's repo root.** `nova_app.py` must land at the top
+level, not inside a `NOVA/` subfolder:
+
+```bash
+cd NOVA
+git init && git branch -M main
+git remote add space https://huggingface.co/spaces/<your-username>/NOVA
+git add -A && git commit -m "NOVA on Gradio"
+git push --force space main
+```
+
+Push over HTTPS and use an [access token](https://huggingface.co/settings/tokens) with
+**write** scope as the password — your account password won't work.
+
+**3. Add the keys** under *Settings → Variables and secrets*, as **Secrets** (not public
+variables): `GROQ_API_KEY`, `SECOND_GROQ_API_KEY`, `TAVILY_API_KEY`. Spaces injects them
+as env vars, which is exactly what the code reads — `load_dotenv()` finding no `.env` is
+fine and expected. Add `HF_TOKEN` too, to lift the anonymous rate limit on the first
+model pull.
+
+Miss a key and the Space still boots, then shows a named error on the splash telling you
+which one — it won't hang.
+
+**4. First boot takes a few minutes** while ~2 GB of weights download. The server binds
+the port *before* loading anything (models load on first visit, via `demo.load`), so the
+Space goes green early and the splash does the waiting.
+
+### Hardware: ZeroGPU or CPU basic
+
+This repo is configured for **ZeroGPU**, and runs unmodified on **CPU basic** too — it
+detects which it's on via `SPACES_ZERO_GPU` and adapts. Note HF only lets free accounts
+pick the free tier *at Space creation*; you can't downgrade into it later.
+
+ZeroGPU imposes two hard rules, and both shape the code:
+
+1. **torch must be 2.11.0 / 2.10.0 / 2.9.1 / 2.8.0**, plain CUDA build — the `+cpu` local
+   version is rejected. Hence the exact pin in `requirements.txt`.
+2. **At least one `@spaces.GPU` function must exist at import**, or the Space dies with
+   *"No @spaces.GPU function detected during startup"*.
+
+A GPU exists **only inside** an `@spaces.GPU` call. Everything outliving that window must
+be CPU-resident, which is why `get_embeddings`, `get_retriever` and the SPECTER model now
+take an explicit `device` (defaulting to `cpu`) instead of auto-detecting. Auto-detect is
+the trap: torch reports a GPU at import on ZeroGPU, so a model would load onto `cuda` and
+then fail on first use out in a LangGraph node.
+
+So exactly one thing runs on the GPU — the bulk chunk embedding in
+[ui/gpu.py](ui/gpu.py), the slowest step in the app. It builds and **persists** the
+vectorstore, then the caller re-opens it on CPU; what crosses the GPU boundary is the
+file on disk, never a cuda-resident object.
+
+### Verified
+
+- `sdk_version: 6.20.0` — the code needs Gradio 6 (`launch()` takes `theme`/`css`/`js`;
+  `Chatbot` dropped `type=`), and 6.20.0 is what Spaces currently serves.
+- `python_version: "3.12"` — the full dependency set resolves cleanly there with
+  `torch==2.11.0`.
+- Off ZeroGPU, `@spaces.GPU` is a transparent no-op, so local runs and CPU hardware are
+  unaffected — the CPU path keeps the threaded progress animation, and only the ZeroGPU
+  path calls the GPU inline (it must run on the caller's thread, not one we spawn).
+
+### Why not Streamlit Community Cloud
+
+That's what this rewrite escapes. It caps at roughly 1 GB of RAM, so ~2 GB of weights got
+the container OOM-killed mid-load; it restarted, re-entered boot, and sat on the splash
+forever. Any host needs **≥ 4 GB RAM**.
+
+## Notes for the next person
+
+- **State.** Streamlit re-ran the script top-to-bottom and branched on
+  `session_state.stage`. Gradio builds the component graph once, so a stage is a
+  `gr.Column` and every handler returns the `visible` flag for all seven. Same state
+  machine, declared instead of re-derived.
+- **The card grid** is inside `@gr.render` because each card's button needs a real
+  handler closing over its paper key. It redraws off `clusters_state.change` — a
+  `gr.State` reassigned to a fresh list, since `gr.render` can't see a dict mutated
+  in place.
+- **Models are process-global**, not per-session, behind a double-checked lock in
+  `ui/agents.py`. Gradio serves from a thread pool, so without the lock two
+  simultaneous first-visitors would each kick off a 2 GB load.
+- **The two graphs share `SECOND_GROQ_API_KEY`** (see `app/modules/search/graph.py`) —
+  `GROQ_API_KEY` is read but only `second_api_key` is actually passed to both `ChatGroq`
+  instances. Pre-existing upstream behaviour, left alone. Set both keys.

app/core/logging_config.py

@@ -0,0 +1,71 @@
+"""
+Per-run logging.
+
+Every user-submitted research query gets one identifier — the `thread_id`
+handed back by POST /intent/start — that flows through both graphs as
+`run_id`: the intent graph's ResearchIntentState carries it from the start,
+and app/modules/search/jobs.py threads the same value into the search
+graph's ResearchSearchState when it kicks off the background job. Every
+node in both graphs, plus the API providers and reranker they call, logs
+through `get_node_logger(run_id, ...)`, so logs/{run_id}.log ends up with
+the complete story for that one input: every node's output, paper counts
+per source, and any failure with its exception, in call order.
+"""
+
+import logging
+from pathlib import Path
+
+LOG_DIR = Path(__file__).resolve().parent.parent.parent / "logs"
+LOG_DIR.mkdir(exist_ok=True)
+
+_FORMATTER = logging.Formatter(
+    fmt="%(asctime)s | %(levelname)-7s | %(name)s | %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+
+_run_loggers: dict[str, logging.Logger] = {}
+
+
+def _safe_filename(run_id: str) -> str:
+    return "".join(c if c.isalnum() or c in "-_" else "_" for c in run_id)
+
+
+def _get_run_logger(run_id: str) -> logging.Logger:
+    """
+    One logger per run_id, writing to logs/{run_id}.log plus the console.
+    propagate=False so records don't also bubble up to the root logger's
+    own console handler and print twice.
+    """
+    if run_id in _run_loggers:
+        return _run_loggers[run_id]
+
+    logger = logging.getLogger(f"run.{run_id}")
+    logger.setLevel(logging.INFO)
+    logger.propagate = False
+
+    # `logging.getLogger` returns the SAME object process-wide, so guard against
+    # re-attaching handlers if this logger was already set up — otherwise a host
+    # that re-executes the module (e.g. Streamlit re-running the script, which can
+    # bypass the _run_loggers cache) stacks duplicate handlers and every line gets
+    # written to the file N times.
+    if not logger.handlers:
+        file_handler = logging.FileHandler(LOG_DIR / f"{_safe_filename(run_id)}.log", encoding="utf-8")
+        file_handler.setFormatter(_FORMATTER)
+        logger.addHandler(file_handler)
+
+        console_handler = logging.StreamHandler()
+        console_handler.setFormatter(_FORMATTER)
+        logger.addHandler(console_handler)
+
+    _run_loggers[run_id] = logger
+    return logger
+
+
+def get_node_logger(run_id: str, node_name: str) -> logging.Logger:
+    """
+    Logger for one node/step within one run, e.g.
+    get_node_logger(run_id, "intent.problem") or
+    get_node_logger(run_id, "search.arxiv"). Records go to
+    logs/{run_id}.log (and the console) via the shared run-level logger.
+    """
+    return _get_run_logger(run_id).getChild(node_name)

app/main.py

@@ -0,0 +1,39 @@
+import logging
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+
+from app.modules.intent.router import router as intent_router
+from app.modules.search.router import router as search_router
+
+# Root config for anything logged outside a specific run (server
+# startup/shutdown, module-level fallback loggers used when a provider
+# function is called standalone rather than from a graph node). Per-request
+# logs go through app.core.logging_config.get_node_logger instead, which
+# writes its own logs/{run_id}.log and does not propagate up to this root
+# handler — so this does not duplicate those.
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s | %(levelname)-7s | %(name)s | %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+
+app = FastAPI(title="Research Assistant Pipeline API")
+
+# Dev-friendly CORS. Lock this down to your actual frontend origin(s) before
+# deploying anywhere real.
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+
+@app.get("/health")
+def health():
+    return {"status": "ok"}
+
+
+app.include_router(intent_router)
+app.include_router(search_router)

app/modules/intent/graph.py

@@ -0,0 +1,308 @@
+from typing import TypedDict
+import os
+from dotenv import load_dotenv
+from langchain_groq import ChatGroq
+from langchain_tavily import TavilySearch
+from langgraph.checkpoint.memory import MemorySaver
+from langgraph.graph import END, START, StateGraph
+from langgraph.types import Command, interrupt
+
+from app.core.logging_config import get_node_logger
+
+load_dotenv()
+
+
+# State definition
+class ResearchIntentState(TypedDict):
+    run_id: str
+    user_query: str
+    problem: str
+    objective: str
+    additional_context: str
+    aggregated_intent: str
+    search_results: str
+    polished_research_intent: str
+    human_verified_intent: str
+
+second_api_key = os.getenv("SECOND_GROQ_API_KEY")
+
+# LLM + tools — instantiated once at import time, reused across requests
+llm = ChatGroq(model="openai/gpt-oss-120b",api_key =second_api_key)
+search_tool = TavilySearch(max_results=5)
+
+
+# Nodes
+def problem_framing(state: ResearchIntentState) -> ResearchIntentState:
+    logger = get_node_logger(state["run_id"], "intent.problem")
+    prompt = f""" You are extracting the PROBLEM STATEMENT from a researcher's raw, informal research idea.
+
+Your only job: describe what is happening — the gap, limitation, failure mode, or open issue in the current state of the field — as understood from the query.
+
+STRICT RULES:
+- Do NOT propose what should be done about it. That is not your job.
+- Do NOT suggest methods, techniques, or solutions, even in passing.
+- Do NOT include constraints, preferences, or scope limits (time range, baselines, excluded approaches). That belongs elsewhere.
+- Write 2-4 sentences, in formal academic register, third person.
+- If the query does not clearly state a problem, infer the most reasonable underlying problem implied by the researcher's framing — do not leave it empty, but do not invent specifics not implied by the query.
+
+Output only the problem statement text. No labels, no preamble.
+
+User's raw query:
+{state['user_query']}
+"""
+    logger.info("Framing problem statement from user query (%d chars)", len(state["user_query"]))
+    try:
+        problem = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while framing the problem statement")
+        raise
+    logger.info("Problem statement produced (%d chars): %s", len(problem), problem[:200].replace("\n", " "))
+    return {"problem": problem}
+
+
+def objective_framing(state: ResearchIntentState) -> ResearchIntentState:
+    logger = get_node_logger(state["run_id"], "intent.objective")
+    prompt = f"""You are extracting the RESEARCH OBJECTIVE from a researcher's raw, informal research idea.
+
+Your only job: describe what the researcher wants to find out, achieve, or resolve — the goal of the research, not the method to get there.
+
+STRICT RULES:
+- Do NOT describe the current problem/gap in the field. That belongs elsewhere.
+- Do NOT propose a specific method, technique, model, or solution as the objective. Phrase the objective in terms of outcome ("identify", "evaluate", "compare", "reduce", "understand"), never in terms of a specific technique to be applied.
+  - Wrong: "Use contrastive learning to reduce hallucinations."
+  - Right: "Identify techniques that reduce hallucinations while preserving answer quality."
+- Do NOT include constraints, preferred methods, or scope limits. That belongs elsewhere.
+- Write 1-3 sentences, in formal academic register, third person.
+
+Output only the objective text. No labels, no preamble.
+
+User's raw query:
+{state['user_query']}
+"""
+    logger.info("Framing research objective from user query")
+    try:
+        objective = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while framing the research objective")
+        raise
+    logger.info("Objective produced (%d chars): %s", len(objective), objective[:200].replace("\n", " "))
+    return {"objective": objective}
+
+
+def additional_context_framing(state: ResearchIntentState) -> ResearchIntentState:
+    logger = get_node_logger(state["run_id"], "intent.context")
+    prompt = f"""You are extracting ADDITIONAL CONTEXT from a researcher's raw, informal research idea.
+
+Your only job: capture optional supporting details the researcher mentioned that are NOT the problem and NOT the objective — for example: preferred methods, known approaches they're already aware of, constraints, application domain, exclusions, time range, baseline models/papers to compare against.
+
+STRICT RULES:
+- Do NOT restate the problem or the objective in different words.
+- Do NOT invent context that isn't implied by the query. It is normal and acceptable for this field to be sparse.
+- If the query genuinely contains no such details, output exactly: "None specified."
+- If details exist, write them as a short list or 1-3 sentences, in formal academic register.
+
+Output only the additional context text (or "None specified."). No labels, no preamble.
+
+User's raw query:
+{state['user_query']}"""
+    logger.info("Extracting additional context from user query")
+    try:
+        additional_context = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while extracting additional context")
+        raise
+    logger.info("Additional context produced (%d chars): %s", len(additional_context), additional_context[:200].replace("\n", " "))
+    return {"additional_context": additional_context}
+
+
+def aggregator(state: ResearchIntentState) -> ResearchIntentState:
+    logger = get_node_logger(state["run_id"], "intent.aggregator")
+    prompt = f"""You are producing the FINAL RESEARCH INTENT document by harmonizing three independently-drafted sections into one coherent whole.
+
+You will receive:
+- PROBLEM: what is currently happening / the gap
+- OBJECTIVE: what the researcher wants to find out or achieve
+- ADDITIONAL_CONTEXT: optional supporting details (may be "None specified.")
+
+Your job is to make these three sections read as if written together, by:
+- Using consistent terminology across all three (e.g., if PROBLEM says "vision-language models" and OBJECTIVE says "VLMs," pick one term and use it consistently)
+- Smoothing awkward phrasing or redundancy between sections
+- Ensuring pronouns and references are unambiguous across sections (e.g., "this issue" in OBJECTIVE should clearly refer back to something named in PROBLEM)
+- Adjusting tone/register so all three sections sound like one voice
+
+STRICT RULES — DO NOT:
+- Do NOT move content between sections. If OBJECTIVE contains a method, do not move it to ADDITIONAL_CONTEXT — leave the section boundary as-is, just smooth the language.
+- Do NOT add new claims, facts, or specifics that were not present in the original three sections.
+- Do NOT remove information for brevity. Every substantive point from all three sections must remain.
+- Do NOT collapse the three sections into a single paragraph. Keep them clearly separated under their own headers.
+- If ADDITIONAL_CONTEXT is "None specified.", keep it exactly as "None specified." in the output — do not fabricate content for it.
+
+Output in exactly this format:
+
+Problem:
+<harmonized problem text>
+
+Objective:
+<harmonized objective text>
+
+Additional Context:
+<harmonized additional context text, or "None specified.">
+
+Do not include any preamble, explanation, or text outside this format.
+
+Inputs:
+PROBLEM:
+{state['problem']}
+
+OBJECTIVE:
+{state['objective']}
+
+ADDITIONAL_CONTEXT:
+{state['additional_context']}"""
+    logger.info("Harmonizing problem/objective/context into one intent document")
+    try:
+        aggregator_intent = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while harmonizing the intent sections")
+        raise
+    logger.info("Aggregated intent produced (%d chars)", len(aggregator_intent))
+    return {"aggregated_intent": aggregator_intent}
+
+
+def web_polish_node(state: ResearchIntentState) -> ResearchIntentState:
+    logger = get_node_logger(state["run_id"], "intent.polish")
+
+    # Step 1: derive a concise search query from the aggregated intent
+    query_gen_prompt = f"""Given this Research Intent, generate ONE concise web search query
+(under 15 words) to find current terminology, related work, or factual grounding
+for the topic. Output only the query text, nothing else.
+
+Research Intent:
+{state['aggregated_intent']}
+"""
+    try:
+        search_query = llm.invoke(query_gen_prompt).content
+    except Exception:
+        logger.exception("LLM call failed while generating the web search query")
+        raise
+    logger.info("Web search query: %s", search_query)
+
+    # Step 2: run the search
+    try:
+        raw_results = search_tool.invoke({"query": search_query})
+    except Exception:
+        logger.exception("Tavily web search failed")
+        raise
+    results = raw_results.get("results", [])
+    logger.info("Web search returned %d result(s)", len(results))
+    results_text = "\n\n".join(
+        f"Source: {r.get('title', 'N/A')}\n{r.get('content', '')}"
+        for r in results
+    )
+
+    # Step 3: polish the intent using search results, with strict boundaries
+    polish_prompt = f"""You are GROUNDING and POLISHING a Research Intent document using web search results.
+
+You will receive the current Research Intent (Problem / Objective / Additional Context)
+and a set of web search results related to its topic.
+
+Your ONLY job:
+- Correct or sharpen field-specific terminology if the current text uses vague or outdated terms
+- Verify and correct named entities (model names, benchmark names, paper titles) if search results
+  show the current text has them wrong or imprecise
+- Improve clarity and precision of wording
+
+STRICT RULES — DO NOT:
+- Do NOT move content between Problem / Objective / Additional Context sections.
+- Do NOT change the meaning, scope, or intent of any section.
+- Do NOT add citations, links, or references to sources in the output text.
+- If the search results are irrelevant or add nothing useful, return the Research Intent unchanged.
+- Preserve the exact three-section format with headers: Problem / Objective / Additional Context.
+
+Output in exactly this format, nothing else:
+
+Problem:
+<text>
+
+Objective:
+<text>
+
+Additional Context:
+<text>
+
+Current Research Intent:
+{state['aggregated_intent']}
+
+Web Search Results:
+{results_text}
+"""
+    try:
+        polished = llm.invoke(polish_prompt)
+    except Exception:
+        logger.exception("LLM call failed while polishing the research intent")
+        raise
+    logger.info("Polished research intent produced (%d chars)", len(polished.content))
+
+    return {
+        "search_results": results_text,
+        "polished_research_intent": polished.content,
+    }
+
+
+def human_review_node(state: ResearchIntentState) -> ResearchIntentState:
+    logger = get_node_logger(state["run_id"], "intent.human_review")
+    logger.info("Pausing for human review of the polished research intent")
+    human_response = interrupt({
+        "polished_research_intent": state["polished_research_intent"],
+        "instruction": "Review and edit this Research Intent.",
+    })
+    logger.info("Human-verified intent received (%d chars)", len(human_response))
+    return {"human_verified_intent": human_response}
+
+
+# Graph building
+builder = StateGraph(ResearchIntentState)
+
+builder.add_node("problem", problem_framing)
+builder.add_node("objective", objective_framing)
+builder.add_node("context", additional_context_framing)
+builder.add_node("human_review", human_review_node)
+builder.add_node("aggregator", aggregator)
+builder.add_node("polish", web_polish_node)
+
+builder.add_edge(START, "problem")
+builder.add_edge(START, "objective")
+builder.add_edge(START, "context")
+builder.add_edge("problem", "aggregator")
+builder.add_edge("objective", "aggregator")
+builder.add_edge("context", "aggregator")
+builder.add_edge("aggregator", "polish")
+builder.add_edge("polish", "human_review")
+builder.add_edge("human_review", END)
+
+# In-process checkpointer — keeps interrupted threads alive between the
+# /intent/start and /intent/resume calls. Lives in this worker's memory only:
+# fine for a single dev server, but state is lost on restart and isn't shared
+# across multiple uvicorn workers. Swap for a SqliteSaver/PostgresSaver if
+# that ever matters.
+checkpointer = MemorySaver()
+
+graph = builder.compile(checkpointer=checkpointer)
+
+
+if __name__ == "__main__":
+    # Manual standalone test — only runs when you execute this file directly
+    # (`python -m app.modules.intent.graph`), never on import.
+    query = """Hi ... well I have been looking into the problem of the driving behaviour in the car following scenarios in this scenarios what needs to be done is I want to analyse the car following behavior for the context of fuel efficicency comparing the human driven vehicle which are not fuel efficient and the RL driven which are excellent in fuel efficiency the comparison is tedious because many variables change at once velocity acceleration relative distance from the vehicle ahead etc so just to compare fuel efficiency is tough. I want to find out the ways in which we can get a good comparing method for both the vehicles"""
+
+    config = {"configurable": {"thread_id": "test-thread-1"}}
+    result = graph.invoke({"user_query": query, "run_id": "standalone-test"}, config=config)
+
+    print("--- PAUSED FOR HUMAN REVIEW ---")
+    print(result["__interrupt__"])
+
+    edited_text = result["__interrupt__"][0].value["polished_research_intent"]
+    final_result = graph.invoke(Command(resume=edited_text), config=config)
+
+    print("\n--- FINAL RESULT ---")
+    print(final_result["human_verified_intent"])

app/modules/intent/router.py

@@ -0,0 +1,77 @@
+import uuid
+
+from fastapi import APIRouter, BackgroundTasks, HTTPException
+from langgraph.types import Command
+
+from app.core.logging_config import get_node_logger
+from app.modules.intent.graph import graph as intent_graph
+from app.modules.intent.schemas import (
+    IntentResumeRequest,
+    IntentResumeResponse,
+    IntentStartRequest,
+    IntentStartResponse,
+)
+from app.modules.search.jobs import create_job, run_search_job
+
+router = APIRouter(prefix="/intent", tags=["intent"])
+
+
+@router.post("/start", response_model=IntentStartResponse)
+def start_intent(payload: IntentStartRequest):
+    """Runs the intent graph's parallel framing + polish nodes. The graph
+    hits human_review_node's interrupt() immediately, so this call always
+    returns the paused state — it never runs to completion on its own."""
+    thread_id = str(uuid.uuid4())
+    config = {"configurable": {"thread_id": thread_id}}
+    logger = get_node_logger(thread_id, "api.intent_start")
+    logger.info("New research query received (%d chars)", len(payload.user_query))
+
+    result = intent_graph.invoke({"user_query": payload.user_query, "run_id": thread_id}, config=config)
+
+    if "__interrupt__" not in result:
+        # Defensive: should never happen given the graph's fixed structure,
+        # but fail loudly rather than return a malformed response.
+        logger.error("Intent graph did not pause for review as expected")
+        raise HTTPException(status_code=500, detail="Intent graph did not pause for review as expected.")
+
+    interrupt_payload = result["__interrupt__"][0].value
+    logger.info("Intent framing complete — paused for human review")
+
+    return IntentStartResponse(
+        thread_id=thread_id,
+        polished_research_intent=interrupt_payload["polished_research_intent"],
+        instruction=interrupt_payload["instruction"],
+    )
+
+
+@router.post("/resume", response_model=IntentResumeResponse)
+def resume_intent(payload: IntentResumeRequest, background_tasks: BackgroundTasks):
+    """Resumes the paused intent graph with the human-edited text, then
+    immediately kicks off the search graph as a background job using the
+    resulting human_verified_intent — no separate /search/start call needed."""
+    config = {"configurable": {"thread_id": payload.thread_id}}
+    logger = get_node_logger(payload.thread_id, "api.intent_resume")
+    logger.info("Resuming intent graph with human-edited text")
+
+    result = intent_graph.invoke(Command(resume=payload.edited_intent), config=config)
+
+    if "human_verified_intent" not in result:
+        logger.error("No paused intent graph found for this thread_id, or it was already resumed")
+        raise HTTPException(
+            status_code=404,
+            detail="No paused intent graph found for this thread_id, or it has already been resumed.",
+        )
+
+    human_verified_intent = result["human_verified_intent"]
+
+    job_id = str(uuid.uuid4())
+    create_job(job_id, payload.thread_id)
+    logger.info("Human-verified intent finalized — starting search job %s", job_id)
+    background_tasks.add_task(run_search_job, job_id, human_verified_intent, payload.thread_id)
+
+    return IntentResumeResponse(
+        thread_id=payload.thread_id,
+        human_verified_intent=human_verified_intent,
+        job_id=job_id,
+        status="running",
+    )

app/modules/intent/schemas.py

@@ -0,0 +1,25 @@
+from typing import Literal
+
+from pydantic import BaseModel
+
+
+class IntentStartRequest(BaseModel):
+    user_query: str
+
+
+class IntentStartResponse(BaseModel):
+    thread_id: str
+    polished_research_intent: str
+    instruction: str
+
+
+class IntentResumeRequest(BaseModel):
+    thread_id: str
+    edited_intent: str
+
+
+class IntentResumeResponse(BaseModel):
+    thread_id: str
+    human_verified_intent: str
+    job_id: str
+    status: Literal["running"]

app/modules/search/graph.py

@@ -0,0 +1,524 @@
+from langchain_groq import ChatGroq
+from langgraph.graph import StateGraph, START, END
+from typing import TypedDict, List, Dict, Literal, Optional
+from pydantic import BaseModel, Field
+from dotenv import load_dotenv
+from app.core.logging_config import get_node_logger
+from app.modules.search.providers.arxiv import search_arxiv
+from app.modules.search.providers.semantic_scholar import search_semantic_scholar
+from app.modules.search.providers.openalex import search_openalex
+from app.modules.search.reranking import rerank_by_relevance
+import re
+import unicodedata
+import os
+
+
+load_dotenv()
+
+second_api_key = os.getenv("SECOND_GROQ_API_KEY")
+api_key = os.getenv("GROQ_API_KEY")
+# Setting up model
+llm = ChatGroq(model = 'openai/gpt-oss-120b',api_key=second_api_key)
+llm_2 = ChatGroq(model='openai/gpt-oss-120b',api_key=second_api_key)
+
+# State of the Agent
+class ResearchSearchState(TypedDict):
+    run_id : str
+    ResearchIntent : str
+    Open_Alex_paper: List[dict]
+    Semantic_Scholar_paper : List[dict]
+    arXiv_paper: List[dict]
+    aggregator : dict
+    reranked_papers : dict
+    clustered_papers : List[dict]
+    # Per-source status so the UI can surface WHY a source came back empty
+    # (e.g. Semantic Scholar rate-limited) instead of it looking silently blank.
+    # Distinct keys per source -> no concurrent-write conflict across the
+    # parallel fetch nodes.
+    arxiv_status : dict
+    semantic_scholar_status : dict
+    open_alex_status : dict
+
+class ClusterItem(BaseModel):
+    cluster_id: int
+    label: str = Field(description="Short, specific approach name")
+    rationale: str = Field(description="2-3 sentence explanation of what unifies this cluster")
+    paper_ids: List[int]
+
+class ClusteringOutput(BaseModel):
+    clusters: List[ClusterItem]
+
+# Defining Nodes
+def arXiv_papers(state: ResearchSearchState)-> ResearchSearchState:
+    logger = get_node_logger(state["run_id"], "search.arxiv")
+    prompt = f"""You are converting a Research Intent into a search query for the arXiv API.
+
+arXiv search is PURELY LEXICAL — it does not understand natural language, intent,
+or reasoning. It only matches literal keyword overlap using field-prefixed terms.
+
+RULES FOR THE QUERY YOU PRODUCE:
+- Use only these field prefixes: ti: (title), abs: (abstract), all: (all fields).
+  Default to all: unless a term is clearly a proper technical name that belongs in a title.
+- Combine terms using ONLY uppercase AND, OR, ANDNOT. No other operators exist.
+- Do NOT nest field prefixes inside parentheses (e.g. all:(ti:(x)) is invalid/ambiguous
+  on arXiv and must be avoided).
+- Keep the structure FLAT: field:term AND field:term AND (term1 OR term2) is fine;
+  deeper nesting is not.
+- Extract only concrete technical noun phrases from the Problem and Objective sections.
+  Do NOT use Additional Context — arXiv cannot reason about constraints, exclusions,
+  or preferences, only literal keyword matches.
+- Multi-word technical phrases should be joined with + between words (arXiv API convention),
+  e.g. reinforcement+learning.
+- Produce 4-6 core technical terms maximum. More terms narrow results without arXiv
+  being able to judge relevance the way a ranked search engine would.
+- Output ONLY the final query string in arXiv API syntax. No explanation, no preamble.
+
+Research Intent:
+{state['ResearchIntent']}"""
+    try:
+        query = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while generating the arXiv query")
+        raise
+    query = query.strip().strip('"').strip("'")
+    logger.info("arXiv query: %s", query)
+    papers = search_arxiv(query=query, logger=logger)
+    logger.info("arXiv returned %d paper(s)", len(papers))
+    return {'arXiv_paper': papers, 'arxiv_status': {"state": "ok", "count": len(papers)}}
+
+def semantic_scholar_papers(state : ResearchSearchState)-> ResearchSearchState:
+    logger = get_node_logger(state["run_id"], "search.semantic_scholar")
+    prompt = f"""You are converting a Research Intent into a search query for the Semantic Scholar Search API. This endpoint matches terms ONLY against each paper's title and abstract.
+
+CRITICAL BEHAVIOR TO UNDERSTAND FIRST:
+This is an EXACT MATCH system, not a semantic/fuzzy search. There is no "close enough."
+Bare words or quoted phrases placed next to each other with no operator between them
+are implicitly ANDed — ALL of them must appear, verbatim, in the same paper's title or
+abstract. This means every additional required term you add narrows the result set
+further and increases the risk of zero results. A SINGLE required phrase that doesn't
+match the field's actual standard terminology (e.g. requiring "human-operated" when
+the literature actually says "human-driven") can zero out the entire query, even if
+every other term in it is correct.
+
+SYNTAX AVAILABLE:
+- Bare word or phrase with no operator: still REQUIRED (ANDed with everything else),
+  not optional. There is no implicit OR between adjacent terms.
+- "exact phrase": double-quote a multi-word phrase that must appear exactly as
+  written, word-for-word. Only use this for terminology you are highly confident
+  is the field's standard phrasing.
+- +term: explicitly required (behaves the same as a bare term — the + is for
+  clarity/emphasis, not a different matching rule).
+- -term: term must NOT appear. Use only for genuine exclusions the researcher
+  explicitly stated.
+- (a | b | c): matches ANY of the listed alternatives — use this to hedge against
+  terminology uncertainty for a single concept.
+- Parentheses group | and + clauses; keep grouping to 1 level deep.
+
+RULES FOR CONSTRUCTING THE QUERY:
+- Identify at most 2 CORE concepts from Problem + Objective that are essential to
+  the research question, and require each with a quoted exact phrase (using + or
+  bare — same effect). Do not require more than 2 concepts — since all required
+  terms are ANDed, each additional one multiplies the risk of zero results.
+- For ANY concept where you are not fully certain of the field's standard exact
+  wording (e.g. how a specific comparison, method, or entity type is typically
+  phrased in academic writing), do NOT lock it to a single quoted phrase. Instead,
+  express it as an OR group of 2-3 plausible standard phrasings, e.g.
+  ("human-driven" | "human-operated" | "manually driven"). This is especially
+  important for adjective/description phrases (how something is characterized),
+  which vary more across papers than core technical nouns do.
+- Use (a | b) groups only for genuine synonymous variants of ONE concept — do not
+  invent alternative concepts, methods, or approaches not present in the Research
+  Intent.
+- Do NOT use - exclusions unless Additional Context explicitly states something
+  should be excluded.
+- Keep the total query under ~15 words worth of terms. Favor 2 required core
+  concepts plus 1 hedged OR group over stacking many required exact phrases.
+- Output ONLY the final query string. No explanation, no preamble.
+
+Research Intent:
+{state['ResearchIntent']}"""
+    try:
+        query = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while generating the Semantic Scholar query")
+        raise
+    query = query.strip().strip('"').strip("'")
+    logger.info("Semantic Scholar query: %s", query)
+    status: dict = {}
+    papers = search_semantic_scholar(query=query, logger=logger, status_out=status)
+    logger.info("Semantic Scholar returned %d paper(s)", len(papers))
+    return {'Semantic_Scholar_paper': papers, 'semantic_scholar_status': status}
+
+def Open_alex_papers(state : ResearchSearchState)-> ResearchSearchState:
+    logger = get_node_logger(state["run_id"], "search.openalex")
+    prompt = f"""You are converting a Research Intent into a search query for the OpenAlex /works
+search endpoint, which searches titles, abstracts, and available fulltext.
+
+CRITICAL BEHAVIOR TO ACCOUNT FOR:
+Words not separated by explicit boolean operators are treated as AND by default.
+This means a long plain-keyword query risks ZERO results if any single term is too
+narrow, because ALL terms must match. You must actively defend against this.
+
+SYNTAX AVAILABLE:
+- Uppercase AND, OR, NOT only (lowercase will not be recognized as operators).
+- "exact phrase" for phrases that must appear together.
+- Parentheses for grouping.
+
+RULES FOR CONSTRUCTING THE QUERY:
+- Identify 2-4 DISTINCT core concepts from Problem + Objective (not more — each
+  AND-ed concept multiplies the risk of zero results).
+- For each core concept, group any synonym/near-synonym terms together with OR
+  inside parentheses, e.g. ("fuel efficiency" OR "fuel consumption").
+- Join the distinct concept groups with AND.
+- Use quotes around every multi-word phrase — unquoted multi-word phrases will be
+  split into individual AND-ed words, which is almost always too strict.
+- Do NOT use NOT unless Additional Context explicitly names something to exclude.
+- Do NOT add synonyms or terms not implied by the Research Intent, even to "help"
+  recall — only group terms that are genuinely equivalent phrasings of the same idea.
+- Output ONLY the final query string. No explanation, no preamble.
+
+Research Intent:
+{state['ResearchIntent']}"""
+    try:
+        query = llm.invoke(prompt).content
+    except Exception:
+        logger.exception("LLM call failed while generating the OpenAlex query")
+        raise
+    query = query.strip().strip('"').strip("'")
+    logger.info("OpenAlex query: %s", query)
+    status: dict = {}
+    papers = search_openalex(query=query, logger=logger, status_out=status)
+    logger.info("OpenAlex returned %d paper(s)", len(papers))
+    return {'Open_Alex_paper': papers, 'open_alex_status': status}
+
+def normalize_title(title: str) -> str:
+    """
+    Produce a normalized string used ONLY for matching titles across sources.
+    Never stored on the paper record itself — purely a dedup key.
+    """
+    if not title:
+        return ""
+
+    # Normalize unicode (curly quotes, accented chars) to a comparable ASCII-ish form
+    text = unicodedata.normalize("NFKD", title)
+    text = "".join(c for c in text if not unicodedata.combining(c))
+
+    text = text.lower()
+    text = re.sub(r"[^\w\s]", "", text)   # strip punctuation (colons, hyphens, periods, etc.)
+    text = re.sub(r"\s+", " ", text)      # collapse whitespace
+    return text.strip()
+
+def merge_paper_entry(existing: dict, new: dict) -> dict:
+    """
+    Merge a newly-encountered paper into an existing aggregated entry
+    for the same normalized title. No information is dropped —
+    source-varying fields (source, citation_count, url, pdf_url)
+    are kept per-source in dicts.
+    """
+    new_source = new.get("source")
+
+    # --- title: keep the longer/more complete one ---
+    existing_title = existing.get("title") or ""
+    new_title = new.get("title") or ""
+    merged_title = new_title if len(new_title) > len(existing_title) else existing_title
+
+    # --- authors: union, order-preserving ---
+    merged_authors = list(existing.get("authors") or [])
+    for author in (new.get("authors") or []):
+        if author not in merged_authors:
+            merged_authors.append(author)
+
+    # --- abstract: keep the longer one ---
+    existing_abstract = existing.get("abstract")
+    new_abstract = new.get("abstract")
+    if existing_abstract and new_abstract:
+        merged_abstract = existing_abstract if len(existing_abstract) >= len(new_abstract) else new_abstract
+    else:
+        merged_abstract = existing_abstract or new_abstract
+
+    # --- year: first non-null wins (should agree across sources) ---
+    merged_year = existing.get("year") if existing.get("year") is not None else new.get("year")
+
+    # --- source-varying fields: dict keyed by source, nothing overwritten ---
+    merged_citation_count = dict(existing.get("citation_count") or {})
+    merged_citation_count[new_source] = new.get("citation_count")
+
+    merged_url = dict(existing.get("url") or {})
+    merged_url[new_source] = new.get("url")
+
+    merged_pdf_url = dict(existing.get("pdf_url") or {})
+    merged_pdf_url[new_source] = new.get("pdf_url")
+
+    merged_sources = list(existing.get("source") or [])
+    if new_source not in merged_sources:
+        merged_sources.append(new_source)
+
+    return {
+        "title": merged_title,
+        "authors": merged_authors or None,
+        "abstract": merged_abstract,
+        "year": merged_year,
+        "source": merged_sources,
+        "citation_count": merged_citation_count,
+        "url": merged_url,
+        "pdf_url": merged_pdf_url,
+    }
+
+def aggregator_node(state: ResearchSearchState) -> ResearchSearchState:
+    logger = get_node_logger(state["run_id"], "search.aggregate")
+    arxiv_papers = state.get("arXiv_paper", []) or []
+    openalex_papers = state.get("Open_Alex_paper", []) or []
+    semantic_scholar_papers_list = state.get("Semantic_Scholar_paper", []) or []
+    all_papers = arxiv_papers + openalex_papers + semantic_scholar_papers_list
+
+    aggregated: dict[str, dict] = {}
+
+    for paper in all_papers:
+        key = normalize_title(paper.get("title"))
+        if not key:
+            continue
+
+        if key in aggregated:
+            aggregated[key] = merge_paper_entry(aggregated[key], paper)
+        else:
+            # first time seeing this title — wrap single-source fields
+            # into the same dict-keyed-by-source shape used by merges,
+            # so every entry has a UNIFORM shape whether it came from
+            # 1 source or all 3
+            source = paper.get("source")
+            aggregated[key] = {
+                "title": paper.get("title"),
+                "authors": paper.get("authors"),
+                "abstract": paper.get("abstract"),
+                "year": paper.get("year"),
+                "source": [source],
+                "citation_count": {source: paper.get("citation_count")},
+                "url": {source: paper.get("url")},
+                "pdf_url": {source: paper.get("pdf_url")},
+            }
+
+    logger.info(
+        "Aggregated %d raw records (arXiv=%d, OpenAlex=%d, SemanticScholar=%d) into %d unique papers after dedup",
+        len(all_papers), len(arxiv_papers), len(openalex_papers), len(semantic_scholar_papers_list), len(aggregated),
+    )
+    return {"aggregator": aggregated}
+
+def reranker_node(state: ResearchSearchState) -> ResearchSearchState:
+    """
+    Reranks the aggregated papers by semantic similarity to the Research Intent,
+    keeps the top 15, and returns the FULL records (not just title:abstract)
+    for those top 15 — in the same order the reranker ranked them.
+    """
+    logger = get_node_logger(state["run_id"], "search.rerank")
+    aggregator = state["aggregator"]
+
+    # --- Build the {normalized_title: abstract} view the reranker expects ---
+    papers_for_rerank: dict[str, str] = {
+        normalized_title: record.get("abstract")
+        for normalized_title, record in aggregator.items()
+    }
+    logger.info("Reranking %d aggregated papers by semantic similarity to the intent", len(papers_for_rerank))
+    # --- Run the rerank ---
+    top_papers = rerank_by_relevance(
+        research_intent=state["ResearchIntent"],
+        papers=papers_for_rerank,
+        top_n=15,
+        logger=logger,
+    )
+
+    # --- Rebuild full records for only the titles that survived reranking,
+    #     preserving the ranked order (top_papers is an ordered dict) ---
+    reranked_full: dict[str, dict] = {}
+    for normalized_title in top_papers:
+        record = aggregator.get(normalized_title)
+        if record is not None:
+            reranked_full[normalized_title] = record
+
+    dropped_count = len(aggregator) - len(reranked_full)
+    logger.info(
+        "Kept top %d of %d papers after semantic reranking (%d dropped)",
+        len(reranked_full), len(aggregator), dropped_count,
+    )
+
+    return {"reranked_papers": reranked_full}
+
+def build_clustered_output(
+    clustering_result: ClusteringOutput,
+    id_to_title: dict[int, str],
+    aggregator: dict,
+) -> list[dict]:
+    """
+    Converts {cluster_id, label, rationale, paper_ids} clusters into:
+    [{"label": ..., "rationale": ..., "papers": {normalized_title: full_aggregator_record, ...}}, ...]
+    """
+    output = []
+
+    for cluster in clustering_result.clusters:
+        papers_in_cluster = {}
+
+        for pid in cluster.paper_ids:
+            normalized_title = id_to_title.get(pid)
+            if normalized_title is None:
+                continue  # LLM hallucinated an ID that was never sent — skip, don't crash
+
+            record = aggregator.get(normalized_title)
+            if record is None:
+                continue  # defensive: shouldn't happen if id_to_title is built correctly
+
+            papers_in_cluster[normalized_title] = record
+
+        output.append({
+            "label": cluster.label,
+            "rationale": cluster.rationale,
+            "papers": papers_in_cluster,
+        })
+
+    return output
+
+def prepare_papers_for_clustering(aggregator: dict) -> tuple[list[dict], dict[int, str]]:
+    """
+    Convert the aggregator dict into the (id, title, abstract) form needed for the
+    clustering prompt, plus an id -> normalized_title map to reverse-lookup results.
+
+    Papers with no abstract are excluded from clustering input (title alone is too
+    weak a signal for approach-based grouping) but returned separately so they can
+    still be surfaced to the user as "not clustered" rather than silently dropped.
+    """
+    clustering_input = []
+    id_to_title: dict[int, str] = {}
+    excluded_no_abstract: list[str] = []
+
+    paper_id = 0
+    for normalized_title, record in aggregator.items():
+        abstract = record.get("abstract")
+
+        if not abstract or not abstract.strip():
+            excluded_no_abstract.append(normalized_title)
+            continue
+
+        clustering_input.append({
+            "id": paper_id,
+            "title": record.get("title") or normalized_title,
+            "abstract": abstract,
+        })
+        id_to_title[paper_id] = normalized_title
+        paper_id += 1
+
+    return clustering_input, id_to_title
+
+def clustering_node(state: ResearchSearchState) -> ResearchSearchState:
+    logger = get_node_logger(state["run_id"], "search.cluster")
+    clustering_input , id_to_title = prepare_papers_for_clustering(state['reranked_papers'])
+    logger.info("Clustering %d paper(s) with usable abstracts into approach groups", len(clustering_input))
+
+    prompt = f"""You are helping a researcher make sense of a set of retrieved papers by organizing
+them into methodologically distinct APPROACH CLUSTERS.
+
+You will receive:
+1. A RESEARCH INTENT (Problem / Objective / Additional Context) — this defines what
+   the researcher is actually trying to figure out.
+2. A numbered list of papers, each with a title and abstract.
+
+YOUR GOAL:
+All these papers were retrieved because they already share a common TOPIC with the
+Research Intent — that overlap is not useful information on its own. Your job is to
+find the more useful signal underneath: the DIFFERENT APPROACHES, METHODS, or
+STRATEGIES these papers take toward addressing that shared topic. Think of each
+cluster as answering the question: "If the researcher wanted to pursue THIS
+direction, which papers would be their starting point?"
+
+USE THE RESEARCH INTENT TO GUIDE WHAT COUNTS AS A MEANINGFUL DISTINCTION:
+- The Objective tells you what decision or comparison the researcher actually cares
+  about — let it sharpen which methodological differences matter versus which are
+  noise. If the Objective is about comparing two paradigms, treat "which paradigm
+  does this paper represent" as a primary clustering axis, not an incidental detail.
+- The Additional Context may reveal known approaches the researcher is already aware
+  of, or constraints (like avoiding full retraining) — clusters that map onto these
+  should be called out explicitly, since they connect directly to something the
+  researcher already flagged as relevant.
+
+FOR EACH CLUSTER, PRODUCE:
+- cluster_id: a sequential integer starting at 0, unique per cluster.
+- label: a short, specific name for the actual approach (not a vague topic word) —
+  e.g. "Reinforcement-learning-based car-following control policies," not "RL papers."
+- rationale: 2-3 sentences explaining WHAT unifies these papers methodologically,
+  and WHY this represents a distinct direction the researcher could pursue or
+  compare against, relative to the other clusters.
+- paper_ids: the list of paper IDs belonging to this cluster.
+
+SIZING THE CLUSTERS:
+- Aim for 4-7 clusters when the paper count comfortably supports that many distinct
+  methodological groupings. This is a target, not a quota — with a small paper set,
+  fewer, larger clusters are correct; do not manufacture extra clusters just to hit
+  the range, and do not force a paper into a cluster it doesn't methodologically fit.
+- If two clusters would end up nearly identical in meaning, merge them instead of
+  keeping both.
+- If some papers do not fit coherently into any clear methodological group, place
+  them together in one final cluster labeled "Other / Mixed Approaches" rather than
+  forcing a false fit elsewhere. One honest catch-all beats several padded clusters.
+
+HARD CONSTRAINTS:
+- Every paper ID in the input must appear in exactly one cluster, including the
+  catch-all if used. None may be omitted or duplicated across clusters.
+- Do not invent methods, results, or claims not supported by the given title/abstract
+  text — ground every rationale only in what's actually stated.
+
+RESEARCH INTENT:
+{state["ResearchIntent"]}
+
+PAPERS:
+{clustering_input}
+"""
+
+    structured_llm = llm_2.with_structured_output(ClusteringOutput)
+
+    try:
+        clustering_result = structured_llm.invoke(prompt)
+    except Exception as e:
+        logger.exception("Structured LLM invocation failed during clustering: %s: %s", type(e).__name__, e)
+        return {"clustered_papers": []}
+
+    clustered = build_clustered_output(clustering_result, id_to_title, state["aggregator"])
+    total_clustered_papers = sum(len(c["papers"]) for c in clustered)
+    logger.info("Clustering produced %d cluster(s) covering %d paper(s)", len(clustered), total_clustered_papers)
+
+    return {"clustered_papers": clustered}
+
+# Building The Graph
+builder = StateGraph(ResearchSearchState)
+
+builder.add_node('arxiv',arXiv_papers)
+builder.add_node('semantic_scholar',semantic_scholar_papers)
+builder.add_node('open_alex',Open_alex_papers)
+builder.add_node('aggregation',aggregator_node)
+builder.add_node('reranker', reranker_node)
+builder.add_node('clustering',clustering_node)
+
+builder.add_edge(START,'arxiv')
+builder.add_edge(START,'semantic_scholar')
+builder.add_edge(START,'open_alex')
+builder.add_edge('arxiv','aggregation')
+builder.add_edge('semantic_scholar','aggregation')
+builder.add_edge('open_alex','aggregation')
+builder.add_edge('aggregation','reranker')
+builder.add_edge('reranker','clustering')
+builder.add_edge('clustering',END)
+
+graph = builder.compile()
+
+if __name__ == "__main__":
+    # Manual standalone test — only runs when you execute this file directly
+    # (`python -m app.modules.search.graph`), never on import.
+    input_research_intent_5 = """Problem:
+Large language models frequently produce fluent, confident-sounding answers that contain factual errors, and existing hallucination-detection methods disagree substantially with each other when applied to the same model outputs, making it unclear which detection approach should be trusted for deployment decisions.
+
+Objective:
+Identify hallucination-detection methods for large language models that show consistent, reliable performance across different model families and task types, rather than being effective only in the narrow setting they were originally evaluated on.
+
+Additional Context:
+- Focus on post-hoc detection methods applied to generated text, not training-time interventions like RLHF.
+- Known approaches the researcher is aware of: self-consistency sampling, retrieval-based fact verification, and uncertainty/entropy-based methods.
+- Interested primarily in open-domain question answering and summarization tasks, not code generation."""
+
+    result = graph.invoke({"ResearchIntent": input_research_intent_5, "run_id": "standalone-test"})
+    print(result["clustered_papers"])

app/modules/search/jobs.py

@@ -0,0 +1,54 @@
+"""
+In-memory job tracking for the search graph.
+
+The search graph (3 external APIs + a local SPECTER rerank pass + a
+structured-output clustering call) can take tens of seconds, so it runs as a
+FastAPI BackgroundTask rather than inline in a request. Progress is tracked
+in this plain dict, which the client polls via GET /search/status/{job_id}.
+
+Import note: `search_graph` is imported at module level (not inside the
+function) so that importing this module — which happens once, at FastAPI
+startup — is what triggers loading the SPECTER embedding model in
+app/modules/search/reranking.py. That way the multi-second model load
+happens once at boot, not on the first user's search request.
+
+Limitation to be aware of: this dict lives in a single Python process's
+memory. It resets on server restart and is NOT shared across multiple
+uvicorn workers (`--workers N`). Fine for a single dev/demo server; swap for
+Redis or a DB-backed job table if this ever needs to run with more than one
+worker process.
+"""
+
+from typing import Optional
+
+from app.core.logging_config import get_node_logger
+from app.modules.search.graph import graph as search_graph
+
+JOBS: dict[str, dict] = {}
+
+
+def create_job(job_id: str, run_id: str) -> None:
+    JOBS[job_id] = {"job_id": job_id, "status": "running", "clustered_papers": None, "error": None}
+    get_node_logger(run_id, "job").info("Search job %s created", job_id)
+
+
+def get_job(job_id: str) -> Optional[dict]:
+    return JOBS.get(job_id)
+
+
+def run_search_job(job_id: str, research_intent: str, run_id: str) -> None:
+    """Runs the search graph synchronously. Called via BackgroundTasks,
+    which executes sync callables in a threadpool — so this does not block
+    the event loop for other requests while it runs."""
+    logger = get_node_logger(run_id, "job")
+    logger.info("Search job %s started", job_id)
+    try:
+        result = search_graph.invoke({"ResearchIntent": research_intent, "run_id": run_id})
+        clustered_papers = result["clustered_papers"]
+        JOBS[job_id]["status"] = "done"
+        JOBS[job_id]["clustered_papers"] = clustered_papers
+        logger.info("Search job %s completed: %d cluster(s) produced", job_id, len(clustered_papers or []))
+    except Exception as e:
+        JOBS[job_id]["status"] = "error"
+        JOBS[job_id]["error"] = f"{type(e).__name__}: {e}"
+        logger.exception("Search job %s failed", job_id)

app/modules/search/providers/arxiv.py

@@ -0,0 +1,99 @@
+import logging
+import xml.etree.ElementTree as ET
+from typing import List, Optional
+
+import requests
+
+from app.modules.search.providers.models import ResearchPaper
+
+_fallback_logger = logging.getLogger(__name__)
+
+
+def search_arxiv(query: str, max_results: int = 10, logger: Optional[logging.Logger] = None) -> List[dict]:
+    """
+    Search papers from arXiv. Returns a list of validated paper dicts.
+    """
+    log = logger or _fallback_logger
+    log.info("Searching arXiv for query: %s", query)
+    query = query.replace(" ", "+")
+
+    url = (
+        f"https://export.arxiv.org/api/query?"
+        f"search_query={query}"
+        f"&start=0"
+        f"&max_results={max_results}"
+        f"&sortBy=relevance"
+        f"&sortOrder=descending"
+    )
+
+    try:
+        response = requests.get(url, timeout=15)
+    except requests.RequestException as e:
+        log.warning("arXiv request failed: %s: %s", type(e).__name__, e)
+        return []
+
+    if response.status_code != 200:
+        log.warning("arXiv returned bad status: %s", response.status_code)
+        return []
+
+    try:
+        root = ET.fromstring(response.content)
+    except ET.ParseError as e:
+        log.warning("arXiv response failed to parse as XML: %s", e)
+        return []
+
+    namespace = {"atom": "http://www.w3.org/2005/Atom"}
+    entries = root.findall("atom:entry", namespace)
+
+    papers = []
+
+    for entry in entries:
+        title_el = entry.find("atom:title", namespace)
+        if title_el is None or not title_el.text or not title_el.text.strip():
+            continue
+        title = title_el.text.strip()
+
+        summary_el = entry.find("atom:summary", namespace)
+        abstract = summary_el.text.strip() if summary_el is not None and summary_el.text else None
+
+        authors = []
+        for author_el in entry.findall("atom:author", namespace):
+            name_el = author_el.find("atom:name", namespace)
+            if name_el is not None and name_el.text:
+                authors.append(name_el.text.strip())
+
+        published_el = entry.find("atom:published", namespace)
+        year = None
+        if published_el is not None and published_el.text and len(published_el.text) >= 4:
+            try:
+                year = int(published_el.text[:4])
+            except ValueError:
+                year = None
+
+        pdf_link = None
+        for link in entry.findall("atom:link", namespace):
+            if link.attrib.get("title") == "pdf":
+                pdf_link = link.attrib.get("href")
+                break
+
+        id_el = entry.find("atom:id", namespace)
+        url_val = id_el.text.strip() if id_el is not None and id_el.text else None
+
+        try:
+            paper = ResearchPaper(
+                title=title,
+                authors=authors if authors else None,
+                abstract=abstract,
+                year=year,
+                citation_count=None,  # arXiv never provides this
+                url=url_val,
+                pdf_url=pdf_link,
+                source="arXiv",
+            )
+            papers.append(paper.model_dump())
+        except Exception as e:
+            log.warning("Skipped malformed arXiv entry: %s", e)
+            continue  # skip malformed entries rather than crash the whole batch
+
+    log.info("arXiv: %d paper(s) retrieved", len(papers))
+    return papers

app/modules/search/providers/models.py

@@ -0,0 +1,16 @@
+from typing import List, Literal, Optional
+
+from pydantic import BaseModel, Field
+
+
+class ResearchPaper(BaseModel):
+    title: str = Field(description="The title of the paper")
+    authors: Optional[List[str]] = Field(description="The list of the authors of the paper mentioned.", default=None)
+    abstract: Optional[str] = Field(description="The abstract or the summary of the mentioned research paper.", default=None)
+    year: Optional[int] = Field(description="The year in which the paper was published, leave if not mentioned", default=None)
+    citation_count: Optional[int] = Field(description="The citation count of the paper, leave if not mentioned", default=None)
+    url: Optional[str] = Field(description="The url of the research given on the source", default=None)
+    pdf_url: Optional[str] = Field(description="The pdf url of the research paper if mentioned", default=None)
+    source: Literal["arXiv", "SemanticScholar", "OpenAlex"] = Field(
+        description="The source from which it was extracted 'arXiv' , 'SemanticScholar' or 'OpenAlex'"
+    )

app/modules/search/providers/openalex.py

@@ -0,0 +1,397 @@
+import logging
+import os
+import re
+from concurrent.futures import ThreadPoolExecutor
+from typing import List, Optional
+
+import requests
+
+from app.modules.search.providers.models import ResearchPaper
+
+_fallback_logger = logging.getLogger(__name__)
+
+CONTACT_EMAIL = os.getenv("OPENALEX_MAILTO") or os.getenv("UNPAYWALL_EMAIL")
+OPENALEX_API_KEY = os.getenv("OPENALEX_API_KEY")
+OPENALEX_CONTENT_API_KEY = os.getenv("OPENALEX_CONTENT_API_KEY") or OPENALEX_API_KEY
+
+_REQUEST_HEADERS = {
+    "User-Agent": "Mozilla/5.0 (compatible; research-pipeline/1.0; +mailto:contact@example.com)"
+}
+_PDF_MAGIC = b"%PDF"
+_CITATION_PDF_URL_RE = re.compile(
+    r'<meta[^>]+name=["\']citation_pdf_url["\'][^>]+content=["\']([^"\']+)["\']'
+    r'|<meta[^>]+content=["\']([^"\']+)["\'][^>]+name=["\']citation_pdf_url["\']',
+    re.IGNORECASE,
+)
+_ARXIV_ID_RE = re.compile(r'arxiv\.org/(?:abs|pdf)/([^\s/?#]+)', re.IGNORECASE)
+_ACL_ID_RE = re.compile(r'aclanthology\.org/([^\s/?#"\']+)', re.IGNORECASE)
+_PMC_ID_RE = re.compile(r'(PMC\d+)', re.IGNORECASE)
+
+
+def decode_abstract(inverted_index):
+    """
+    Convert OpenAlex's abstract_inverted_index into a readable abstract.
+    Returns None if the index is missing or empty.
+    """
+    if not inverted_index:  # handles both None and empty dict {}
+        return None
+
+    try:
+        max_position = max(
+            pos for positions in inverted_index.values() for pos in positions
+        )
+    except ValueError:
+        # inverted_index had keys but all position lists were empty
+        return None
+
+    words = [""] * (max_position + 1)
+    for word, positions in inverted_index.items():
+        for position in positions:
+            words[position] = word
+
+    result = " ".join(words).strip()
+    return result if result else None
+
+
+# --------------------------------------------------------------------------
+# PDF resolution / verification helpers
+#
+# OpenAlex's primary_location.pdf_url is often null even when a free copy
+# exists elsewhere (it's the version closest to the record, frequently the
+# paywalled publisher copy). Even best_oa_location.pdf_url can legitimately
+# be null while a landing page is known. So we walk every location OpenAlex
+# knows about, try deterministic direct-PDF reconstruction for arXiv / ACL
+# Anthology / PubMedCentral repos, fall back to scraping the citation_pdf_url
+# meta tag, then DOI -> Unpaywall, and VERIFY every candidate by checking the
+# real response body for the "%PDF" magic number rather than trusting a
+# Content-Type header or a field name.
+# --------------------------------------------------------------------------
+
+def _is_verified_pdf(url: str, timeout: int = 8) -> bool:
+    if not url:
+        return False
+    try:
+        with requests.get(
+            url, headers=_REQUEST_HEADERS, stream=True, timeout=timeout, allow_redirects=True
+        ) as resp:
+            if resp.status_code != 200:
+                return False
+            chunk = next(resp.iter_content(chunk_size=16), b"")
+            return chunk.startswith(_PDF_MAGIC)
+    except requests.RequestException:
+        return False
+
+
+def _extract_citation_pdf_url(landing_page_url: str, timeout: int = 8) -> Optional[str]:
+    """Scrape the citation_pdf_url meta tag off an HTML landing page."""
+    try:
+        resp = requests.get(landing_page_url, headers=_REQUEST_HEADERS, timeout=timeout)
+        if resp.status_code != 200:
+            return None
+        match = _CITATION_PDF_URL_RE.search(resp.text[:20000])
+        if not match:
+            return None
+        return match.group(1) or match.group(2)
+    except requests.RequestException:
+        return None
+
+
+def _resolve_pmc_pdf(pmc_id: str, timeout: int = 8) -> Optional[str]:
+    """Try Europe PMC's render endpoint for a PubMedCentral ID."""
+    clean_id = pmc_id if pmc_id.upper().startswith("PMC") else f"PMC{pmc_id}"
+    candidate = f"https://europepmc.org/articles/{clean_id}?pdf=render"
+    return candidate if _is_verified_pdf(candidate, timeout=timeout) else None
+
+
+def _resolve_unpaywall_pdf(doi: str, email: str, timeout: int = 8) -> Optional[str]:
+    """Last-resort fallback: ask Unpaywall for the best OA location's direct PDF."""
+    try:
+        resp = requests.get(
+            f"https://api.unpaywall.org/v2/{doi}", params={"email": email}, timeout=timeout
+        )
+        if resp.status_code != 200:
+            return None
+        best_location = (resp.json() or {}).get("best_oa_location") or {}
+        return best_location.get("url_for_pdf")
+    except (requests.RequestException, ValueError):
+        return None
+
+
+def _scan_locations_for_repo_ids(locations: List[dict]) -> dict:
+    """
+    Walk every location OpenAlex knows about for this work and pull out
+    arXiv / ACL Anthology / PubMedCentral identifiers wherever they show up,
+    so a deterministic direct-PDF link can be reconstructed even when
+    best_oa_location isn't the repo that actually has one.
+    """
+    found = {}
+    for loc in locations:
+        landing = (loc or {}).get("landing_page_url") or ""
+        pdf = (loc or {}).get("pdf_url") or ""
+        haystack = f"{landing} {pdf}"
+
+        if "arxiv" not in found:
+            m = _ARXIV_ID_RE.search(haystack)
+            if m:
+                found["arxiv"] = m.group(1)
+
+        if "acl" not in found:
+            m = _ACL_ID_RE.search(haystack)
+            if m:
+                acl_id = m.group(1)
+                if acl_id.lower().endswith(".pdf"):
+                    acl_id = acl_id[:-4]
+                found["acl"] = acl_id
+
+        if "pmc" not in found:
+            m = _PMC_ID_RE.search(haystack)
+            if m:
+                found["pmc"] = m.group(1)
+
+    return found
+
+
+def resolve_openalex_pdf(
+    paper: dict,
+    unpaywall_email: Optional[str] = None,
+    use_paid_content_api: bool = False,
+    openalex_content_api_key: Optional[str] = None,
+    log: Optional[logging.Logger] = None,
+) -> Optional[str]:
+    """Runs the full escalation ladder and returns a VERIFIED direct PDF url, or None."""
+    log = log or _fallback_logger
+    primary = paper.get("primary_location") or {}
+    best_oa = paper.get("best_oa_location") or {}
+    locations = paper.get("locations") or []
+    all_locations = [primary, best_oa] + locations
+
+    # 1. Any pdf_url OpenAlex already gave us directly, wherever it's hiding.
+    seen = set()
+    direct_candidates = []
+    for loc in all_locations:
+        u = (loc or {}).get("pdf_url")
+        if u and u not in seen:
+            seen.add(u)
+            direct_candidates.append(u)
+
+    for candidate in direct_candidates:
+        if _is_verified_pdf(candidate):
+            return candidate
+
+    # 2. Deterministic reconstruction from repo IDs found anywhere in locations.
+    repo_ids = _scan_locations_for_repo_ids(all_locations)
+
+    if repo_ids.get("arxiv"):
+        candidate = f"https://arxiv.org/pdf/{repo_ids['arxiv']}"
+        if _is_verified_pdf(candidate):
+            return candidate
+
+    if repo_ids.get("acl"):
+        candidate = f"https://aclanthology.org/{repo_ids['acl']}.pdf"
+        if _is_verified_pdf(candidate):
+            return candidate
+
+    if repo_ids.get("pmc"):
+        candidate = _resolve_pmc_pdf(repo_ids["pmc"])
+        if candidate:
+            return candidate
+
+    # 3. Scrape citation_pdf_url off whatever landing pages we have (capped
+    #    at 3 to bound worst-case latency per paper).
+    seen = set()
+    landing_pages = []
+    for loc in [best_oa, primary] + locations:
+        lp = (loc or {}).get("landing_page_url")
+        if lp and lp not in seen:
+            seen.add(lp)
+            landing_pages.append(lp)
+
+    for page in landing_pages[:3]:
+        scraped = _extract_citation_pdf_url(page)
+        if scraped and _is_verified_pdf(scraped):
+            return scraped
+
+    # 4. DOI -> Unpaywall, only if we have a contact email to use.
+    doi = paper.get("doi")
+    if doi and unpaywall_email:
+        doi_clean = doi.replace("https://doi.org/", "").replace("http://doi.org/", "")
+        candidate = _resolve_unpaywall_pdf(doi_clean, unpaywall_email)
+        if candidate and _is_verified_pdf(candidate):
+            return candidate
+
+    # 5. Opt-in, PAID last resort: OpenAlex's own hosted content API.
+    has_content = paper.get("has_content") or {}
+    if use_paid_content_api and has_content.get("pdf") and openalex_content_api_key:
+        work_id = (paper.get("id") or "").rstrip("/").split("/")[-1]
+        if work_id:
+            candidate = (
+                f"https://content.openalex.org/works/{work_id}.pdf"
+                f"?api_key={openalex_content_api_key}"
+            )
+            if _is_verified_pdf(candidate):
+                return candidate
+
+    log.debug("No verified PDF resolved for OpenAlex work %s", paper.get("id"))
+    return None
+
+
+def resolve_openalex_pdf_fast(paper: dict) -> Optional[str]:
+    """
+    HYBRID fast path: a cheap, ZERO-network best-effort PDF link.
+
+    Takes any direct pdf_url OpenAlex already handed us, or rebuilds a
+    deterministic repo URL (arXiv / ACL / PubMedCentral) from ids found in the
+    work's locations — WITHOUT downloading or verifying anything. The expensive
+    verify + landing-page scrape + Unpaywall ladder is deferred to 'Chat it out'
+    time (see resolve_openalex_pdf), so search stays fast.
+    """
+    primary = paper.get("primary_location") or {}
+    best_oa = paper.get("best_oa_location") or {}
+    locations = paper.get("locations") or []
+    all_locations = [best_oa, primary] + locations
+
+    for loc in all_locations:
+        u = (loc or {}).get("pdf_url")
+        if u:
+            return u
+
+    repo_ids = _scan_locations_for_repo_ids(all_locations)
+    if repo_ids.get("arxiv"):
+        return f"https://arxiv.org/pdf/{repo_ids['arxiv']}"
+    if repo_ids.get("acl"):
+        return f"https://aclanthology.org/{repo_ids['acl']}.pdf"
+    if repo_ids.get("pmc"):
+        clean = repo_ids["pmc"] if str(repo_ids["pmc"]).upper().startswith("PMC") else f"PMC{repo_ids['pmc']}"
+        return f"https://europepmc.org/articles/{clean}?pdf=render"
+    return None
+
+
+def search_openalex(
+    query: str,
+    limit: int = 15,
+    sort="cited_by_count:desc",
+    logger: Optional[logging.Logger] = None,
+    max_candidates: int = 40,
+    use_paid_content_api: bool = False,
+    status_out: Optional[dict] = None,
+    fast: bool = True,
+) -> List[dict]:
+    """
+    Search research papers using OpenAlex. Returns a list of validated paper dicts.
+    sort: 'cited_by_count:desc' for most-cited, 'publication_date:desc' for latest,
+          or None for relevance (OpenAlex default).
+
+    Only papers for which a VERIFIED, direct PDF link could be resolved are
+    returned -- papers with no reachable PDF are filtered out entirely,
+    never returned with pdf_url=None.
+
+    max_candidates: how many raw results to pull from OpenAlex and PDF-check
+          before stopping at `limit` verified papers, since each candidate
+          can cost several extra network calls to resolve/verify.
+    use_paid_content_api: opt-in only -- lets the resolution ladder's last
+          step (OpenAlex's own hosted, metered content API) run as a final
+          fallback. Requires OPENALEX_CONTENT_API_KEY to be set. Off by
+          default since it costs a small amount per download.
+    """
+    log = logger or _fallback_logger
+    url = "https://api.openalex.org/works"
+    params = {"search": query, "per_page": max_candidates}
+    if sort:
+        params["sort"] = sort
+    if CONTACT_EMAIL:
+        # OpenAlex's "polite pool" -- faster, more consistent response times.
+        params["mailto"] = CONTACT_EMAIL
+
+    log.info("Searching OpenAlex for: %r (sort=%s)", query, sort or "relevance")
+
+    try:
+        response = requests.get(url, params=params, timeout=15)
+    except requests.RequestException as e:
+        log.warning("OpenAlex request failed: %s: %s", type(e).__name__, e)
+        if status_out is not None:
+            status_out.update(state="error", detail=f"{type(e).__name__}: {e}")
+        return []
+
+    if response.status_code != 200:
+        log.warning("OpenAlex returned bad status: %s", response.status_code)
+        if status_out is not None:
+            state = "rate_limited" if response.status_code == 429 else "error"
+            status_out.update(state=state, http=response.status_code)
+        return []
+
+    try:
+        data = response.json()
+    except ValueError as e:
+        log.warning("OpenAlex response failed to parse as JSON: %s", e)
+        if status_out is not None:
+            status_out.update(state="error", detail="bad JSON")
+        return []
+
+    raw_papers = data.get("results", [])
+
+    # Build lightweight candidate metadata first (no network calls here).
+    candidates = []
+    for paper in raw_papers:
+        title = paper.get("display_name")
+        if not title or not title.strip():
+            continue
+        authors = []
+        for authorship in paper.get("authorships", []) or []:
+            author_obj = authorship.get("author") or {}
+            name = author_obj.get("display_name")
+            if name:
+                authors.append(name)
+        abstract = decode_abstract(paper.get("abstract_inverted_index"))
+        landing_url = (paper.get("primary_location") or {}).get("landing_page_url")
+        candidates.append((paper, title, authors, abstract, landing_url))
+
+    # Resolve the slow, network-bound PDF checks for all candidates CONCURRENTLY
+    # rather than one at a time. This is what turns an ~80s wait into a few
+    # seconds; the escalation ladder and the verified-PDF-only result are
+    # unchanged -- only the wall-clock changes.
+    if fast:
+        # HYBRID default: cheap zero-network best-effort links, keep ALL papers
+        # (deep verification is deferred to 'Chat it out'). No network here.
+        resolved_pdfs = [resolve_openalex_pdf_fast(paper) for (paper, *_rest) in candidates]
+    else:
+        # Deep path: full verify ladder in parallel, keep only verified-PDF papers.
+        def _resolve(item):
+            return resolve_openalex_pdf(
+                item[0],
+                unpaywall_email=CONTACT_EMAIL,
+                use_paid_content_api=use_paid_content_api,
+                openalex_content_api_key=OPENALEX_CONTENT_API_KEY,
+                log=log,
+            )
+        with ThreadPoolExecutor(max_workers=min(12, len(candidates))) as ex:
+            resolved_pdfs = list(ex.map(_resolve, candidates)) if candidates else []
+
+    # Assemble in original (citation-sorted) order, keeping the first `limit` papers.
+    papers = []
+    for (paper, title, authors, abstract, landing_url), pdf_url in zip(candidates, resolved_pdfs):
+        if len(papers) >= limit:
+            break
+        if not fast and not pdf_url:
+            # Deep mode only: drop papers with no verifiable PDF.
+            continue
+        try:
+            paper_obj = ResearchPaper(
+                title=title.strip(),
+                authors=authors if authors else None,
+                abstract=abstract,
+                year=paper.get("publication_year"),
+                citation_count=paper.get("cited_by_count"),
+                url=landing_url,
+                pdf_url=pdf_url,
+                source="OpenAlex",
+            )
+            papers.append(paper_obj.model_dump())
+        except Exception as e:
+            log.warning("Skipped malformed OpenAlex paper: %s", e)
+            continue
+
+    if status_out is not None:
+        status_out.update(state="ok", count=len(papers))
+    log.info("OpenAlex: %d paper(s) retrieved (%s) out of %d candidates",
+              len(papers), "fast/best-effort PDF" if fast else "verified PDF", len(raw_papers))
+    return papers

app/modules/search/providers/semantic_scholar.py

@@ -0,0 +1,309 @@
+import logging
+import os
+import re
+import time
+from concurrent.futures import ThreadPoolExecutor
+from typing import List, Optional
+
+import requests
+from dotenv import load_dotenv
+
+from app.modules.search.providers.models import ResearchPaper
+
+load_dotenv()
+
+_fallback_logger = logging.getLogger(__name__)
+
+_REQUEST_HEADERS = {
+    # Some publisher/repo servers block requests with no browser-like UA.
+    "User-Agent": "Mozilla/5.0 (compatible; research-pipeline/1.0; +mailto:contact@example.com)"
+}
+_PDF_MAGIC = b"%PDF"
+_CITATION_PDF_URL_RE = re.compile(
+    r'<meta[^>]+name=["\']citation_pdf_url["\'][^>]+content=["\']([^"\']+)["\']'
+    r'|<meta[^>]+content=["\']([^"\']+)["\'][^>]+name=["\']citation_pdf_url["\']',
+    re.IGNORECASE,
+)
+
+
+# --------------------------------------------------------------------------
+# PDF resolution / verification helpers
+#
+# Semantic Scholar's openAccessPdf.url is a best-effort pointer -- sometimes
+# a raw PDF, sometimes a landing page that merely displays one. This runs
+# each candidate through an escalation ladder and VERIFIES the result
+# actually serves PDF bytes (checks the real response body's magic number,
+# "%PDF", rather than trusting a Content-Type header or a field name)
+# before ever handing it back.
+# --------------------------------------------------------------------------
+
+def _is_verified_pdf(url: str, timeout: int = 8) -> bool:
+    if not url:
+        return False
+    try:
+        with requests.get(
+            url, headers=_REQUEST_HEADERS, stream=True, timeout=timeout, allow_redirects=True
+        ) as resp:
+            if resp.status_code != 200:
+                return False
+            chunk = next(resp.iter_content(chunk_size=16), b"")
+            return chunk.startswith(_PDF_MAGIC)
+    except requests.RequestException:
+        return False
+
+
+def _extract_citation_pdf_url(landing_page_url: str, timeout: int = 8) -> Optional[str]:
+    """Scrape the citation_pdf_url meta tag off an HTML landing page."""
+    try:
+        resp = requests.get(landing_page_url, headers=_REQUEST_HEADERS, timeout=timeout)
+        if resp.status_code != 200:
+            return None
+        match = _CITATION_PDF_URL_RE.search(resp.text[:20000])
+        if not match:
+            return None
+        return match.group(1) or match.group(2)
+    except requests.RequestException:
+        return None
+
+
+def _resolve_pmc_pdf(pmc_id: str, timeout: int = 8) -> Optional[str]:
+    """Try Europe PMC's render endpoint for a PubMedCentral ID."""
+    clean_id = pmc_id if str(pmc_id).upper().startswith("PMC") else f"PMC{pmc_id}"
+    candidate = f"https://europepmc.org/articles/{clean_id}?pdf=render"
+    return candidate if _is_verified_pdf(candidate, timeout=timeout) else None
+
+
+def _resolve_unpaywall_pdf(doi: str, email: str, timeout: int = 8) -> Optional[str]:
+    """Last-resort fallback: ask Unpaywall for the best OA location's direct PDF."""
+    try:
+        resp = requests.get(
+            f"https://api.unpaywall.org/v2/{doi}", params={"email": email}, timeout=timeout
+        )
+        if resp.status_code != 200:
+            return None
+        best_location = (resp.json() or {}).get("best_oa_location") or {}
+        return best_location.get("url_for_pdf")
+    except (requests.RequestException, ValueError):
+        return None
+
+
+def resolve_pdf_url(
+    external_ids: dict,
+    fallback_url: Optional[str],
+    unpaywall_email: Optional[str] = None,
+    log: Optional[logging.Logger] = None,
+) -> Optional[str]:
+    """
+    Runs the full escalation ladder and returns a VERIFIED direct PDF url,
+    or None if nothing in the chain resolves to actual PDF bytes.
+
+        1. externalIds.ArXiv         -> arxiv.org/pdf/{id}                 (deterministic)
+        2. externalIds.ACL           -> aclanthology.org/{id}.pdf          (deterministic)
+        3. externalIds.PubMedCentral -> Europe PMC render endpoint         (verified)
+        4. openAccessPdf.url         -> verified directly, or scraped for
+                                         the citation_pdf_url meta tag if
+                                         it turns out to be an HTML page
+        5. externalIds.DOI           -> Unpaywall best_oa_location.url_for_pdf
+                                         (only if unpaywall_email is set)
+    """
+    log = log or _fallback_logger
+    external_ids = external_ids or {}
+
+    arxiv_id = external_ids.get("ArXiv")
+    if arxiv_id:
+        candidate = f"https://arxiv.org/pdf/{arxiv_id}"
+        if _is_verified_pdf(candidate):
+            return candidate
+
+    acl_id = external_ids.get("ACL")
+    if acl_id:
+        candidate = f"https://aclanthology.org/{acl_id}.pdf"
+        if _is_verified_pdf(candidate):
+            return candidate
+
+    pmc_id = external_ids.get("PubMedCentral")
+    if pmc_id:
+        candidate = _resolve_pmc_pdf(pmc_id)
+        if candidate:
+            return candidate
+
+    if fallback_url:
+        if _is_verified_pdf(fallback_url):
+            return fallback_url
+        scraped = _extract_citation_pdf_url(fallback_url)
+        if scraped and _is_verified_pdf(scraped):
+            return scraped
+
+    doi = external_ids.get("DOI")
+    if doi and unpaywall_email:
+        candidate = _resolve_unpaywall_pdf(doi, unpaywall_email)
+        if candidate and _is_verified_pdf(candidate):
+            return candidate
+
+    log.debug("No verified PDF resolved for Semantic Scholar paper externalIds=%s", external_ids)
+    return None
+
+
+def resolve_pdf_url_fast(external_ids: dict, fallback_url: Optional[str]) -> Optional[str]:
+    """
+    HYBRID fast path: a cheap, ZERO-network best-effort PDF link.
+
+    Builds the deterministic repo URL (arXiv / ACL / PubMedCentral) or takes
+    Semantic Scholar's own openAccessPdf link as-is, WITHOUT downloading or
+    verifying anything. The expensive verify + landing-page scrape + Unpaywall
+    ladder is deferred to 'Chat it out' time (see resolve_pdf_url), so search
+    stays fast while most links still point straight at a real PDF.
+    """
+    external_ids = external_ids or {}
+    arxiv_id = external_ids.get("ArXiv")
+    if arxiv_id:
+        return f"https://arxiv.org/pdf/{arxiv_id}"
+    acl_id = external_ids.get("ACL")
+    if acl_id:
+        return f"https://aclanthology.org/{acl_id}.pdf"
+    pmc_id = external_ids.get("PubMedCentral")
+    if pmc_id:
+        clean = pmc_id if str(pmc_id).upper().startswith("PMC") else f"PMC{pmc_id}"
+        return f"https://europepmc.org/articles/{clean}?pdf=render"
+    return fallback_url or None
+
+
+def search_semantic_scholar(
+    query: str,
+    limit: int = 15,
+    sort="citationCount:desc",
+    logger: Optional[logging.Logger] = None,
+    max_candidates: int = 25,
+    status_out: Optional[dict] = None,
+    fast: bool = True,
+) -> List[dict]:
+    """
+    Search research papers using the Semantic Scholar Bulk Search API (supports sorting).
+
+    Only papers for which a VERIFIED, direct PDF link could be resolved are
+    returned -- papers with no reachable PDF are filtered out entirely,
+    never returned with pdf_url=None.
+
+    max_candidates: how many raw search results to PDF-check before
+          stopping at `limit` verified papers, since each candidate can
+          cost 1-2 extra network calls to resolve/verify.
+    status_out: optional dict the caller can pass to learn WHY this source
+          came back empty -- populated with {"state": "ok"|"rate_limited"|
+          "error", ...} so a rate-limit (HTTP 429) is surfaced instead of
+          silently looking like "no results".
+    """
+    log = logger or _fallback_logger
+    url = "https://api.semanticscholar.org/graph/v1/paper/search"
+    api_key = os.getenv("SEMANTIC_SCHOLAR_API_KEY")
+    unpaywall_email = os.getenv("UNPAYWALL_EMAIL")
+    # Only send the header when we actually have a key -- an `x-api-key: None`
+    # header is meaningless and unauthenticated requests are throttled harder.
+    headers = {"x-api-key": api_key} if api_key else {}
+    log.info("Searching Semantic Scholar for: %r (sort=%s, keyed=%s)", query, sort or "default", bool(api_key))
+
+    params = {
+        "query": query,
+        "fields": ",".join([
+            "title", "abstract", "authors", "year",
+            "citationCount", "url", "openAccessPdf", "externalIds"
+        ])
+    }
+    if sort:
+        params["sort"] = sort
+
+    # Semantic Scholar's free tier throttles aggressively (HTTP 429). Retry a
+    # few times with backoff before giving up, and surface the rate-limit
+    # explicitly via status_out so it never just goes silently blank.
+    response = None
+    for attempt in range(3):
+        try:
+            response = requests.get(url, headers=headers, params=params, timeout=15)
+        except requests.RequestException as e:
+            log.warning("Semantic Scholar request failed: %s: %s", type(e).__name__, e)
+            if status_out is not None:
+                status_out.update(state="error", detail=f"{type(e).__name__}: {e}")
+            return []
+        if response.status_code != 429:
+            break
+        wait = 2 * (attempt + 1)
+        log.warning("Semantic Scholar rate-limited (HTTP 429) — attempt %d/3, retrying in %ds", attempt + 1, wait)
+        time.sleep(wait)
+
+    if response.status_code == 429:
+        log.warning("Semantic Scholar STILL rate-limited (HTTP 429) after retries — 0 results from this source.")
+        if status_out is not None:
+            status_out.update(state="rate_limited", http=429)
+        return []
+
+    if response.status_code != 200:
+        log.warning("Semantic Scholar returned bad status %s: %s", response.status_code, response.text[:300])
+        if status_out is not None:
+            status_out.update(state="error", http=response.status_code)
+        return []
+
+    try:
+        data = response.json()
+    except ValueError as e:
+        log.warning("Semantic Scholar response failed to parse as JSON: %s (raw: %s)", e, response.text[:500])
+        if status_out is not None:
+            status_out.update(state="error", detail="bad JSON")
+        return []
+
+    total_available = data.get("total", "unknown")
+    log.info("Semantic Scholar reports %s total match(es)", total_available)
+
+    raw_papers = data.get("data", [])[:max_candidates]
+
+    # Build candidate metadata first (no network), then resolve every
+    # candidate's PDF CONCURRENTLY instead of one at a time.
+    candidates = []
+    for paper in raw_papers:
+        title = paper.get("title")
+        if not title or not title.strip():
+            continue
+        authors = [a.get("name") for a in (paper.get("authors") or []) if a.get("name")]
+        openaccess_url = (paper.get("openAccessPdf") or {}).get("url")
+        external_ids = paper.get("externalIds") or {}
+        candidates.append((paper, title, authors, openaccess_url, external_ids))
+
+    if fast:
+        # HYBRID default: cheap zero-network best-effort links, keep ALL papers
+        # (deep verification is deferred to 'Chat it out'). No thread pool needed
+        # since resolution does no network here.
+        resolved = [resolve_pdf_url_fast(ext, oa) for (_, _, _, oa, ext) in candidates]
+    else:
+        # Deep path: full verify ladder in parallel, keep only verified-PDF papers.
+        def _resolve(item):
+            _, _, _, openaccess_url, external_ids = item
+            return resolve_pdf_url(external_ids, openaccess_url, unpaywall_email=unpaywall_email, log=log)
+        with ThreadPoolExecutor(max_workers=min(12, len(candidates))) as ex:
+            resolved = list(ex.map(_resolve, candidates)) if candidates else []
+
+    papers = []
+    for (paper, title, authors, openaccess_url, external_ids), pdf_url in zip(candidates, resolved):
+        if len(papers) >= limit:
+            break
+        if not fast and not pdf_url:
+            # Deep mode only: drop papers with no verifiable PDF.
+            continue
+        try:
+            paper_obj = ResearchPaper(
+                title=title.strip(),
+                authors=authors if authors else None,
+                abstract=paper.get("abstract"),
+                year=paper.get("year"),
+                citation_count=paper.get("citationCount"),
+                url=paper.get("url"),
+                pdf_url=pdf_url,
+                source="SemanticScholar",
+            )
+            papers.append(paper_obj.model_dump())
+        except Exception as e:
+            log.warning("Skipped malformed Semantic Scholar paper: %s", e)
+            continue
+
+    if status_out is not None:
+        status_out.update(state="ok", count=len(papers))
+    log.info("Semantic Scholar: %d paper(s) retrieved (%s) out of %d candidates",
+              len(papers), "fast/best-effort PDF" if fast else "verified PDF", len(raw_papers))
+    return papers

app/modules/search/reranking.py

@@ -0,0 +1,159 @@
+"""
+reranking.py
+
+Reranks papers by semantic similarity to the Research Intent using SPECTER
+(allenai-specter) — an embedding model trained specifically on academic
+paper title/abstract pairs, rather than a generic sentence embedding model.
+
+Install once:
+    pip install sentence-transformers
+"""
+
+import logging
+from typing import Optional
+
+from sentence_transformers import SentenceTransformer
+import numpy as np
+
+# Loaded once at module import time — NOT inside the function.
+# Loading a transformer model from disk/HF hub takes a few seconds; if this
+# were inside the function, every call would reload it, which is wasteful
+# if this function gets called more than once in a session.
+#
+# device is pinned explicitly instead of left to SentenceTransformer's
+# auto-detect. On HF ZeroGPU, torch reports a GPU at import time but only
+# actually grants one inside an @spaces.GPU window — and this model is called
+# from a LangGraph node, which is outside any such window. Auto-detect would
+# therefore load it onto "cuda" and fail on first encode. Reranking ~30 abstracts
+# is a couple of seconds on CPU.
+_MODEL = SentenceTransformer("sentence-transformers/allenai-specter", device="cpu")
+
+_fallback_logger = logging.getLogger(__name__)
+
+
+def rerank_by_relevance(
+    research_intent: str,
+    papers: dict[str, str],
+    top_n: int = 10,
+    logger: Optional[logging.Logger] = None,
+) -> dict[str, str]:
+    """
+    Rerank papers by semantic similarity to the Research Intent.
+
+    Parameters
+    ----------
+    research_intent : str
+        The full Research Intent text (Problem + Objective + Additional
+        Context, or however you've combined it) — used as the query vector.
+    papers : dict[str, str]
+        {normalized_title: abstract} — the aggregator's title/abstract pairs.
+    top_n : int
+        How many top-ranked papers to keep. Default 15.
+    logger : logging.Logger, optional
+        Node-scoped logger from the caller. Falls back to a module logger
+        when this function is used standalone.
+
+    Returns
+    -------
+    dict[str, str]
+        A NEW dict, same {normalized_title: abstract} shape, containing only
+        the top_n most relevant entries, ordered from most to least relevant.
+        Insertion order is preserved (Python 3.7+ dicts are ordered), so
+        iterating this dict gives you the ranking directly.
+    """
+    log = logger or _fallback_logger
+
+    if not papers:
+        return {}
+
+    # --- Guard against empty/whitespace-only abstracts ---
+    # These can't be meaningfully embedded for relevance comparison. Rather
+    # than crash or silently mis-rank them, exclude them from ranking and
+    # log which ones were skipped so nothing disappears without a trace.
+    valid_titles = []
+    valid_abstracts = []
+    skipped_no_abstract = []
+
+    for normalized_title, abstract in papers.items():
+        if abstract and abstract.strip():
+            valid_titles.append(normalized_title)
+            valid_abstracts.append(abstract)
+        else:
+            skipped_no_abstract.append(normalized_title)
+
+    if skipped_no_abstract:
+        log.info("Skipped %d paper(s) with no abstract: %s", len(skipped_no_abstract), skipped_no_abstract)
+
+    if not valid_abstracts:
+        log.info("No papers had usable abstracts — returning empty result.")
+        return {}
+
+    # --- Embed the query (Research Intent) and all candidate abstracts ---
+    # normalize_embeddings=True means each vector has unit length, so a
+    # simple dot product between two vectors IS the cosine similarity —
+    # no separate cosine-similarity library call needed.
+    query_embedding = _MODEL.encode(
+        research_intent,
+        normalize_embeddings=True,
+        convert_to_numpy=True,
+    )
+
+    paper_embeddings = _MODEL.encode(
+        valid_abstracts,
+        normalize_embeddings=True,
+        convert_to_numpy=True,
+        batch_size=32,
+        show_progress_bar=False,
+    )
+
+    # --- Cosine similarity of every paper against the query, in one shot ---
+    similarities = paper_embeddings @ query_embedding  # shape: (num_papers,)
+
+    # --- Sort by similarity, descending ---
+    ranked_indices = np.argsort(-similarities)
+
+    # --- Log the full ranking for visibility/debugging before truncating ---
+    ranking_lines = "\n".join(
+        f"    {similarities[idx]:.4f}  {valid_titles[idx]}" for idx in ranked_indices
+    )
+    log.info("Full relevance ranking (%d papers, title : similarity score):\n%s", len(valid_titles), ranking_lines)
+
+    top_indices = ranked_indices[:top_n]
+
+    dropped_count = len(valid_titles) - len(top_indices)
+    if dropped_count > 0:
+        log.info("Kept top %d, dropped %d lower-relevance paper(s).", len(top_indices), dropped_count)
+
+    # --- Build the result dict in ranked order ---
+    result = {
+        valid_titles[idx]: valid_abstracts[idx]
+        for idx in top_indices
+    }
+
+    return result
+
+
+if __name__ == "__main__":
+    # Quick standalone test
+    test_intent = (
+        "Identify a robust methodology for comparing the fuel efficiency of "
+        "human-driven and reinforcement-learning-controlled vehicles in "
+        "car-following maneuvers, accounting for speed, acceleration, and headway."
+    )
+    test_papers = {
+        "ecofollower an environmentfriendly car following model": (
+            "This study introduces EcoFollower, a novel eco-car-following "
+            "model developed using reinforcement learning to optimize fuel "
+            "consumption in car-following scenarios."
+        ),
+        "predicting fuel research octane number using spectra": (
+            "We show that an accurate statistical model for the Research "
+            "Octane Number of gasoline can be constructed using infrared "
+            "absorbance spectroscopy data."
+        ),
+    }
+
+    reranked = rerank_by_relevance(test_intent, test_papers, top_n=15)
+    print("\nFinal reranked result:")
+    for title, abstract in reranked.items():
+        print(f"- {title}")

app/modules/search/router.py

@@ -0,0 +1,16 @@
+from fastapi import APIRouter, HTTPException
+
+from app.modules.search.jobs import get_job
+from app.modules.search.schemas import SearchStatusResponse
+
+router = APIRouter(prefix="/search", tags=["search"])
+
+
+@router.get("/status/{job_id}", response_model=SearchStatusResponse)
+def search_status(job_id: str):
+    """Poll this until status is 'done' (or 'error'). On 'done',
+    clustered_papers is the final payload the UI renders."""
+    job = get_job(job_id)
+    if job is None:
+        raise HTTPException(status_code=404, detail="Unknown job_id.")
+    return SearchStatusResponse(**job)

app/modules/search/schemas.py

@@ -0,0 +1,10 @@
+from typing import List, Literal, Optional
+
+from pydantic import BaseModel
+
+
+class SearchStatusResponse(BaseModel):
+    job_id: str
+    status: Literal["running", "done", "error"]
+    clustered_papers: Optional[List[dict]] = None
+    error: Optional[str] = None

chatbot_core/Qa.py

@@ -0,0 +1,171 @@
+"""
+qa.py
+-----
+Interactive Q&A over a single research paper. Handles vectorization
+automatically -- just point it at a PDF:
+
+    python qa.py /path/to/paper.pdf
+
+First run on a given PDF builds the vectorstore (may take a moment while the
+embedding model loads); every run after that loads the cached vectorstore
+instantly since it's keyed by the PDF's content hash.
+
+Requires a Groq API key. Create a .env file in this same directory containing:
+
+    GROQ_API_KEY=your-key-here
+
+Install once (on top of vectorize.py's requirements):
+    pip install --break-system-packages langchain langchain-community \
+        langchain-classic langchain-groq python-dotenv
+
+Note: LangChain 1.0+ split ContextualCompressionRetriever and
+CrossEncoderReranker out of the core `langchain` package into the new
+`langchain-classic` package -- that's why langchain-classic is required above.
+
+Swap LLM providers: replace get_llm() below with e.g. ChatOpenAI or
+ChatAnthropic if you'd rather not use Groq.
+"""
+
+import os
+import sys
+
+from dotenv import load_dotenv
+from langchain_groq import ChatGroq
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.messages import HumanMessage, AIMessage
+from langchain_community.cross_encoders import HuggingFaceCrossEncoder
+from langchain_classic.retrievers.document_compressors import CrossEncoderReranker
+from langchain_classic.retrievers.contextual_compression import ContextualCompressionRetriever
+
+from vectorizeer import build_vectorstore
+
+load_dotenv()  # reads GROQ_API_KEY from a .env file in the current directory
+
+QA_SYSTEM_PROMPT = """You are a research assistant helping someone deeply understand a specific paper.
+Treat every question as a chance to teach, not just retrieve -- they want both
+the facts and why those facts matter.
+
+Ground every answer strictly in the excerpts below. Do not use outside knowledge,
+and do not fill gaps with what a similar paper would typically say. If the
+excerpts don't contain the answer, say so plainly rather than guessing.
+
+When you answer:
+- Be comprehensive: explain the relevant method, result, or claim fully rather
+  than a one-line summary. If the question touches a mechanism (an algorithm, a
+  fine-tuning task, an experimental setup), walk through how it actually works,
+  not just what it's called.
+- Surface significance: don't just report what the paper found -- explain why
+  it matters. What problem does it solve, what breaks without it, how does it
+  compare to prior approaches, what does it enable going forward.
+- Stay accessible: write for someone smart but not necessarily a specialist in
+  this exact subfield. Define acronyms and technical terms the first time you
+  use them, and prefer plain language wherever it loses no precision.
+- Cite as you go: every claim should be traceable to a page number from the
+  tagged excerpts below. Weave citations naturally into the explanation rather
+  than listing them at the end.
+- Stay focused: comprehensive isn't the same as padded. Cut anything that
+  doesn't help the reader actually understand the paper better.
+
+EXCERPTS:
+{context}
+"""
+
+
+def get_llm():
+    api_key = os.environ.get("SECOND_GROQ_API_KEY")
+    if not api_key:
+        raise EnvironmentError(
+            "GROQ_API_KEY not found. Add it to a .env file in this directory "
+            "(GROQ_API_KEY=your-key-here) -- load_dotenv() picks it up automatically."
+        )
+    return ChatGroq(model="llama-3.3-70b-versatile", temperature=0)
+
+
+def get_retriever(vectorstore, k: int = 15, top_n: int = 5, device: str = "cpu"):
+    """Retrieve k candidates by similarity, then rerank down to the best top_n
+    with a small cross-encoder -- this is the single biggest accuracy lever
+    on top of the chunking itself.
+
+    device is pinned to "cpu" by default rather than left to auto-detect. The
+    retriever outlives any single call, so on HF ZeroGPU -- where a GPU exists
+    only inside an @spaces.GPU window -- a cuda-resident cross-encoder here would
+    fail the moment it's used. Reranking 15 short chunks is ~a second on CPU."""
+    base_retriever = vectorstore.as_retriever(search_kwargs={"k": k})
+    reranker_model = HuggingFaceCrossEncoder(
+        model_name="BAAI/bge-reranker-base",
+        model_kwargs={"device": device},
+    )
+    compressor = CrossEncoderReranker(model=reranker_model, top_n=top_n)
+    return ContextualCompressionRetriever(base_compressor=compressor, base_retriever=base_retriever)
+
+
+def format_docs(docs) -> str:
+    parts = []
+    for d in docs:
+        tag = f"[Section: {d.metadata.get('section', '?')} | Page: {d.metadata.get('page', '?')}]"
+        parts.append(f"{tag}\n{d.page_content}")
+    return "\n\n---\n\n".join(parts)
+
+
+def build_chain(llm):
+    prompt = ChatPromptTemplate.from_messages([
+        ("system", QA_SYSTEM_PROMPT),
+        MessagesPlaceholder("chat_history"),
+        ("human", "{question}"),
+    ])
+    return prompt | llm
+
+
+def main():
+    if len(sys.argv) < 2:
+        print("Usage: python qa.py /path/to/paper.pdf")
+        sys.exit(1)
+
+    pdf_path = sys.argv[1]
+    if not os.path.isfile(pdf_path):
+        print(f"File not found: {pdf_path}")
+        sys.exit(1)
+
+    vectorstore = build_vectorstore(pdf_path)
+    retriever = get_retriever(vectorstore)
+    llm = get_llm()
+    chain = build_chain(llm)
+
+    chat_history = []
+    print("\nReady. Ask questions about the paper (type 'exit' to quit).\n")
+
+    while True:
+        try:
+            question = input("You: ").strip()
+        except (EOFError, KeyboardInterrupt):
+            break
+        if question.lower() in ("exit", "quit"):
+            break
+        if not question:
+            continue
+
+        docs = retriever.invoke(question)
+        context = format_docs(docs)
+
+        print("\nAssistant: ", end="", flush=True)
+        answer = ""
+        for chunk in chain.stream({
+            "question": question,
+            "chat_history": chat_history,
+            "context": context,
+        }):
+            token = chunk.content
+            if token:
+                print(token, end="", flush=True)
+                answer += token
+        print("\n")
+ 
+        sources = sorted(set(f"p.{d.metadata.get('page')}" for d in docs), key=lambda s: s)
+        print(f"[sources: {', '.join(sources)}]\n")
+ 
+        chat_history.append(HumanMessage(content=question))
+        chat_history.append(AIMessage(content=answer))
+
+
+if __name__ == "__main__":
+    main()

chatbot_core/vectorizeer.py

@@ -0,0 +1,216 @@
+"""
+vectorize.py
+------------
+Builds (or loads, if already built) a persisted vectorstore for a single research
+paper PDF. Designed to be imported by qa.py, but can also be run standalone:
+
+    python vectorize.py /path/to/paper.pdf
+
+Chunking strategy:
+  1. Split the paper by detected section headers ("1 Introduction", "3.2 Inference
+     with V-RAG", etc.) so each chunk is a real semantic unit, not an arbitrary
+     page cut.
+  2. Within each section, protect table-like blocks (lines dense with numbers) so
+     they stay intact as their own chunk instead of getting sliced by the
+     recursive splitter.
+  3. Any remaining oversized prose is split with RecursiveCharacterTextSplitter
+     (paragraph -> sentence -> word fallback separators).
+  4. If no section headers are detected at all (unusual paper formatting), falls
+     back to page-wise chunks with a small overlap so nothing is silently lost.
+
+Install once:
+    pip install --break-system-packages pymupdf langchain langchain-core \
+        langchain-text-splitters langchain-chroma langchain-community \
+        sentence-transformers chromadb
+
+Embedding model: BAAI/bge-base-en-v1.5 (local, ~440MB, runs fine on CPU or a
+sliver of GPU -- no API key, no rate limits, no VRAM worries).
+"""
+
+import os
+import re
+import sys
+import hashlib
+
+import fitz  # PyMuPDF
+from langchain_core.documents import Document
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from langchain_chroma import Chroma
+
+# Matches headers like "1 Introduction", "3.2 Inference with V-RAG", "6 Conclusion"
+HEADER_PATTERN = re.compile(r'\n(\d{1,2}(?:\.\d{1,2})?\s+[A-Z][A-Za-z][^\n]{2,60})(?=\n)')
+
+# A line "looks like a table row" if enough of its tokens are numeric.
+NUMERIC_TOKEN = re.compile(r'^-?\d+\.\d+$|^\d+$')
+PAGE_OVERLAP_CHARS = 200
+MAX_CHUNK_CHARS = 900
+
+
+def get_pdf_hash(pdf_path: str) -> str:
+    """Short content hash -> stable, unique persist directory per PDF."""
+    with open(pdf_path, "rb") as f:
+        return hashlib.md5(f.read()).hexdigest()[:12]
+
+
+def extract_sections(pdf_path: str):
+    """Return a list of {title, text, page} dicts, one per detected section.
+
+    Falls back to page-wise chunks (with overlap) if no section headers are
+    detected, so unusual paper formats still produce usable chunks.
+    """
+    doc = fitz.open(pdf_path)
+    full_text = ""
+    page_map = []  # (char_start, char_end, page_number) per page
+    for i, page in enumerate(doc):
+        t = page.get_text()
+        page_map.append((len(full_text), len(full_text) + len(t), i + 1))
+        full_text += t
+
+    matches = list(HEADER_PATTERN.finditer(full_text))
+    sections = []
+
+    if matches:
+        # Preamble before the first header = title/authors/abstract
+        if matches[0].start() > 0:
+            preamble = full_text[:matches[0].start()].strip()
+            if preamble:
+                page_no = next(p for s, e, p in page_map if s <= 0 < e)
+                sections.append({"title": "Abstract / Preamble", "text": preamble, "page": page_no})
+
+        for idx, m in enumerate(matches):
+            start = m.start()
+            end = matches[idx + 1].start() if idx + 1 < len(matches) else len(full_text)
+            title = m.group(1).strip()
+            body = full_text[m.end():end].strip()
+            if body:
+                page_no = next(p for s, e, p in page_map if s <= start < e)
+                sections.append({"title": title, "text": body, "page": page_no})
+    else:
+        print("[vectorize] No section headers detected -- falling back to page-wise chunking.")
+        page_texts = [page.get_text() for page in doc]
+        for i, t in enumerate(page_texts):
+            prefix = page_texts[i - 1][-PAGE_OVERLAP_CHARS:] if i > 0 else ""
+            sections.append({"title": f"Page {i + 1}", "text": prefix + t, "page": i + 1})
+
+    return sections
+
+
+def _looks_tabular(line: str) -> bool:
+    tokens = line.split()
+    if len(tokens) < 3:
+        return False
+    numeric = sum(1 for t in tokens if NUMERIC_TOKEN.match(t))
+    return (numeric / len(tokens)) >= 0.4
+
+
+def _split_protecting_tables(text: str, splitter: RecursiveCharacterTextSplitter):
+    """Group consecutive table-like lines into their own block; split the rest
+    normally. Returns a list of (is_table: bool, chunk_text: str)."""
+    lines = text.split("\n")
+    blocks = []
+    buf, buf_is_table = [], None
+
+    for line in lines:
+        is_table = _looks_tabular(line)
+        if buf_is_table is None:
+            buf_is_table = is_table
+        if is_table == buf_is_table:
+            buf.append(line)
+        else:
+            blocks.append((buf_is_table, "\n".join(buf)))
+            buf, buf_is_table = [line], is_table
+    if buf:
+        blocks.append((buf_is_table, "\n".join(buf)))
+
+    chunks = []
+    for is_table, block_text in blocks:
+        block_text = block_text.strip()
+        if not block_text:
+            continue
+        if is_table or len(block_text) <= MAX_CHUNK_CHARS:
+            chunks.append((is_table, block_text))
+        else:
+            for piece in splitter.split_text(block_text):
+                chunks.append((False, piece))
+    return chunks
+
+
+def get_embeddings(device: str = "cpu"):
+    """Local, lightweight embedding model -- no API key, no rate limits.
+
+    device stays "cpu" by default (standalone runs behave exactly as before).
+    It's a parameter rather than a hardcoded string because on HF ZeroGPU only
+    code inside an @spaces.GPU window may touch "cuda" -- so the caller has to
+    say which side of that window it's on. Letting the model auto-detect would
+    put it on "cuda" at import and blow up on first use outside the window.
+    """
+    from langchain_community.embeddings import HuggingFaceBgeEmbeddings
+    return HuggingFaceBgeEmbeddings(
+        model_name="BAAI/bge-base-en-v1.5",
+        model_kwargs={"device": device},
+        encode_kwargs={"normalize_embeddings": True},
+    )
+
+
+def build_vectorstore(pdf_path: str, persist_root: str = "./vectorstores", force_rebuild: bool = False,
+                      device: str = "cpu"):
+    """Build a new vectorstore for pdf_path, or load the existing one if it was
+    already built for this exact file.
+
+    device is forwarded to get_embeddings. Building on "cuda" and re-opening on
+    "cpu" is a supported pattern: the persisted vectors are identical either way,
+    so the expensive bulk encode can run on a GPU and the cheap query-time encode
+    on the CPU.
+    """
+    pdf_hash = get_pdf_hash(pdf_path)
+    persist_dir = os.path.join(persist_root, pdf_hash)
+    embeddings = get_embeddings(device)
+
+    if os.path.isdir(persist_dir) and os.listdir(persist_dir) and not force_rebuild:
+        print(f"[vectorize] Existing vectorstore found at {persist_dir} -- loading it.")
+        return Chroma(persist_directory=persist_dir, embedding_function=embeddings)
+
+    print(f"[vectorize] Building vectorstore for: {pdf_path}")
+    sections = extract_sections(pdf_path)
+    splitter = RecursiveCharacterTextSplitter(
+        chunk_size=700, chunk_overlap=100,
+        separators=["\n\n", "\n", ". ", " "],
+    )
+
+    docs = []
+    for sec in sections:
+        for is_table, piece_text in _split_protecting_tables(sec["text"], splitter):
+            docs.append(Document(
+                page_content=piece_text,
+                metadata={
+                    "section": sec["title"],
+                    "page": sec["page"],
+                    "type": "table" if is_table else "text",
+                    "source": os.path.basename(pdf_path),
+                },
+            ))
+
+    if not docs:
+        raise ValueError(
+            f"No extractable text found in {pdf_path}. This PDF may be scanned "
+            "(image-only) rather than a native text PDF -- it needs OCR first."
+        )
+
+    print(f"[vectorize] {len(docs)} chunks created across {len(sections)} sections.")
+    os.makedirs(persist_dir, exist_ok=True)
+    vectorstore = Chroma.from_documents(docs, embeddings, persist_directory=persist_dir)
+    print(f"[vectorize] Saved to {persist_dir}")
+    return vectorstore
+
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("Usage: python vectorize.py /path/to/paper.pdf")
+        sys.exit(1)
+
+    path = sys.argv[1]
+    if not os.path.isfile(path):
+        print(f"File not found: {path}")
+        sys.exit(1)
+
+    build_vectorstore(path)

nova_app.py

@@ -0,0 +1,544 @@
+"""
+NOVA — Research, guided by SONIC
+================================================================
+A single Gradio app that stitches together the two projects, unchanged:
+
+  • app/            the research pipeline (structured_agent's `app/` package):
+                    INTENT graph  -> SEARCH + CLUSTER graph
+  • chatbot_core/   the single-PDF Q&A chatbot (Qa.py + vectorizeer.py)
+
+NOVA is the product. SONIC is the assistant persona that talks you through it.
+
+Flow:
+  USER RESEARCH IDEA
+     -> INTENT agent frames it (Problem / Objective / Additional Context)
+     -> you review/edit it
+     -> SEARCH agent fetches + reranks + clusters papers
+     -> results shown as clean thumbnails (title, authors, links)
+     -> "Chat it out" on any paper: its PDF is downloaded, vectorized by
+        vectorizeer.build_vectorstore, and you Q&A over it with Qa.py's chain.
+
+This file is UI + wiring only. It does NOT change any agent or chatbot logic —
+it imports their functions and drives them.
+
+Where Streamlit re-executed one script top-to-bottom on every interaction, Gradio
+builds a persistent component graph once and fires explicit handlers. So the
+"stage" that Streamlit kept in session_state and branched on is here a set of
+Columns whose `visible` flag every handler returns. Same state machine — declared
+once instead of re-derived per rerun.
+
+Run:
+    python nova_app.py            # from inside the NOVA/ folder
+"""
+
+from ui import paths  # noqa: F401  — MUST be first: wires sys.path + chdir + .env
+
+import shutil
+import uuid
+
+import gradio as gr
+
+from ui.agents import load_agents, warm_chatbot_models_async
+from ui.chat_engine import prepare_chat_stream
+from ui.constants import CLUSTER_ACCENTS, SOURCE_COLORS
+from ui.intent_text import join_intent_sections, split_intent_sections
+from ui.papers import author_line, first_available
+from ui.paths import DOWNLOADS_DIR, VECTORSTORES_DIR
+from ui.search_progress import search_steps_html
+from ui.sonic import SONIC_AVATAR, SONIC_DATA_URI, USER_AVATAR, sonic_says
+from ui.theme import CSS, FORCE_DARK, NOVA_THEME
+
+# ---------------------------------------------------------------------------
+# 1. SESSION STATE
+# ---------------------------------------------------------------------------
+STAGE_NAMES = ("boot", "welcome", "refining", "review", "searching", "results", "chat")
+
+
+def new_state() -> dict:
+    """One of these per browser session. Gradio deep-copies it into each new
+    session, so the mutable members below are never shared across users."""
+    return {
+        "run_id": "",
+        "user_query": "",
+        "problem": "",
+        "objective": "",
+        "context": "",
+        "clusters": [],
+        "papers_by_key": {},   # normalized_title -> full record (flattened, for chat lookup)
+        "active_chat": None,   # normalized_title of the paper being chatted, or None
+        "chats": {},           # normalized_title -> {retriever, chain, messages, title}
+        "source_status": {},   # {"Semantic Scholar": {"state": "rate_limited", ...}, ...}
+    }
+
+
+def wipe_disk_cache():
+    """Delete every cached vectorstore and downloaded PDF so a new search starts
+    from a clean slate — no old paper's chunks or PDFs can leak in."""
+    for folder in (VECTORSTORES_DIR, DOWNLOADS_DIR):
+        try:
+            if folder.exists():
+                shutil.rmtree(folder, ignore_errors=True)
+            folder.mkdir(exist_ok=True)
+        except Exception:
+            pass
+
+
+def _stages(active: str):
+    """Visibility updates for every stage Column, in STAGE_NAMES order."""
+    return tuple(gr.update(visible=(name == active)) for name in STAGE_NAMES)
+
+
+# ---------------------------------------------------------------------------
+# 2. STATIC MARKUP
+# ---------------------------------------------------------------------------
+HEADER_HTML = """
+<div class="nova-brand">
+  <span class="nova-star">✦</span>
+  <span class="nova-mark">NOVA</span>
+  <span class="nova-sub">Research&nbsp;Assistant</span>
+  <span class="sonic-chip"><span class="sonic-dot"></span> SONIC online</span>
+</div>
+"""
+
+HERO_FIGURE_HTML = (
+    f'<div class="hero-figure"><img src="{SONIC_DATA_URI}" alt="SONIC"/>'
+    f'<div class="hero-name">SONIC · your research buddy</div></div>'
+)
+
+HERO_SPEECH_HTML = """
+<div class="hero-speech">
+  <div class="sonic-name">SONIC</div>
+  hey, wass up 👋<br>what's on your mind about research today?<br>
+  Dump the raw idea on me — the messier the better. I'll shape it into something sharp.
+</div>
+<div class="hero-answer-label">✍️ your answer</div>
+"""
+
+
+def boot_html(phase: str, pct: int) -> str:
+    return (
+        f'<div class="vec-wrap">'
+        f'  <div class="vec-figure"><img src="{SONIC_DATA_URI}" alt="SONIC"/></div>'
+        f'  <div class="vec-quote"><span class="q">SONIC:</span> &ldquo;{phase}&rdquo;</div>'
+        f'  <div class="vec-bar"><div class="vec-fill" style="width:{pct}%"></div></div>'
+        f'</div>'
+    )
+
+
+def source_status_html(status: dict) -> str:
+    """Show per-source status so a rate-limited/failed source is never invisible."""
+    if not status:
+        return ""
+    chips = []
+    for name, s in status.items():
+        state = (s or {}).get("state")
+        if state == "ok":
+            chips.append(f'<span class="src-stat ok">{name} ✓ {s.get("count", 0)}</span>')
+        elif state == "rate_limited":
+            chips.append(f'<span class="src-stat warn">{name} ⚠ rate-limited (HTTP {s.get("http", 429)})</span>')
+        elif state == "error":
+            detail = s.get("detail") or f'HTTP {s.get("http", "?")}'
+            chips.append(f'<span class="src-stat err">{name} ✕ {detail}</span>')
+        else:
+            chips.append(f'<span class="src-stat muted">{name} —</span>')
+    return '<div class="src-stat-row">' + "".join(chips) + "</div>"
+
+
+def paper_card_html(norm_title: str, record: dict) -> str:
+    title = record.get("title") or norm_title.title()
+    badges = "".join(
+        f'<span class="src-badge" style="color:{SOURCE_COLORS.get(s, "#8b93a7")};'
+        f'border-color:{SOURCE_COLORS.get(s, "#8b93a7")}55;'
+        f'background:{SOURCE_COLORS.get(s, "#8b93a7")}18;">{s}</span>'
+        for s in (record.get("source") or [])
+    )
+    return (
+        f'<div class="paper-title">{title}</div>'
+        f'<div class="paper-meta">{author_line(record.get("authors"), record.get("year"))}<br>{badges}</div>'
+    )
+
+
+def card_links_html(page_url: str, pdf_url: str) -> str:
+    """The 📄 Paper / ⬇ PDF pair. Plain anchors rather than gr.Button: they're
+    pure navigation, and a real <a> opens a new tab with no server round-trip."""
+    paper = (f'<a class="card-link" href="{page_url}" target="_blank" rel="noopener">📄 Paper</a>'
+             if page_url else '<span class="card-link dead">📄 Paper</span>')
+    pdf = (f'<a class="card-link" href="{pdf_url}" target="_blank" rel="noopener">⬇ PDF</a>'
+           if pdf_url else '<span class="card-link dead">⬇ PDF</span>')
+    return f'<div class="card-links">{paper}{pdf}</div>'
+
+
+# ---------------------------------------------------------------------------
+# 3. HANDLERS THAT TOUCH NO COMPONENTS
+# ---------------------------------------------------------------------------
+def open_chat_for(norm_title: str):
+    """Build a per-card click handler. The card grid is generated in a loop, so
+    each button needs to close over its own paper key."""
+    def _open(st):
+        st["active_chat"] = norm_title
+        record = st["papers_by_key"].get(norm_title, {})
+        title = record.get("title") or norm_title.title()
+        head = (f'<div class="cluster-head"><div class="cluster-bar" style="background:#7c5cff;"></div>'
+                f'<div class="cluster-title">💬 {title}</div></div>')
+        cached = st["chats"].get(norm_title)
+        return (*_stages("chat"), head,
+                gr.update(value="", visible=not cached),
+                gr.update(value=(cached["messages"] if cached else []), visible=bool(cached)),
+                gr.update(visible=bool(cached)),
+                st)
+    return _open
+
+
+def prep_chat(st):
+    """Download + vectorize this paper, animating SONIC's pep-quotes while the
+    real work runs on a worker thread. No-op if this chat is already built."""
+    key = st["active_chat"]
+    if not key or key in st["chats"]:
+        return
+
+    record = st["papers_by_key"].get(key, {})
+    session = error = None
+    for html, done, session, error in prepare_chat_stream(record):
+        if not done:
+            yield (gr.update(value=html, visible=True), gr.update(visible=False),
+                   gr.update(visible=False), st)
+
+    if error or not session:
+        msg = error or "Couldn't prepare this paper for chat."
+        yield (gr.update(value=f'<div class="nova-error">{msg}</div>', visible=True),
+               gr.update(visible=False), gr.update(visible=False), st)
+        return
+
+    session.update({"messages": [], "title": record.get("title") or key.title()})
+    st["chats"][key] = session
+    yield (gr.update(value="", visible=False), gr.update(value=[], visible=True),
+           gr.update(visible=True), st)
+
+
+# ---------------------------------------------------------------------------
+# 4. THE APP
+# ---------------------------------------------------------------------------
+# Gradio 6 moved theme/css/js off the Blocks constructor and onto launch().
+with gr.Blocks(title="NOVA · Research Assistant", analytics_enabled=False) as demo:
+    state = gr.State(new_state())
+    # Mirrors state["clusters"]. gr.render can't watch a dict mutated in place,
+    # so the search handler reassigns this to a fresh list to trigger a redraw.
+    clusters_state = gr.State([])
+
+    with gr.Column(elem_id="nova-root"):
+        gr.HTML(HEADER_HTML)
+
+        # ---------------- BOOT ----------------
+        with gr.Column(visible=True) as boot_col:
+            boot_panel = gr.HTML(boot_html("Waking up SONIC — loading the research + reading models…", 8))
+
+        # ---------------- WELCOME ----------------
+        with gr.Column(visible=False) as welcome_col:
+            with gr.Row(equal_height=False):
+                with gr.Column(scale=9):
+                    gr.HTML(HERO_FIGURE_HTML)
+                with gr.Column(scale=11):
+                    gr.HTML(HERO_SPEECH_HTML)
+                    query_box = gr.Textbox(
+                        lines=6, max_lines=12, show_label=False, container=False,
+                        placeholder="e.g. I want to compare fuel efficiency of human-driven vs RL-controlled "
+                                    "cars in car-following… comparing is hard because velocity, acceleration, "
+                                    "headway all change at once…",
+                    )
+                    go_btn = gr.Button("Let's go  ✦", variant="primary")
+
+        # ---------------- REFINING ----------------
+        with gr.Column(visible=False) as refining_col:
+            gr.HTML(sonic_says("that seems great — lemme juss refine it ✨"))
+            refining_panel = gr.HTML()
+
+        # ---------------- REVIEW ----------------
+        with gr.Column(visible=False) as review_col:
+            gr.HTML(sonic_says("here's how I framed it. Tweak anything that's off, then I'll go hunting 🔍"))
+            gr.HTML('<div class="field-label">🧩 Problem</div>')
+            problem_box = gr.Textbox(lines=5, show_label=False, container=False)
+            gr.HTML('<div class="field-label">🎯 Objective</div>')
+            objective_box = gr.Textbox(lines=4, show_label=False, container=False)
+            gr.HTML('<div class="field-label">🗂️ Additional Context</div>')
+            context_box = gr.Textbox(lines=4, show_label=False, container=False)
+            with gr.Row():
+                find_btn = gr.Button("Find the papers  🔍", variant="primary", scale=2)
+                over_btn = gr.Button("Start over", scale=1)
+                gr.HTML("")   # spacer: keeps the two buttons off full width
+
+        # ---------------- SEARCHING ----------------
+        with gr.Column(visible=False) as searching_col:
+            gr.HTML(sonic_says("on it — scouring arXiv, Semantic Scholar &amp; OpenAlex, then reranking "
+                               "and clustering by approach 🔎"))
+            search_panel = gr.HTML()
+
+        # ---------------- RESULTS ----------------
+        # Body is filled in by the @gr.render below, once every component it
+        # needs to drive (the chat stage) exists.
+        with gr.Column(visible=False) as results_col:
+            results_head = gr.HTML()
+            with gr.Row():
+                new_search_btn = gr.Button("🔄 New search", scale=1)
+                gr.HTML("")   # spacer
+
+        # ---------------- CHAT ----------------
+        with gr.Column(visible=False) as chat_col:
+            with gr.Row():
+                back_btn = gr.Button("← Back to papers", scale=1)
+                gr.HTML("")   # spacer
+            chat_title = gr.HTML()
+            chat_vec = gr.HTML()
+            # Gradio 6 speaks the {"role","content"} message format natively —
+            # no type="messages" to opt into it any more.
+            chatbot = gr.Chatbot(
+                height=520, show_label=False, visible=False,
+                elem_id="nova-chat", avatar_images=(USER_AVATAR, SONIC_AVATAR),
+                placeholder="ask me anything about this paper — I've read every page 📄",
+            )
+            with gr.Row(visible=False) as chat_input_row:
+                chat_input = gr.Textbox(show_label=False, container=False, scale=9,
+                                        placeholder="Ask about this paper…")
+                send_btn = gr.Button("Send", variant="primary", scale=1)
+
+    STAGE_COLS = [boot_col, welcome_col, refining_col, review_col, searching_col, results_col, chat_col]
+
+    # Re-enter the results Column now that the chat components exist, so each
+    # card's "Chat it out" button can wire straight into them.
+    with results_col:
+        @gr.render(inputs=[clusters_state, state], triggers=[clusters_state.change])
+        def draw_results(clusters, st):
+            """Redrawn whenever a search completes. Streamlit rebuilt this grid on
+            every rerun for free; in Gradio the per-card buttons need real event
+            handlers, so the whole thing is (re)declared here."""
+            if not clusters:
+                return
+            for i, cluster in enumerate(clusters):
+                papers = cluster.get("papers") or {}
+                if not papers:
+                    continue
+                accent = CLUSTER_ACCENTS[i % len(CLUSTER_ACCENTS)]
+                gr.HTML(
+                    f'<div class="cluster-head">'
+                    f'  <div class="cluster-bar" style="background:{accent};"></div>'
+                    f'  <div class="cluster-title">{cluster.get("label", "Approach")}</div>'
+                    f'</div>'
+                )
+                if cluster.get("rationale"):
+                    gr.HTML(f'<div class="cluster-why">{cluster["rationale"]}</div>')
+
+                items = list(papers.items())
+                for row_start in range(0, len(items), 2):
+                    with gr.Row(equal_height=True):
+                        for norm_title, record in items[row_start:row_start + 2]:
+                            with gr.Column(elem_classes=["paper-card"]):
+                                gr.HTML(paper_card_html(norm_title, record))
+                                page_url = first_available(record.get("url"))
+                                pdf_url = first_available(record.get("pdf_url"))
+                                gr.HTML(card_links_html(page_url, pdf_url))
+                                # The real PDF is resolved/verified on click (the
+                                # HYBRID deep step), so any link is enough to try.
+                                chat_btn = gr.Button(
+                                    "💬 Chat it out", variant="primary", size="sm",
+                                    interactive=bool(page_url or pdf_url),
+                                )
+                                chat_btn.click(
+                                    open_chat_for(norm_title),
+                                    inputs=[state],
+                                    outputs=[*STAGE_COLS, chat_title, chat_vec, chatbot,
+                                             chat_input_row, state],
+                                ).then(
+                                    prep_chat,
+                                    inputs=[state],
+                                    outputs=[chat_vec, chatbot, chat_input_row, state],
+                                )
+
+    # -----------------------------------------------------------------------
+    # 5. WIRING
+    # -----------------------------------------------------------------------
+    def do_boot():
+        """Runs once per page load, behind the splash.
+
+        Only the agents load synchronously — the chatbot's ~1.5 GB of embedding +
+        cross-encoder weights warm on a background thread. The Streamlit build
+        blocked on both, which is why it sat on this splash forever on a small host.
+        """
+        yield (*_stages("boot"), boot_html("Waking up SONIC — loading the research + reading models…", 35))
+        try:
+            load_agents()
+        except Exception as e:
+            yield (*_stages("boot"),
+                   f'<div class="nova-error">SONIC couldn\'t wake up: {type(e).__name__}: {e}<br>'
+                   f'Check that GROQ_API_KEY / SECOND_GROQ_API_KEY / TAVILY_API_KEY are set.</div>')
+            return
+        warm_chatbot_models_async()
+        yield (*_stages("welcome"), "")
+
+    demo.load(do_boot, outputs=[*STAGE_COLS, boot_panel])
+
+    def go(query, st):
+        """WELCOME -> REFINING -> REVIEW. Runs the INTENT graph up to its
+        human-review interrupt, then hands the framed sections to the form."""
+        if not (query or "").strip():
+            gr.Warning("Give me something to work with first 🙂")
+            yield (*_stages("welcome"), gr.update(), gr.update(), gr.update(), st)
+            return
+
+        st["user_query"] = query.strip()
+        st["run_id"] = str(uuid.uuid4())
+        yield (*_stages("refining"), gr.update(), gr.update(), gr.update(), st)
+
+        intent_graph, _ = load_agents()
+        config = {"configurable": {"thread_id": st["run_id"]}}
+        try:
+            result = intent_graph.invoke({"user_query": st["user_query"], "run_id": st["run_id"]},
+                                         config=config)
+            payload = result["__interrupt__"][0].value
+            problem, objective, context = split_intent_sections(payload["polished_research_intent"])
+        except Exception as e:
+            gr.Warning(f"Something went sideways: {type(e).__name__}: {e}")
+            yield (*_stages("welcome"), gr.update(), gr.update(), gr.update(), st)
+            return
+
+        st["problem"], st["objective"], st["context"] = problem, objective, context
+        yield (*_stages("review"), problem, objective, context, st)
+
+    go_btn.click(go, inputs=[query_box, state],
+                 outputs=[*STAGE_COLS, problem_box, objective_box, context_box, state])
+
+    def find(problem, objective, context, st):
+        """REVIEW -> SEARCHING -> RESULTS. Resumes the INTENT graph past its
+        interrupt, then STREAMS the SEARCH + CLUSTER graph so the checklist ticks
+        each step off live instead of hanging on one spinner."""
+        from langgraph.types import Command
+
+        if not (problem or "").strip() or not (objective or "").strip():
+            gr.Warning("Problem and Objective can't be empty.")
+            yield (*_stages("review"), gr.update(), gr.update(), st, gr.update())
+            return
+
+        st["problem"], st["objective"], st["context"] = problem, objective, context
+        yield (*_stages("searching"), search_steps_html(set(), {}), gr.update(), st, gr.update())
+
+        intent_graph, search_graph = load_agents()
+        config = {"configurable": {"thread_id": st["run_id"]}}
+        edited_intent = join_intent_sections(problem, objective, context)
+
+        try:
+            resume_result = intent_graph.invoke(Command(resume=edited_intent), config=config)
+            human_verified_intent = resume_result["human_verified_intent"]
+
+            completed, counts, final_state = set(), {}, {}
+            source_field = {"arxiv": "arXiv_paper", "semantic_scholar": "Semantic_Scholar_paper",
+                            "open_alex": "Open_Alex_paper"}
+            for update in search_graph.stream(
+                {"ResearchIntent": human_verified_intent, "run_id": st["run_id"]},
+                stream_mode="updates",
+            ):
+                for node_name, delta in update.items():
+                    completed.add(node_name)
+                    if isinstance(delta, dict):
+                        final_state.update(delta)
+                        if node_name in source_field:
+                            counts[node_name] = len(delta.get(source_field[node_name]) or [])
+                yield (*_stages("searching"), search_steps_html(completed, counts),
+                       gr.update(), st, gr.update())
+        except Exception as e:
+            gr.Warning(f"Something went sideways: {type(e).__name__}: {e}")
+            yield (*_stages("review"), gr.update(), gr.update(), st, gr.update())
+            return
+
+        clusters = final_state.get("clustered_papers") or []
+        # flatten every paper into a lookup keyed by its normalized_title (the
+        # cluster dict key) so "Chat it out" can find the record anywhere.
+        papers_by_key = {}
+        for cluster in clusters:
+            for norm_title, record in (cluster.get("papers") or {}).items():
+                papers_by_key[norm_title] = record
+
+        st["clusters"] = clusters
+        st["papers_by_key"] = papers_by_key
+        st["source_status"] = {
+            "arXiv": final_state.get("arxiv_status") or {},
+            "Semantic Scholar": final_state.get("semantic_scholar_status") or {},
+            "OpenAlex": final_state.get("open_alex_status") or {},
+        }
+
+        total = sum(len(c.get("papers") or {}) for c in clusters)
+        if total == 0:
+            head = sonic_says("hmm, I couldn't pull solid matches for that one — see the source status below. "
+                              "If a source is rate-limited, that's usually why. Try again in a bit, or "
+                              "loosen the framing.")
+        else:
+            head = sonic_says(f"these are the best matches 🎯<br>{total} papers, grouped into "
+                              f"{len(clusters)} approaches. Hit <b>Chat it out</b> on any paper to "
+                              f"actually talk to it.")
+        head += source_status_html(st["source_status"])
+
+        yield (*_stages("results"), gr.update(), head, st, list(clusters))
+
+    find_btn.click(find, inputs=[problem_box, objective_box, context_box, state],
+                   outputs=[*STAGE_COLS, search_panel, results_head, state, clusters_state])
+
+    def start_over():
+        """Full memory refresh: clear this session's results + open chats, and wipe
+        the on-disk vectorstore/PDF caches too."""
+        wipe_disk_cache()
+        return (*_stages("welcome"), "", new_state(), [])
+
+    over_btn.click(start_over, outputs=[*STAGE_COLS, query_box, state, clusters_state])
+    new_search_btn.click(start_over, outputs=[*STAGE_COLS, query_box, state, clusters_state])
+
+    def answer(message, st):
+        """Stream one grounded answer, then append the page citations."""
+        from langchain_core.messages import AIMessage, HumanMessage
+        from Qa import format_docs
+
+        message = (message or "").strip()
+        if not message or not st.get("active_chat"):
+            yield gr.update(), ""
+            return
+
+        session = st["chats"][st["active_chat"]]
+        session["messages"].append({"role": "user", "content": message})
+        yield [dict(m) for m in session["messages"]], ""
+
+        # LangChain chat history from prior turns (excludes the just-added question)
+        history = [HumanMessage(content=m["content"]) if m["role"] == "user"
+                   else AIMessage(content=m["content"])
+                   for m in session["messages"][:-1]]
+
+        try:
+            docs = session["retriever"].invoke(message)
+            context = format_docs(docs)
+            session["messages"].append({"role": "assistant", "content": ""})
+            for chunk in session["chain"].stream(
+                {"question": message, "chat_history": history, "context": context}
+            ):
+                if chunk.content:
+                    session["messages"][-1]["content"] += chunk.content
+                    yield [dict(m) for m in session["messages"]], ""
+            pages = sorted({f"p.{d.metadata.get('page')}" for d in docs})
+            if pages:
+                session["messages"][-1]["content"] += "\n\n*sources: " + ", ".join(pages) + "*"
+        except Exception as e:
+            session["messages"].append(
+                {"role": "assistant",
+                 "content": f"Sorry — I hit an error answering that: {type(e).__name__}: {e}"}
+            )
+        yield [dict(m) for m in session["messages"]], ""
+
+    for trigger in (chat_input.submit, send_btn.click):
+        trigger(answer, inputs=[chat_input, state], outputs=[chatbot, chat_input])
+
+    def back_to_papers(st):
+        st["active_chat"] = None
+        return (*_stages("results"), st)
+
+    back_btn.click(back_to_papers, inputs=[state], outputs=[*STAGE_COLS, state])
+
+
+if __name__ == "__main__":
+    demo.queue(default_concurrency_limit=4).launch(
+        theme=NOVA_THEME, css=CSS, js=FORCE_DARK,
+        server_name="0.0.0.0", server_port=7860,
+    )

requirements.txt

@@ -0,0 +1,51 @@
+# NOVA = research pipeline (agent) + single-PDF Q&A (chatbot) + Gradio UI.
+# One environment runs all three. Versions below match what both sub-projects
+# were already pinned to, so nothing gets upgraded out from under them.
+
+# --- ZeroGPU ---
+# `spaces` provides the @spaces.GPU decorator. ZeroGPU refuses to start a Space
+# that has none ("No @spaces.GPU function detected during startup"), and it only
+# attaches a GPU for the duration of such a call. See ui/gpu.py for the one
+# function that uses it. Off ZeroGPU the decorator is a transparent no-op, so
+# this package is harmless locally and on CPU hardware.
+spaces
+
+# --- torch ---
+# 2.11.0 exactly: ZeroGPU accepts only 2.11.0 / 2.10.0 / 2.9.1 / 2.8.0 and fails
+# the build at config-check otherwise. 2.11.0 is the newest it allows.
+#
+# Plain (CUDA) build on purpose — ZeroGPU rejects the `+cpu` local version, since
+# it needs a CUDA-capable torch to hand the GPU over. That means ~2.5 GB of
+# nvidia-* wheels; unavoidable on this hardware.
+#
+# For CPU-only hardware (Spaces "CPU basic", or a local box), these lines are
+# smaller and faster and still work — NOVA falls back to CPU automatically:
+#     --extra-index-url https://download.pytorch.org/whl/cpu
+#     torch==2.11.0+cpu ; sys_platform == "linux"
+#     torch ; sys_platform != "linux"
+torch==2.11.0
+
+# --- UI ---
+# 6.0 is a hard floor, not caution: launch() takes theme/css/js (Blocks did in 5),
+# and Chatbot speaks {"role","content"} natively (5 needed type="messages").
+gradio>=6.0
+
+# --- research agent (intent + search/cluster graphs) ---
+langgraph
+langchain-core
+langchain-groq
+langchain-tavily
+sentence-transformers          # SPECTER reranker in the search graph
+python-dotenv
+pydantic
+requests
+numpy
+
+# --- chatbot (single-PDF Q&A) ---
+langchain
+langchain-community            # HuggingFaceCrossEncoder, HuggingFaceBgeEmbeddings
+langchain-classic              # CrossEncoderReranker + ContextualCompressionRetriever
+langchain-text-splitters
+langchain-chroma
+chromadb
+pymupdf                        # `import fitz`

ui/__init__.py

@@ -0,0 +1,6 @@
+"""NOVA's Gradio UI layer.
+
+Everything in here is presentation + wiring. The two backend sub-projects --
+`app/` (the research pipeline) and `chatbot_core/` (the single-PDF Q&A) -- are
+imported and driven, never modified.
+"""

ui/agents.py

@@ -0,0 +1,71 @@
+"""
+ui/agents.py
+------------
+Heavy model / graph loading, done once per server process and shared by every
+browser session.
+
+Streamlit gave this to us for free via @st.cache_resource. Gradio has no
+equivalent, and it serves requests from a thread pool, so a naive "load if
+None" would let two simultaneous first-visitors each start a ~2 GB model load.
+Hence the explicit double-checked locking below: the lock is held across the
+load, and a second caller blocks and then sees the finished object.
+"""
+
+import threading
+
+_agents_lock = threading.Lock()
+_agents = None
+
+_chatbot_lock = threading.Lock()
+_chatbot_models = None
+
+
+def load_agents():
+    """Import the compiled LangGraph agents. Importing the search graph also
+    loads the SPECTER reranker model at module import time (by design)."""
+    global _agents
+    if _agents is None:
+        with _agents_lock:
+            if _agents is None:
+                from app.modules.intent.graph import graph as intent_graph
+                from app.modules.search.graph import graph as search_graph
+                _agents = (intent_graph, search_graph)
+    return _agents
+
+
+def warm_chatbot_models():
+    """Pre-load the chatbot's embedding + cross-encoder models so the first
+    'Chat it out' click doesn't pay the model-load cost. We instantiate the
+    exact models the chatbot uses (BAAI/bge-base-en-v1.5 + BAAI/bge-reranker-base),
+    warming the weights into the HF/torch cache.
+
+    Both are pinned to CPU: this runs at boot, outside any ZeroGPU window, and
+    its whole job is to pull weights down — the GPU copy is made later, inside
+    ui.gpu.vectorize_on_gpu, from the same warmed cache."""
+    global _chatbot_models
+    if _chatbot_models is None:
+        with _chatbot_lock:
+            if _chatbot_models is None:
+                from vectorizeer import get_embeddings
+                from langchain_community.cross_encoders import HuggingFaceCrossEncoder
+                embeddings = get_embeddings(device="cpu")
+                reranker = HuggingFaceCrossEncoder(
+                    model_name="BAAI/bge-reranker-base",
+                    model_kwargs={"device": "cpu"},
+                )
+                _chatbot_models = (embeddings, reranker)
+    return _chatbot_models
+
+
+def warm_chatbot_models_async():
+    """Kick the chatbot warm-up onto a daemon thread.
+
+    The Streamlit app warmed BOTH model sets behind one blocking splash, which
+    meant nobody saw a usable page until ~2 GB of weights had downloaded. Only
+    the agents are needed to act on the very first click, so we block on those
+    and let the chatbot models finish in the background — they have until the
+    user has framed an intent, run a search, and picked a paper, which is far
+    longer than the load takes. ensure_chat_ready() calls warm_chatbot_models()
+    anyway, so if the thread hasn't finished it simply blocks on the same lock.
+    """
+    threading.Thread(target=warm_chatbot_models, daemon=True).start()

ui/chat_engine.py

@@ -0,0 +1,148 @@
+"""
+ui/chat_engine.py
+-----------------
+Chat-prep: turning a paper record into a retriever + chain, in three phases, with
+SONIC's progress bar and pep-quotes animating over the slow bits.
+
+`prepare_chat_stream()` is the public entrypoint. It's a generator so the Gradio
+handler can relay each frame straight to the page.
+
+The phase split exists because of ZeroGPU. Each phase wants something different:
+
+  1. fetch      — network-bound. No GPU. Threaded, so the bar animates.
+  2. vectorize  — the expensive encode. THE one GPU step (see ui/gpu.py).
+  3. assemble   — re-open the persisted store on CPU + build the chain. Fast.
+
+Phase 2 is the awkward one: ZeroGPU dispatches to its GPU worker from the calling
+thread, so it must NOT run on a thread we spawned ourselves — which is precisely
+what the original single-threaded-worker design did. On ZeroGPU we therefore call
+it inline and the bar holds still for the few seconds it takes. Off ZeroGPU
+there's no such constraint and the encode is slow (tens of seconds on CPU), so we
+keep the old threaded animation. Same UX where it matters, correctness where it
+counts.
+"""
+
+import random
+import threading
+import time
+
+from ui.constants import SONIC_QUOTES
+from ui.gpu import ON_ZEROGPU, vectorize_on_gpu
+from ui.papers import resolve_and_download_pdf
+from ui.sonic import SONIC_DATA_URI
+
+
+def vec_loading_html(quote: str, phase: str, pct: int) -> str:
+    return (
+        f'<div class="vec-wrap">'
+        f'  <div class="vec-figure"><img src="{SONIC_DATA_URI}" alt="SONIC"/></div>'
+        f'  <div class="vec-quote"><span class="q">SONIC:</span> &ldquo;{quote}&rdquo;</div>'
+        f'  <div class="vec-bar"><div class="vec-fill" style="width:{pct}%"></div></div>'
+        f'  <div class="vec-phase">{phase}</div>'
+        f'</div>'
+    )
+
+
+def _quote_at(quotes, start):
+    """Rotate every ~4s, as the Streamlit build did."""
+    return quotes[int((time.time() - start) // 4) % len(quotes)]
+
+
+# --- phase workers (touch no UI) -------------------------------------------
+
+def _download_blocking(record: dict, holder: dict):
+    try:
+        pdf_path = resolve_and_download_pdf(record)
+        if not pdf_path:
+            holder["error"] = ("Couldn't find a readable open-access PDF for this paper — it may be "
+                               "paywalled. Use the 📄 Paper button to read it on the source site.")
+            return
+        holder["pdf_path"] = pdf_path
+    except Exception as e:
+        holder["error"] = f"Couldn't fetch this paper: {type(e).__name__}: {e}"
+
+
+def _vectorize_blocking(pdf_path: str, holder: dict):
+    try:
+        vectorize_on_gpu(pdf_path)
+    except Exception as e:
+        holder["error"] = f"Couldn't read this paper: {type(e).__name__}: {e}"
+
+
+def _assemble_session(pdf_path: str) -> dict:
+    """Re-open the vectorstore on CPU and build the retriever + chain.
+
+    Cheap: phase 2 already persisted the vectors, so build_vectorstore
+    short-circuits to a plain load. Everything here is CPU-resident by design —
+    it outlives the GPU window (see ui/gpu.py).
+    """
+    from vectorizeer import build_vectorstore
+    from Qa import build_chain, get_llm, get_retriever
+    vs = build_vectorstore(pdf_path)
+    return {"retriever": get_retriever(vs), "chain": build_chain(get_llm())}
+
+
+# --- the stream -------------------------------------------------------------
+
+def prepare_chat_stream(record: dict):
+    """Yield (html, done, session, error) frames while the paper is fetched and
+    vectorized. Every frame before the last has done=False; the final frame
+    carries either a session or an error."""
+    quotes = random.sample(SONIC_QUOTES, len(SONIC_QUOTES))
+    start = time.time()
+    holder: dict = {}
+
+    def fail(msg):
+        return vec_loading_html(_quote_at(quotes, start), "…", 100), True, None, msg
+
+    # 1. FETCH — threaded so the bar moves while the network does its thing.
+    worker = threading.Thread(target=_download_blocking, args=(record, holder), daemon=True)
+    worker.start()
+    pct = 6
+    # Emit one frame up front: a cached PDF can finish before the first is_alive()
+    # check, and without this the panel would sit blank until the next phase.
+    yield vec_loading_html(_quote_at(quotes, start), "Fetching the PDF…", pct), False, None, None
+    while worker.is_alive():
+        pct = min(pct + 2, 44)
+        yield vec_loading_html(_quote_at(quotes, start), "Fetching the PDF…", pct), False, None, None
+        time.sleep(0.35)
+    worker.join()
+    if holder.get("error"):
+        yield fail(holder["error"])
+        return
+    pdf_path = holder["pdf_path"]
+
+    # 2. VECTORIZE — the GPU step.
+    if ON_ZEROGPU:
+        # Must run on this thread: ZeroGPU hands the GPU to the caller, and a
+        # thread we spawned isn't one. It's only a few seconds on a GPU, so the
+        # bar simply holds rather than animating.
+        yield vec_loading_html(_quote_at(quotes, start), "Reading & vectorizing every page…", 55), False, None, None
+        _vectorize_blocking(pdf_path, holder)
+        if holder.get("error"):
+            yield fail(holder["error"])
+            return
+    else:
+        # No such constraint on CPU — and here the encode is genuinely slow, so
+        # the animation earns its keep.
+        worker = threading.Thread(target=_vectorize_blocking, args=(pdf_path, holder), daemon=True)
+        worker.start()
+        while worker.is_alive():
+            pct = min(pct + 2, 88)
+            yield vec_loading_html(_quote_at(quotes, start),
+                                   "Reading & vectorizing every page…", pct), False, None, None
+            time.sleep(0.35)
+        worker.join()
+        if holder.get("error"):
+            yield fail(holder["error"])
+            return
+
+    # 3. ASSEMBLE — CPU, fast.
+    yield vec_loading_html(_quote_at(quotes, start), "Almost there…", 94), False, None, None
+    try:
+        session = _assemble_session(pdf_path)
+    except Exception as e:
+        yield fail(f"Couldn't prepare this paper for chat: {type(e).__name__}: {e}")
+        return
+
+    yield vec_loading_html(quotes[0], "Ready.", 100), True, session, None

ui/constants.py

@@ -0,0 +1,23 @@
+"""
+ui/constants.py
+---------------
+Static data shared across views: cluster accent colors, per-source badge
+colors, and SONIC's motivational one-liners shown while a paper is being
+vectorized.
+"""
+
+# Decorative, index-based — purely visual.
+CLUSTER_ACCENTS = ["#7c5cff", "#22d3ee", "#f59e0b", "#ec4899", "#34d399", "#60a5fa", "#a78bfa", "#f472b6"]
+SOURCE_COLORS = {"arXiv": "#e05263", "SemanticScholar": "#4c8bf5", "OpenAlex": "#2fa572"}
+
+# SONIC's motivational one-liners, shown while a paper is being vectorized.
+SONIC_QUOTES = [
+    "Great research isn't found — it's framed. You already did the hard part.",
+    "Every paper you read is a shortcut someone left for you. Let's decode this one.",
+    "Reading fast is fine. Understanding deeply is the flex. Hang tight — I'm doing the deep part.",
+    "The best researchers ask better questions, not more of them. Get yours ready.",
+    "Curiosity is a muscle. You're clearly training it. Almost there…",
+    "One good paper can change the whole direction. Could be this one.",
+    "I'm turning every page into something you can just… ask. Two seconds.",
+    "Knowledge compounds. This is you, quietly getting sharper.",
+]

ui/gpu.py

@@ -0,0 +1,54 @@
+"""
+ui/gpu.py
+---------
+ZeroGPU wiring — the only place in NOVA that touches a GPU.
+
+HF's ZeroGPU hardware hands a Space a GPU *only* for the duration of a call to an
+@spaces.GPU-decorated function, and refuses to boot at all if it can't find one
+at import time ("No @spaces.GPU function detected during startup"). That single
+constraint drives the whole design here.
+
+NOVA is CPU-first and stays that way. Exactly one operation runs on the GPU: the
+bulk embedding of a paper's chunks during vectorizing, which is by far the
+slowest thing in the app (tens of seconds on CPU, a few on GPU). Everything else
+— SPECTER reranking, query embedding, the cross-encoder — is pinned to CPU *on
+purpose*, because those run outside any GPU window and a cuda-resident model
+there would fail on first use. That's why the three backend call sites now take
+an explicit `device` instead of auto-detecting.
+
+Off ZeroGPU (local, or Spaces "CPU basic") @spaces.GPU is a transparent
+passthrough and ON_ZEROGPU is False, so this module quietly degrades to plain
+CPU work and nothing else in the app changes.
+"""
+
+import os
+
+import spaces
+
+# Set by the ZeroGPU runtime; `spaces.config` reads the same variable.
+ON_ZEROGPU = os.getenv("SPACES_ZERO_GPU", "").lower() in ("1", "t", "true")
+
+# The device to use *inside* a GPU window. Outside one, always "cpu".
+GPU_DEVICE = "cuda" if ON_ZEROGPU else "cpu"
+
+# Generous but bounded. The window has to cover PDF text extraction and chunking
+# (CPU work that unavoidably happens inside build_vectorstore) plus the encode
+# itself. A long paper on a cold cache is the worst case.
+_VECTORIZE_SECONDS = 120
+
+
+@spaces.GPU(duration=_VECTORIZE_SECONDS)
+def vectorize_on_gpu(pdf_path: str) -> None:
+    """Build and persist this paper's vectorstore with the embedder on GPU.
+
+    Returns None deliberately. ZeroGPU runs this in its own GPU worker, so a
+    Chroma handle created here would carry a cuda-resident embedding model back
+    to a caller that no longer holds the GPU — useless at best, a crash at worst.
+    What crosses the boundary is the *persisted vectorstore on disk*, which is
+    device-independent.
+
+    The caller then re-opens it on CPU, which costs nothing: build_vectorstore
+    short-circuits to a plain load as soon as the persist dir exists.
+    """
+    from vectorizeer import build_vectorstore
+    build_vectorstore(pdf_path, device=GPU_DEVICE)

ui/intent_text.py

@@ -0,0 +1,33 @@
+"""
+ui/intent_text.py
+-----------------
+INTENT text <-> 3-field form conversion.
+
+The backend hands us ONE string with three headed sections. We split it into
+three editable fields and MUST rebuild the exact same "Problem: / Objective:
+/ Additional Context:" shape before sending it into the search graph -- its
+prompts assume that literal structure.
+"""
+
+import re
+
+_SECTION_PATTERN = re.compile(
+    r"Problem:\s*(.*?)\n\s*Objective:\s*(.*?)\n\s*Additional Context:\s*(.*)",
+    re.S,
+)
+
+
+def split_intent_sections(text: str):
+    m = _SECTION_PATTERN.search(text or "")
+    if m:
+        return m.group(1).strip(), m.group(2).strip(), m.group(3).strip()
+    return (text or "").strip(), "", ""
+
+
+def join_intent_sections(problem: str, objective: str, additional_context: str) -> str:
+    ac = additional_context.strip() or "None specified."
+    return (
+        f"Problem:\n{problem.strip()}\n\n"
+        f"Objective:\n{objective.strip()}\n\n"
+        f"Additional Context:\n{ac}"
+    )

ui/papers.py

@@ -0,0 +1,89 @@
+"""
+ui/papers.py
+------------
+Paper record + PDF helpers: pulling a display value out of a per-source
+dict, formatting an author line, and robustly resolving + downloading a real
+PDF for a paper at "Chat it out" time.
+"""
+
+import hashlib
+
+import requests
+
+from ui.paths import DOWNLOADS_DIR, PDF_UA
+
+
+def first_available(d):
+    """First non-empty value in a {source: value} dict (record fields are
+    dict-keyed by source after aggregation)."""
+    if not isinstance(d, dict):
+        return d or None
+    for v in d.values():
+        if v:
+            return v
+    return None
+
+
+def author_line(authors, year):
+    authors = authors or []
+    if authors:
+        shown = ", ".join(authors[:4]) + (" et al." if len(authors) > 4 else "")
+    else:
+        shown = "Unknown authors"
+    return f"{shown}  ·  {year or 'n.d.'}"
+
+
+_PDF_MAGIC = b"%PDF"
+
+
+def _download_if_pdf(url: str) -> "str | None":
+    """Download url, but only keep it if the bytes are a real PDF (starts with
+    %PDF) — a best-effort link that's actually an HTML landing page returns None
+    so we can fall back to deep resolution. Cached by URL hash."""
+    if not url:
+        return None
+    key = hashlib.md5(url.encode("utf-8")).hexdigest()[:16]
+    dest = DOWNLOADS_DIR / f"{key}.pdf"
+    if dest.is_file() and dest.stat().st_size > 0:
+        return str(dest)
+    try:
+        resp = requests.get(url, headers=PDF_UA, timeout=45, stream=True, allow_redirects=True)
+        if resp.status_code != 200:
+            return None
+        it = resp.iter_content(chunk_size=32768)
+        first = next(it, b"")
+        if not first.startswith(_PDF_MAGIC):
+            return None  # HTML landing page or something else, not a PDF
+        with open(dest, "wb") as f:
+            f.write(first)
+            for chunk in it:
+                if chunk:
+                    f.write(chunk)
+        return str(dest) if dest.stat().st_size > 0 else None
+    except requests.RequestException:
+        return None
+
+
+def resolve_and_download_pdf(record: dict) -> "str | None":
+    """Robustly obtain a readable PDF for a paper at 'Chat it out' time (the
+    HYBRID deep step). Search only stored a cheap best-effort link; here we:
+       1) try each best-effort pdf link directly (keep it only if it's a real PDF);
+       2) if those are landing pages / dead, scrape the citation_pdf_url meta tag
+          off them and off the paper's source page(s), then download + verify.
+    Returns a local path, or None if nothing yields real PDF bytes."""
+    from app.modules.search.providers.semantic_scholar import _extract_citation_pdf_url
+
+    pdf_candidates = [v for v in (record.get("pdf_url") or {}).values() if v]
+    page_candidates = [v for v in (record.get("url") or {}).values() if v]
+
+    for cand in pdf_candidates:                       # 1) direct best-effort PDFs
+        path = _download_if_pdf(cand)
+        if path:
+            return path
+    for page in pdf_candidates + page_candidates:     # 2) deep: scrape landing pages
+        scraped = _extract_citation_pdf_url(page)
+        if scraped:
+            path = _download_if_pdf(scraped)
+            if path:
+                return path
+    return None

ui/paths.py

@@ -0,0 +1,38 @@
+"""
+ui/paths.py
+-----------
+Filesystem + import-path wiring for NOVA.
+
+This module MUST be the first `ui` import in nova_app.py. It does the sys.path
++ chdir + .env side effects that make `import app...` (the agent package) and
+`from vectorizeer import ...` / `from Qa import ...` (the chatbot) resolve
+later -- so it has to run before any other `ui` module that needs BASE,
+DOWNLOADS_DIR, VECTORSTORES_DIR, or PDF_UA. Once nova_app.py imports this
+first, every other module gets the same path setup for free via Python's module
+cache (re-importing `ui.paths` elsewhere just returns the already-initialized
+module, it doesn't re-run this file).
+"""
+
+import os
+import sys
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# PATHS  — make both sub-projects importable without touching their code
+# ---------------------------------------------------------------------------
+BASE = Path(__file__).resolve().parent.parent
+os.chdir(BASE)                                   # ./vectorstores, ./downloads resolve under NOVA/
+sys.path.insert(0, str(BASE))                    # `import app...`  (the agent package)
+sys.path.insert(0, str(BASE / "chatbot_core"))   # `import Qa`, `from vectorizeer import ...`
+
+from dotenv import load_dotenv
+load_dotenv(BASE / ".env")
+
+DOWNLOADS_DIR = BASE / "downloads"
+DOWNLOADS_DIR.mkdir(exist_ok=True)
+VECTORSTORES_DIR = BASE / "vectorstores"
+
+ASSETS_DIR = BASE / "assets"
+ASSETS_DIR.mkdir(exist_ok=True)
+
+PDF_UA = {"User-Agent": "Mozilla/5.0 (compatible; NOVA-research/1.0)"}

ui/search_progress.py

@@ -0,0 +1,41 @@
+"""
+ui/search_progress.py
+---------------------
+Live checklist renderer for the SEARCHING stage: turns the set of completed
+LangGraph node names + per-source paper counts into the step-by-step HTML
+checklist markup.
+"""
+
+_SEARCH_STEPS = [
+    ("arxiv", "Searching arXiv"),
+    ("semantic_scholar", "Searching Semantic Scholar"),
+    ("open_alex", "Searching OpenAlex"),
+    ("aggregation", "Merging & deduplicating"),
+    ("reranker", "Re-ranking by relevance (SPECTER)"),
+    ("clustering", "Clustering by approach"),
+]
+
+
+def search_steps_html(completed: set, counts: dict) -> str:
+    sources_done = all(s in completed for s in ("arxiv", "semantic_scholar", "open_alex"))
+
+    def state_of(key):
+        if key in completed:
+            return "done"
+        if key in ("arxiv", "semantic_scholar", "open_alex"):
+            return "active"
+        if key == "aggregation":
+            return "active" if sources_done else "pending"
+        if key == "reranker":
+            return "active" if "aggregation" in completed else "pending"
+        if key == "clustering":
+            return "active" if "reranker" in completed else "pending"
+        return "pending"
+
+    rows = []
+    for key, label in _SEARCH_STEPS:
+        s = state_of(key)
+        ic = "✓" if s == "done" else ("•" if s == "pending" else "")
+        cnt = f'<span class="step-count">{counts[key]} papers</span>' if key in counts else ""
+        rows.append(f'<div class="step-row {s}"><div class="step-ic">{ic}</div><div>{label}</div>{cnt}</div>')
+    return '<div class="search-wrap">' + "".join(rows) + "</div>"

ui/sonic.py

@@ -0,0 +1,128 @@
+"""
+ui/sonic.py
+-----------
+SONIC — the assistant caricature, drawn as an inline SVG (no external asset to
+fetch, so it renders inside Gradio's HTML blocks with no network round-trip).
+A friendly bespectacled researcher: brown quiff, black glasses, big smile,
+white shirt + navy striped tie, suspenders. Used big on the welcome screen and
+small as the chat avatar.
+
+Two exports, for two different consumers:
+  SONIC_DATA_URI  -> for <img src="..."> inside gr.HTML markup
+  SONIC_AVATAR    -> a real file on disk, because gr.Chatbot's avatar_images
+                     takes a path/URL and will not accept a data: URI
+"""
+
+import base64
+
+from ui.paths import ASSETS_DIR
+
+SONIC_SVG = """
+<svg viewBox="0 0 240 280" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="SONIC the research assistant">
+  <defs>
+    <linearGradient id="s-card" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0" stop-color="#2b2168"/><stop offset="1" stop-color="#141a2e"/>
+    </linearGradient>
+    <linearGradient id="s-skin" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0" stop-color="#f8d0a8"/><stop offset="1" stop-color="#e7ad7f"/>
+    </linearGradient>
+    <linearGradient id="s-hair" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0" stop-color="#7d5636"/><stop offset="1" stop-color="#4e3421"/>
+    </linearGradient>
+    <linearGradient id="s-tie" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0" stop-color="#3a4f8a"/><stop offset="1" stop-color="#26325c"/>
+    </linearGradient>
+    <clipPath id="s-clip"><rect x="8" y="8" width="224" height="264" rx="34"/></clipPath>
+    <clipPath id="s-tieclip"><path d="M115 232 L125 232 L131 279 L120 289 L109 279 Z"/></clipPath>
+  </defs>
+  <rect x="8" y="8" width="224" height="264" rx="34" fill="url(#s-card)"/>
+  <g clip-path="url(#s-clip)">
+    <!-- shirt -->
+    <path d="M18 280 C22 226 64 202 120 202 C176 202 218 226 222 280 Z" fill="#eef2f8"/>
+    <!-- suspenders -->
+    <path d="M80 205 L96 205 L112 280 L96 280 Z" fill="#b45540"/>
+    <path d="M160 205 L144 205 L128 280 L144 280 Z" fill="#b45540"/>
+    <!-- collar -->
+    <path d="M104 200 L120 217 L98 226 Z" fill="#ffffff"/>
+    <path d="M136 200 L120 217 L142 226 Z" fill="#ffffff"/>
+    <!-- tie -->
+    <path d="M120 214 L110 223 L120 234 L130 223 Z" fill="url(#s-tie)"/>
+    <path d="M115 232 L125 232 L131 279 L120 289 L109 279 Z" fill="url(#s-tie)"/>
+    <g clip-path="url(#s-tieclip)" stroke="#5a70b4" stroke-width="4" opacity="0.8">
+      <line x1="104" y1="250" x2="136" y2="218"/>
+      <line x1="104" y1="264" x2="140" y2="228"/>
+      <line x1="108" y1="280" x2="140" y2="248"/>
+    </g>
+    <!-- neck -->
+    <rect x="105" y="156" width="30" height="48" rx="14" fill="#e6ad7f"/>
+    <!-- ears -->
+    <circle cx="66" cy="118" r="11" fill="url(#s-skin)"/>
+    <circle cx="174" cy="118" r="11" fill="url(#s-skin)"/>
+    <!-- head -->
+    <ellipse cx="120" cy="112" rx="56" ry="62" fill="url(#s-skin)"/>
+    <!-- cheeks -->
+    <ellipse cx="84" cy="140" rx="10" ry="6" fill="#ef9f8f" opacity="0.35"/>
+    <ellipse cx="156" cy="140" rx="10" ry="6" fill="#ef9f8f" opacity="0.35"/>
+    <!-- hair -->
+    <path d="M66 110 C62 54 92 38 120 38 C150 38 178 54 173 110
+             C167 92 155 84 140 82 C151 74 154 62 154 62
+             C141 76 129 79 120 79 C110 79 101 74 94 64
+             C96 74 99 82 99 82 C85 84 73 92 66 110 Z" fill="url(#s-hair)"/>
+    <!-- eyebrows -->
+    <path d="M83 97 Q97 90 110 97" stroke="#5c3f28" stroke-width="4" fill="none" stroke-linecap="round"/>
+    <path d="M130 97 Q143 90 157 97" stroke="#5c3f28" stroke-width="4" fill="none" stroke-linecap="round"/>
+    <!-- eyes -->
+    <ellipse cx="101" cy="117" rx="9" ry="10" fill="#ffffff"/>
+    <ellipse cx="139" cy="117" rx="9" ry="10" fill="#ffffff"/>
+    <circle cx="103" cy="118" r="5" fill="#2a2438"/>
+    <circle cx="137" cy="118" r="5" fill="#2a2438"/>
+    <circle cx="105" cy="116" r="1.6" fill="#ffffff"/>
+    <circle cx="139" cy="116" r="1.6" fill="#ffffff"/>
+    <!-- nose -->
+    <path d="M115 145 Q120 149 125 145" stroke="#d99b6f" stroke-width="3" fill="none" stroke-linecap="round"/>
+    <!-- smile -->
+    <path d="M97 156 Q120 186 143 156 Q120 172 97 156 Z" fill="#6d3630"/>
+    <path d="M101 157 Q120 170 139 157 Q120 165 101 157 Z" fill="#ffffff"/>
+    <!-- glasses -->
+    <g fill="none" stroke="#171b28" stroke-width="5">
+      <rect x="82" y="101" width="38" height="32" rx="13" fill="rgba(255,255,255,0.10)"/>
+      <rect x="120" y="101" width="38" height="32" rx="13" fill="rgba(255,255,255,0.10)"/>
+      <line x1="118" y1="115" x2="122" y2="115"/>
+      <line x1="82" y1="110" x2="67" y2="113"/>
+      <line x1="158" y1="110" x2="173" y2="113"/>
+    </g>
+  </g>
+</svg>
+"""
+
+SONIC_DATA_URI = "data:image/svg+xml;base64," + base64.b64encode(SONIC_SVG.encode("utf-8")).decode("ascii")
+
+# gr.Chatbot(avatar_images=...) resolves paths/URLs only, so materialize the SVG.
+SONIC_AVATAR = str(ASSETS_DIR / "sonic.svg")
+(ASSETS_DIR / "sonic.svg").write_text(SONIC_SVG, encoding="utf-8")
+
+USER_SVG = """
+<svg viewBox="0 0 64 64" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="You">
+  <defs>
+    <linearGradient id="u-bg" x1="0" y1="0" x2="1" y2="1">
+      <stop offset="0" stop-color="#1c2540"/><stop offset="1" stop-color="#0f1526"/>
+    </linearGradient>
+  </defs>
+  <rect width="64" height="64" rx="18" fill="url(#u-bg)"/>
+  <circle cx="32" cy="25" r="10" fill="#7c5cff" opacity="0.9"/>
+  <path d="M12 56 C14 42 22 37 32 37 C42 37 50 42 52 56 Z" fill="#22d3ee" opacity="0.75"/>
+</svg>
+"""
+USER_AVATAR = str(ASSETS_DIR / "user.svg")
+(ASSETS_DIR / "user.svg").write_text(USER_SVG, encoding="utf-8")
+
+
+def sonic_says(message: str) -> str:
+    """SONIC's speech bubble. Returns markup for a gr.HTML block (the Streamlit
+    original wrote straight to the page; here the caller decides where it goes)."""
+    return (
+        f'<div class="sonic-row">'
+        f'  <div class="sonic-avatar"><img src="{SONIC_DATA_URI}" alt="SONIC"/></div>'
+        f'  <div class="sonic-bubble"><div class="sonic-name">SONIC</div>{message}</div>'
+        f'</div>'
+    )

ui/theme.py

@@ -0,0 +1,267 @@
+"""
+ui/theme.py
+-----------
+NOVA's look: a Gradio theme object for the component primitives, plus the CSS
+that carries the actual identity.
+
+Ported from the Streamlit build's utils/styles.py — same palette, same brand
+header, same speech bubbles, same cluster/paper cards, same live checklist.
+Gradio hands us a real DOM instead of Streamlit's opaque `data-testid` tree, so
+everything here hooks `elem_classes`/`elem_id` we set ourselves rather than
+selectors that break on the next framework release.
+
+FORCE_DARK is a js hook, not CSS: Gradio decides light/dark from a URL param
+before our stylesheet loads, and the palette below only exists in dark.
+"""
+
+import gradio as gr
+
+NOVA_THEME = gr.themes.Base(
+    primary_hue=gr.themes.Color(
+        c50="#f2efff", c100="#e5dfff", c200="#cabfff", c300="#b09fff",
+        c400="#957fff", c500="#7c5cff", c600="#6344e6", c700="#4a2fbf",
+        c800="#331f99", c900="#1f1273", c950="#0f0847",
+    ),
+    secondary_hue=gr.themes.Color(
+        c50="#ecfeff", c100="#cffafe", c200="#a5f3fc", c300="#67e8f9",
+        c400="#22d3ee", c500="#06b6d4", c600="#0891b2", c700="#0e7490",
+        c800="#155e75", c900="#164e63", c950="#083344",
+    ),
+    neutral_hue=gr.themes.Color(
+        c50="#f2f4fb", c100="#e6e9f2", c200="#cbd0e0", c300="#8b93a7",
+        c400="#5b6377", c500="#3a4157", c600="#252c40", c700="#111828",
+        c800="#0d1322", c900="#080b16", c950="#05070f",
+    ),
+    font=[gr.themes.GoogleFont("Inter"), "system-ui", "sans-serif"],
+    font_mono=[gr.themes.GoogleFont("JetBrains Mono"), "monospace"],
+).set(
+    body_background_fill="#080b16",
+    body_background_fill_dark="#080b16",
+    block_background_fill="transparent",
+    block_background_fill_dark="transparent",
+    block_border_width="0px",
+    block_shadow="none",
+    panel_background_fill="transparent",
+    panel_background_fill_dark="transparent",
+    border_color_primary="rgba(255,255,255,0.08)",
+    border_color_primary_dark="rgba(255,255,255,0.08)",
+    input_background_fill="rgba(8,11,22,0.7)",
+    input_background_fill_dark="rgba(8,11,22,0.7)",
+    input_border_color="rgba(255,255,255,0.08)",
+    input_border_color_dark="rgba(255,255,255,0.08)",
+)
+
+# Gradio reads ?__theme= before our CSS lands; the palette is dark-only, so pin it.
+FORCE_DARK = """
+function() {
+  const url = new URL(window.location);
+  if (url.searchParams.get('__theme') !== 'dark') {
+    url.searchParams.set('__theme', 'dark');
+    window.location.replace(url.href);
+  }
+}
+"""
+
+CSS = """
+:root {
+  --nova-bg: #080b16;
+  --nova-card: #111828;
+  --nova-card-2: #0d1322;
+  --nova-border: rgba(255,255,255,0.08);
+  --nova-text: #e6e9f2;
+  --nova-muted: #8b93a7;
+  --nova-accent: #7c5cff;
+  --nova-accent-2: #22d3ee;
+}
+
+/* ---- page shell: cosmic gradient + a slow drifting aurora ---- */
+gradio-app, .gradio-container {
+  background:
+    radial-gradient(1200px 600px at 12% -10%, rgba(124,92,255,0.18), transparent 55%),
+    radial-gradient(1000px 500px at 100% 0%, rgba(34,211,238,0.12), transparent 50%),
+    var(--nova-bg) !important;
+  max-width: 100% !important;
+  color: var(--nova-text);
+}
+.gradio-container { padding-top: 1.6rem !important; }
+
+/* starfield + aurora live behind everything and never eat clicks */
+gradio-app::before {
+  content: ""; position: fixed; inset: 0; pointer-events: none; z-index: 0;
+  background-image:
+    radial-gradient(1.4px 1.4px at 12% 18%, rgba(255,255,255,.55), transparent),
+    radial-gradient(1.2px 1.2px at 72% 12%, rgba(255,255,255,.42), transparent),
+    radial-gradient(1.6px 1.6px at 38% 62%, rgba(255,255,255,.35), transparent),
+    radial-gradient(1.1px 1.1px at 88% 74%, rgba(255,255,255,.40), transparent),
+    radial-gradient(1.3px 1.3px at 24% 88%, rgba(255,255,255,.30), transparent),
+    radial-gradient(1.2px 1.2px at 58% 34%, rgba(255,255,255,.28), transparent);
+  animation: twinkle 7s ease-in-out infinite alternate;
+}
+@keyframes twinkle { from { opacity: .35; } to { opacity: .9; } }
+
+#nova-root { position: relative; z-index: 1; max-width: 1180px; margin: 0 auto; }
+
+/* hide Gradio's own chrome */
+footer, .built-with, .show-api { display: none !important; }
+
+/* ---- NOVA brand header ---- */
+.nova-brand { display:flex; align-items:center; gap:14px; margin-bottom: 2px; }
+.nova-mark {
+  font-size: 2.1rem; font-weight: 800; letter-spacing: .06em;
+  background: linear-gradient(120deg, #b8a7ff, #7c5cff 45%, #22d3ee);
+  background-size: 200% auto;
+  -webkit-background-clip: text; background-clip: text; -webkit-text-fill-color: transparent;
+  animation: shimmer 6s linear infinite;
+}
+@keyframes shimmer { to { background-position: 200% center; } }
+.nova-star { font-size: 1.7rem; filter: drop-shadow(0 0 8px rgba(124,92,255,.7)); animation: pulse-star 3s ease-in-out infinite; }
+@keyframes pulse-star { 50% { filter: drop-shadow(0 0 16px rgba(124,92,255,1)); transform: scale(1.08); } }
+.nova-sub { color: var(--nova-muted); font-size: .82rem; letter-spacing:.14em; text-transform: uppercase; }
+.sonic-chip {
+  margin-left:auto; display:flex; align-items:center; gap:8px;
+  background: rgba(124,92,255,0.1); border:1px solid rgba(124,92,255,0.35);
+  color:#cdbcff; padding:6px 14px; border-radius:999px; font-size:.78rem; font-weight:600;
+}
+.sonic-dot { width:8px; height:8px; border-radius:50%; background:#22d3ee; box-shadow:0 0 8px #22d3ee; animation: blink 2s ease-in-out infinite; }
+@keyframes blink { 50% { opacity:.35; } }
+
+/* ---- SONIC speech bubble ---- */
+.sonic-row { display:flex; gap:14px; align-items:flex-start; margin: 20px 0 8px; }
+.sonic-avatar {
+  flex:0 0 48px; width:48px; height:48px; border-radius:14px; overflow:hidden;
+  box-shadow: 0 0 22px rgba(124,92,255,.45);
+}
+.sonic-avatar img { width:100%; height:100%; object-fit:cover; display:block; }
+.sonic-bubble {
+  background: var(--nova-card); border:1px solid var(--nova-border);
+  border-radius: 4px 16px 16px 16px; padding: 14px 18px; color: var(--nova-text);
+  font-size: 1.02rem; line-height:1.5; max-width: 760px;
+  animation: bubble-in .32s cubic-bezier(.2,.9,.3,1.2);
+}
+@keyframes bubble-in { from { opacity:0; transform: translateY(6px) scale(.98); } }
+.sonic-name { color:#9d8cff; font-weight:700; font-size:.72rem; letter-spacing:.1em; text-transform:uppercase; margin-bottom:4px; }
+
+/* ---- cluster section header ---- */
+.cluster-head { display:flex; align-items:center; gap:12px; margin: 26px 0 6px; }
+.cluster-bar { width:5px; height:30px; border-radius:3px; }
+.cluster-title { font-size:1.22rem; font-weight:700; color:#f2f4fb; }
+.cluster-why { color:var(--nova-muted); font-size:.88rem; line-height:1.5; margin: 0 0 6px 17px; max-width: 900px; }
+
+/* ---- paper card ---- */
+.paper-card {
+  background: linear-gradient(180deg, var(--nova-card), var(--nova-card-2));
+  border: 1px solid var(--nova-border) !important; border-radius: 16px !important;
+  padding: 16px !important;
+  transition: transform .15s ease, border-color .15s ease, box-shadow .15s ease;
+}
+.paper-card:hover {
+  transform: translateY(-3px); border-color: rgba(124,92,255,0.5) !important;
+  box-shadow: 0 12px 30px rgba(0,0,0,0.4);
+}
+.paper-title { font-size: 1.02rem; font-weight: 700; color:#f2f4fb; line-height:1.35; margin-bottom:6px; }
+.paper-meta { color: var(--nova-muted); font-size: .82rem; margin-bottom: 10px; }
+.src-badge { display:inline-block; padding:2px 9px; border-radius:999px; font-size:.68rem; font-weight:700;
+  margin-right:6px; border:1px solid transparent; }
+
+/* ---- buttons ---- */
+#nova-root button.secondary, #nova-root a.button {
+  border-radius: 10px !important; font-weight: 600 !important; font-size: .84rem !important;
+  border: 1px solid var(--nova-border) !important; background: rgba(255,255,255,0.04) !important;
+  color: #dfe3ee !important; transition: all .14s ease;
+}
+#nova-root button.secondary:hover, #nova-root a.button:hover {
+  border-color: rgba(124,92,255,0.6) !important; background: rgba(124,92,255,0.12) !important; color:#fff !important;
+}
+#nova-root button.primary {
+  background: linear-gradient(120deg, #7c5cff, #22d3ee) !important; border: none !important;
+  color: #0a0e1a !important; font-weight: 800 !important; border-radius: 10px !important;
+  box-shadow: 0 6px 18px rgba(124,92,255,.4) !important; transition: filter .14s ease, transform .14s ease;
+}
+#nova-root button.primary:hover { filter: brightness(1.08); transform: translateY(-1px); }
+#nova-root button:disabled { opacity:.4 !important; cursor: not-allowed !important; }
+
+/* link-buttons rendered as anchors inside cards */
+.card-link {
+  display:inline-flex; align-items:center; justify-content:center; gap:6px;
+  padding:7px 12px; border-radius:10px; font-weight:600; font-size:.8rem; text-decoration:none !important;
+  border:1px solid var(--nova-border); background:rgba(255,255,255,.04); color:#dfe3ee !important;
+  transition: all .14s ease; flex:1; text-align:center;
+}
+.card-link:hover { border-color:rgba(124,92,255,.6); background:rgba(124,92,255,.12); color:#fff !important; }
+.card-link.dead { opacity:.35; pointer-events:none; }
+.card-links { display:flex; gap:8px; margin-top:2px; }
+
+/* ---- inputs ---- */
+#nova-root textarea, #nova-root input[type="text"] {
+  background: rgba(8,11,22,0.7) !important; border:1px solid var(--nova-border) !important;
+  color: var(--nova-text) !important; border-radius: 12px !important; font-size:.95rem !important;
+}
+#nova-root textarea:focus, #nova-root input[type="text"]:focus {
+  border-color: rgba(124,92,255,.6) !important; box-shadow: 0 0 0 3px rgba(124,92,255,.15) !important;
+}
+.field-label { font-weight:700; color:#cbd0e0; margin: 8px 0 2px; font-size:.9rem; }
+
+/* ---- welcome hero ---- */
+.hero-figure { display:flex; flex-direction:column; align-items:center; justify-content:center; }
+.hero-figure img { height:52vh; max-height:540px; width:auto; filter: drop-shadow(0 20px 40px rgba(124,92,231,.35)); animation: float 6s ease-in-out infinite; }
+@keyframes float { 50% { transform: translateY(-12px); } }
+.hero-name { margin-top:14px; font-weight:800; letter-spacing:.12em; color:#c9c3ff; font-size:.9rem; }
+.hero-speech {
+  background: var(--nova-card); border:1px solid var(--nova-border);
+  border-radius: 18px 18px 18px 4px; padding: 20px 22px; color: var(--nova-text);
+  font-size: 1.18rem; line-height:1.55; position:relative; margin-bottom: 18px;
+}
+.hero-speech .sonic-name { color:#9d8cff; font-weight:700; font-size:.72rem; letter-spacing:.12em; text-transform:uppercase; margin-bottom:8px; }
+.hero-answer-label { color:var(--nova-muted); font-size:.82rem; text-transform:uppercase; letter-spacing:.1em; margin: 6px 0 8px; }
+
+/* ---- live search checklist ---- */
+.search-wrap { max-width: 620px; margin: 6px auto 0; }
+.step-row { display:flex; align-items:center; gap:14px; padding:11px 4px; font-size:1rem; color:var(--nova-muted); border-bottom:1px solid rgba(255,255,255,.05); }
+.step-ic { width:26px; height:26px; border-radius:50%; flex:0 0 26px; display:flex; align-items:center; justify-content:center; font-size:.8rem; border:2px solid rgba(255,255,255,.14); }
+.step-row.done { color:#dfe3ee; }
+.step-row.done .step-ic { background:linear-gradient(120deg,#7c5cff,#22d3ee); border-color:transparent; color:#0a0e1a; font-weight:800; }
+.step-row.active { color:#fff; }
+.step-row.active .step-ic { border-color:#7c5cff; border-top-color:transparent; animation: spin .8s linear infinite; }
+.step-count { margin-left:auto; font-size:.8rem; color:var(--nova-muted); }
+@keyframes spin { to { transform: rotate(360deg); } }
+
+/* ---- boot splash + vectorize loading ---- */
+.vec-wrap { max-width: 640px; margin: 10px auto; text-align:center; }
+.vec-figure img { height:150px; width:auto; filter: drop-shadow(0 10px 26px rgba(124,92,231,.4)); animation: float 6s ease-in-out infinite; }
+.vec-quote { color:#cbd0e6; font-size:1.12rem; line-height:1.55; font-style:italic; margin:18px auto 6px; max-width:520px; min-height:3em; }
+.vec-quote .q { color:#9d8cff; font-weight:700; }
+.vec-phase { color:var(--nova-muted); font-size:.85rem; margin-top:10px; }
+.vec-bar { height:6px; border-radius:999px; background:rgba(255,255,255,.07); overflow:hidden; margin:16px auto 0; max-width:520px; }
+.vec-fill { height:100%; border-radius:999px; background:linear-gradient(90deg,#7c5cff,#22d3ee); transition: width .35s ease; }
+
+/* ---- per-source status chips ---- */
+.src-stat-row { display:flex; flex-wrap:wrap; gap:8px; margin: 2px 0 18px; }
+.src-stat { font-size:.74rem; font-weight:600; padding:4px 11px; border-radius:999px; border:1px solid transparent; }
+.src-stat.ok    { color:#5eead4; background:rgba(45,212,191,.10); border-color:rgba(45,212,191,.35); }
+.src-stat.warn  { color:#fbbf24; background:rgba(251,191,36,.12); border-color:rgba(251,191,36,.45); }
+.src-stat.err   { color:#f87171; background:rgba(248,113,113,.12); border-color:rgba(248,113,113,.45); }
+.src-stat.muted { color:#8b93a7; background:rgba(255,255,255,.04); border-color:var(--nova-border); }
+
+/* ---- chat ---- */
+#nova-chat { background: transparent !important; border: none !important; }
+#nova-chat .message-wrap { gap: 14px !important; }
+#nova-chat .message {
+  border-radius: 4px 16px 16px 16px !important; border:1px solid var(--nova-border) !important;
+  background: var(--nova-card) !important; color: var(--nova-text) !important; line-height:1.55 !important;
+}
+#nova-chat .message.user {
+  border-radius: 16px 4px 16px 16px !important;
+  background: rgba(124,92,231,.12) !important; border-color: rgba(124,92,231,.3) !important;
+}
+#nova-chat .avatar-container img { border-radius: 12px !important; box-shadow: 0 0 14px rgba(124,92,255,.35); }
+
+.nova-error {
+  border:1px solid rgba(251,191,36,.45); background:rgba(251,191,36,.10); color:#fbbf24;
+  border-radius:12px; padding:12px 16px; margin:10px 0; font-size:.9rem;
+}
+
+@media (max-width: 860px) {
+  .hero-figure img { height:32vh; }
+  .card-links { flex-wrap: wrap; }
+}
+"""