Initial deployment: NeuroLens cognitive health screening pipeline

README.md

@@ -1,20 +1,196 @@
+# NeuroLens
+
+### Cognitive Health Screening & Coaching Pipeline
+
+NeuroLens is a three-stage system that takes a short conversational language
+assessment, extracts linguistic biomarkers documented in the cognitive-aging
+research literature, and generates a citation-grounded, retrieval-augmented
+prevention coaching summary.
+
+**Live demo:** _add your HuggingFace Spaces link here after deploying_
+**Author:** Kshamaa
+
+---
+
+## Overview
+
+| | |
+|---|---|
+| **Stage 1** | Conversational cognitive assessment — timed verbal fluency, narrative description, delayed recall |
+| **Stage 2** | NLP-based linguistic biomarker extraction (spaCy / NLTK) |
+| **Stage 3** | Retrieval-augmented generation over a curated literature corpus, with an automated faithfulness check |
+
+The project was built to demonstrate applied NLP and RAG system design in a
+health-adjacent domain, with explicit attention to scientific grounding,
+citation discipline, and output evaluation — engineering practices that
+matter disproportionately when a system touches health-related content.
+
+---
+
+## Disclaimer
+
+This is a research and engineering demonstration prototype, not a validated
+clinical or diagnostic tool. It is inspired by published, peer-reviewed
+cognitive-linguistic assessment paradigms (verbal fluency, narrative
+description, delayed recall), with the following caveats:
+
+- It has not been clinically validated against any diagnostic gold standard.
+- Its reference ranges are rough midpoints drawn from published study
+  ranges, not a clinically derived normative dataset.
+- Language samples are typed, not spoken, and collected in an unsupervised
+  setting — both differ materially from standardized clinical administration.
+- A production deployment in this space would require IRB-reviewed
+  prospective studies, licensed clinical oversight, and regulatory review.
+
+The system is designed to demonstrate AI engineering — NLP feature
+extraction, retrieval-augmented generation, and faithfulness evaluation —
+applied with genuine literacy in the underlying scientific domain.
+
 ---
-title: NeuroLens
-emoji: 🚀
-colorFrom: red
-colorTo: red
-sdk: docker
-app_port: 8501
-tags:
-- streamlit
-pinned: false
-short_description: Streamlit template space
-license: mit
+
+## Architecture
+
+```
+Stage 1: Conversational Assessment (Streamlit)
+   → semantic fluency, phonemic fluency, narrative description, delayed recall
+        ↓
+Stage 2: Linguistic Biomarker Extraction (spaCy / NLTK)
+   → lexical diversity (TTR/MATTR), fluency counts, syntactic complexity,
+     disfluency rate, approximate idea density
+        ↓
+Stage 3: Grounded Prevention Coaching (RAG)
+   → retrieval over a curated 18-paper corpus (e5-large-v2 + FAISS, with a
+     TF-IDF fallback) → generated summary (openai/gpt-oss-120b via Groq)
+     → RAGAS-style faithfulness check against retrieved sources
+```
+
+## Repository Structure
+
+```
+.
+├── app.py              # Streamlit application — all three stages
+├── biomarkers.py        # Stage 2: NLP feature extraction
+├── rag_engine.py         # Stage 3: retrieval, generation, faithfulness check
+├── data/
+│   └── corpus.json        # Curated literature corpus (18 entries)
+├── requirements.txt
+└── README.md
+```
+
 ---
 
-# Welcome to Streamlit!
+## Stage 2 — Linguistic Biomarkers
+
+| Marker | What it measures | Research basis |
+|---|---|---|
+| Semantic (category) fluency | Unique animals named in 60 seconds | One of the most-replicated early markers of cognitive decline; shown to decline faster than letter fluency specifically in those at elevated Alzheimer's risk (longitudinal cohort, n=2,261; PMC7403823). Also shown to meaningfully discriminate normal aging / MCI / AD (PMC9153280). |
+| Phonemic (letter) fluency | Unique F-words named in 60 seconds | Standard companion measure to semantic fluency; tends to remain stable until closer to dementia onset (PMC7403823). |
+| Lexical diversity (TTR / MATTR) | Vocabulary variety, length-corrected | Reduced lexical diversity is associated with reduced vocabulary access in spontaneous speech (Covington & McFall, 2010). |
+| Approximate idea density | Propositions per 10 words (POS-based approximation) | Modeled on the Nun Study: idea density in autobiographies written at ~age 22 predicted Alzheimer's neuropathology roughly 60 years later (Snowdon et al., 1996; retrospective analysis, PMC11852352). This implementation uses a simplified POS-tag-based approximation, not the original CPIDR scoring system. |
+| Syntactic complexity | Subordination index, clause density | Simplified sentence structure has been associated with increased cognitive load during language production. |
+| Delayed recall | Words correctly recalled after a distractor task | Standard episodic memory paradigm, simplified here. |
+
+### Validation extension (planned, not yet executed)
+
+Biomarker extraction could be validated against the **DementiaBank Pitt
+Corpus** — transcribed Cookie Theft picture descriptions from healthy
+controls and dementia patients (Becker et al., 1994). This corpus is
+access-gated (approved-research-use only via TalkBank/DementiaBank), so it
+remains a documented next step rather than something bundled into the repo.
+
+---
+
+## Stage 3 — Literature Corpus & Retrieval
+
+The corpus (`data/corpus.json`) contains 18 entries across six domains, each
+summarized in original prose with source attribution and a link to the
+original publication:
+
+- **APOE4 genetics × lifestyle interaction** — meta-analysis of FINGER, MAPT,
+  and J-MINT (n>3,400) finding lifestyle intervention benefits APOE4 carriers
+  as much as or more than non-carriers (PMC12726239).
+- **Exercise & cognitive reserve** — mechanistic and outcome evidence linking
+  physical activity to neuroplasticity, glymphatic clearance, and preserved
+  executive function.
+- **Diet & nutrition** — the MIND diet RCT (NEJM, 2023) alongside a
+  field-wide caution piece (*Nature*, 2025) on the limits of diet-only effects
+  relative to combined multidomain interventions.
+- **Cognitive training** — the FINGER trial and its global WW-FINGERS
+  adaptations (~70 countries), showing ~25% greater cognitive improvement in
+  the multidomain intervention group versus control.
+- **Social engagement** — social activity as a structural component of
+  FINGER-style trials, including digitally-delivered versions.
+- **Cognitive reserve / early-life enrichment** — the Nun Study's idea
+  density findings and related exercise results.
+
+Retrieval is marker-driven: a Stage 2 score below the reference range
+triggers retrieval from the literature domain most relevant to that specific
+marker (e.g., low semantic fluency → cognitive training and social
+engagement literature), rather than relying on raw text similarity alone.
+
+### Faithfulness evaluation
+
+Every generated coaching summary is checked with a second model call
+(`openai/gpt-oss-120b` via Groq) that decomposes the text into individual
+factual claims and verdicts each as SUPPORTED / PARTIAL / UNSUPPORTED
+against the retrieved source excerpts — a RAGAS-style faithfulness check,
+surfaced in the app's "Faithfulness check" panel. This is treated as the
+most important evaluation step in the pipeline: given the health-adjacent
+subject matter, demonstrating that the system catches its own unsupported
+claims matters more here than in a typical RAG demo.
+
+---
+
+## Setup
+
+### Local installation
+
+```bash
+pip install -r requirements.txt
+python -m spacy download en_core_web_sm   # if not already pulled via requirements.txt
+export GROQ_API_KEY=your_key_here
+streamlit run app.py
+```
+
+A Groq API key is required for Stage 3 generation: create one at
+[console.groq.com](https://console.groq.com) under **API Keys**.
+
+### Deployment (HuggingFace Spaces)
+
+1. Create a new Space (SDK: Streamlit).
+2. Push `app.py`, `biomarkers.py`, `rag_engine.py`, `data/corpus.json`, and `requirements.txt`.
+3. Add `GROQ_API_KEY` as a Space secret under **Settings → Variables and secrets**.
+4. On first run, the app downloads the spaCy model and the `e5-large-v2`
+   sentence-embedding model. If `sentence-transformers` or its weights are
+   unavailable for any reason, the retriever automatically falls back to a
+   TF-IDF retriever (`rag_engine.py: LiteratureRetriever`), so the pipeline
+   degrades gracefully rather than failing.
+
+---
+
+## Known Limitations
+
+- Reference ranges are literature-informed approximations, not a validated
+  clinical normative sample — flagged explicitly in the app.
+- Typed responses differ from spoken responses; several markers (disfluency,
+  timing) are conventionally derived from speech, not text.
+- Fluency tasks use an enforced 60-second countdown (30 seconds for the
+  recall distractor), implemented via periodic Streamlit reruns rather than
+  a client-side timer — accurate to roughly one second, sufficient for this
+  use case.
+- Idea density is a simplified POS-based approximation of the
+  Snowdon/Kemper method, not the original CPIDR scoring tool.
+- The animal/F-word validity lists used for fluency scoring are compact,
+  curated lists rather than an exhaustive lexical resource; uncommon but
+  valid answers may be flagged as "possible intrusions" rather than counted.
+- No clinical validation has been performed; DementiaBank validation is a
+  documented next step pending approved data access.
 
-Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :heart:
+## Future Work
 
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
+- Speech-to-text capture for analysis of true spoken language
+- Client-side (JS-based) frame-accurate timing
+- DementiaBank Pitt Corpus validation, reporting a basic separation result
+  (e.g. effect size) between healthy-control and dementia transcripts
+- Expansion of the literature corpus and per-domain retrieval re-ranking
+- Opt-in longitudinal tracking of results across multiple sessions

app.py

@@ -0,0 +1,355 @@
+"""
+app.py — NeuroLens: End-to-End Cognitive Health Screening & Coaching Pipeline
+
+A three-stage Streamlit app:
+  1. Conversational cognitive assessment (typed language samples, enforced timers)
+  2. Linguistic biomarker extraction + dashboard (Stage 2)
+  3. Grounded, citation-backed prevention coaching via RAG (Stage 3)
+
+IMPORTANT: This is a research/portfolio demonstration prototype inspired by
+published cognitive assessment paradigms. It is NOT a validated clinical or
+diagnostic instrument. See the "About this project" section in the sidebar
+and README.md for full limitations and citations.
+"""
+
+import os
+import time
+import streamlit as st
+from streamlit_autorefresh import st_autorefresh
+
+from biomarkers import build_profile
+from rag_engine import LiteratureRetriever, build_retrieved_context, generate_coaching, check_faithfulness
+
+st.set_page_config(page_title="NeuroLens", page_icon="🧠", layout="wide")
+
+RECALL_WORDS = ["apple", "river", "hammer", "candle", "bicycle", "lantern"]
+FLUENCY_TIME_LIMIT = 60   # seconds, enforced — matches the standard 60s fluency-task paradigm
+DISTRACTOR_TIME_LIMIT = 30  # seconds, enforced counting-backward distractor
+
+TASK_FLOW = ["semantic", "phonemic", "narrative", "distractor", "recall"]
+
+# ---------------------------------------------------------------------------
+# Timer helper — enforces a hard time limit on a task using periodic reruns.
+# The text typed so far is preserved (tied to session_state by widget key);
+# once time is up the input is disabled and the user must continue.
+# ---------------------------------------------------------------------------
+def enforced_timer(task_key: str, limit_seconds: int):
+    start_key = f"{task_key}_start_time"
+    if start_key not in st.session_state:
+        st.session_state[start_key] = time.time()
+
+    elapsed = time.time() - st.session_state[start_key]
+    remaining = max(0, limit_seconds - elapsed)
+    timed_out = remaining <= 0
+
+    if not timed_out:
+        # Re-run this script once per second until the limit is hit, so the
+        # countdown visibly ticks down and the widget gets disabled the
+        # moment time expires — without needing any JS component.
+        st_autorefresh(interval=1000, limit=limit_seconds + 2, key=f"refresh_{task_key}")
+
+    mins, secs = divmod(int(remaining), 60)
+    bar_col, time_col = st.columns([4, 1])
+    with bar_col:
+        st.progress(min(1.0, elapsed / limit_seconds) if limit_seconds else 1.0)
+    with time_col:
+        st.markdown(f"**⏱️ 0:{secs:02d}**" if mins == 0 else f"**⏱️ {mins}:{secs:02d}**")
+
+    if timed_out:
+        st.warning("⏰ Time's up — your response has been locked in.", icon="⏰")
+
+    return timed_out
+
+
+def reset_task_timer(task_key: str):
+    st.session_state.pop(f"{task_key}_start_time", None)
+
+
+# ---------------------------------------------------------------------------
+# Sidebar — framing & disclosure (lead with the limitation, not the capability)
+# ---------------------------------------------------------------------------
+with st.sidebar:
+    st.markdown("## 🧠 About NeuroLens")
+    st.warning(
+        "**This is a research / portfolio demonstration prototype**, not a "
+        "validated diagnostic or clinical tool. It is inspired by published "
+        "cognitive-linguistic assessment paradigms but has not been clinically "
+        "validated. Any real-world deployment would require IRB-reviewed "
+        "studies and licensed clinical oversight.",
+        icon="⚠️",
+    )
+    st.markdown(
+        "**What it actually demonstrates:**\n"
+        "- NLP feature extraction grounded in real cognitive-aging research\n"
+        "- Enforced, timed task administration (matching the standard 60s "
+        "fluency-task paradigm used in the literature)\n"
+        "- A retrieval-augmented generation (RAG) pipeline with a faithfulness check\n"
+        "- End-to-end product thinking applied to a health-adjacent domain"
+    )
+    st.markdown("---")
+    st.markdown(
+        "Built as a proof-of-concept exploring the kind of AI engineering + "
+        "domain literacy relevant to evidence-informed brain-health products. "
+        "See `README.md` for full citations and the linguistic-biomarker literature."
+    )
+    api_key_present = bool(os.environ.get("GROQ_API_KEY"))
+    if not api_key_present:
+        st.info("Set the `GROQ_API_KEY` environment variable to enable Stage 3 coaching generation.", icon="🔑")
+
+# ---------------------------------------------------------------------------
+# Session state
+# ---------------------------------------------------------------------------
+if "stage" not in st.session_state:
+    st.session_state.stage = 1
+if "task_idx" not in st.session_state:
+    st.session_state.task_idx = 0
+if "responses" not in st.session_state:
+    st.session_state.responses = {}
+if "profile" not in st.session_state:
+    st.session_state.profile = None
+if "coaching" not in st.session_state:
+    st.session_state.coaching = None
+
+st.title("🧠 NeuroLens")
+st.caption("Research-prototype cognitive-linguistic screening → biomarker dashboard → grounded prevention coaching")
+
+stage_labels = ["1. Assessment", "2. Biomarker Dashboard", "3. Prevention Coaching"]
+st.progress((st.session_state.stage - 1) / 2, text=stage_labels[st.session_state.stage - 1])
+
+
+def goto_next_task():
+    st.session_state.task_idx += 1
+    if st.session_state.task_idx >= len(TASK_FLOW):
+        st.session_state.profile = build_profile(st.session_state.responses)
+        st.session_state.stage = 2
+    st.rerun()
+
+
+# ---------------------------------------------------------------------------
+# STAGE 1 — Conversational Cognitive Assessment (each task individually timed)
+# ---------------------------------------------------------------------------
+if st.session_state.stage == 1:
+    st.header("Stage 1 · Short Language Assessment")
+    st.caption(
+        "Simplified, non-clinical versions of standard cognitive-linguistics research tasks. "
+        "The fluency tasks are timed (60s) to match the standard paradigm; just type freely — "
+        "there are no right or wrong answers."
+    )
+
+    current_task = TASK_FLOW[st.session_state.task_idx]
+    st.subheader(f"Task {st.session_state.task_idx + 1} of {len(TASK_FLOW)}")
+
+    # --- Semantic (category) fluency: 60s enforced ---
+    if current_task == "semantic":
+        st.markdown("### 🐾 Category Fluency")
+        st.write("List as many **animals** as you can before the timer runs out. Separate with commas or spaces.")
+        timed_out = enforced_timer("semantic", FLUENCY_TIME_LIMIT)
+        text = st.text_area("Animals", key="semantic_input", height=110,
+                             placeholder="e.g. dog, cat, elephant, ...", disabled=timed_out,
+                             label_visibility="collapsed")
+        c1, c2 = st.columns([1, 1])
+        with c1:
+            early = st.button("I'm done →", disabled=timed_out)
+        if timed_out or early:
+            st.session_state.responses["semantic_fluency_text"] = st.session_state.get("semantic_input", "")
+            reset_task_timer("semantic")
+            goto_next_task()
+
+    # --- Phonemic (letter) fluency: 60s enforced ---
+    elif current_task == "phonemic":
+        st.markdown("### 🔤 Letter Fluency")
+        st.write("List as many words as you can that start with the letter **F** (no names of people/places).")
+        timed_out = enforced_timer("phonemic", FLUENCY_TIME_LIMIT)
+        text = st.text_area("F words", key="phonemic_input", height=110,
+                             placeholder="e.g. fish, fast, forest, ...", disabled=timed_out,
+                             label_visibility="collapsed")
+        early = st.button("I'm done →", disabled=timed_out)
+        if timed_out or early:
+            st.session_state.responses["phonemic_fluency_text"] = st.session_state.get("phonemic_input", "")
+            st.session_state.responses["phonemic_letter"] = "F"
+            reset_task_timer("phonemic")
+            goto_next_task()
+
+    # --- Narrative description: open-ended, no enforced timer (per paradigm) ---
+    elif current_task == "narrative":
+        st.markdown("### 📝 Narrative Description")
+        st.write("Describe your **morning routine** in as much detail as you can. Take your time — this task isn't timed.")
+        text = st.text_area("Your morning routine", key="narrative_input", height=160,
+                             placeholder="I usually wake up around...", label_visibility="collapsed")
+        if st.button("Continue →"):
+            st.session_state.responses["narrative_text"] = st.session_state.get("narrative_input", "")
+            goto_next_task()
+
+    # --- Distractor task: 30s enforced counting-backward, before recall ---
+    elif current_task == "distractor":
+        st.markdown("### 🔢 Quick Mental Task")
+        st.write("Try to keep the six words below in mind, then count backward from 100 by 7s "
+                 "(100, 93, 86 ...) until the timer ends.")
+        st.markdown("#### " + "  •  ".join(w.upper() for w in RECALL_WORDS))
+        timed_out = enforced_timer("distractor", DISTRACTOR_TIME_LIMIT)
+        if timed_out:
+            if st.button("Continue to recall →"):
+                reset_task_timer("distractor")
+                goto_next_task()
+        else:
+            st.caption("The recall task will unlock automatically when the timer ends.")
+
+    # --- Delayed recall: open-ended, no enforced timer ---
+    elif current_task == "recall":
+        st.markdown("### 🧩 Delayed Recall")
+        st.write("Now list as many of the six words from a moment ago as you can remember:")
+        text = st.text_area("Recalled words", key="recall_input", height=90, label_visibility="collapsed")
+        if st.button("Submit assessment →", type="primary"):
+            st.session_state.responses["recall_response_text"] = st.session_state.get("recall_input", "")
+            st.session_state.responses["recall_target_words"] = RECALL_WORDS
+            goto_next_task()
+
+# ---------------------------------------------------------------------------
+# STAGE 2 — Linguistic Biomarker Dashboard
+# ---------------------------------------------------------------------------
+elif st.session_state.stage == 2:
+    st.header("Stage 2 · Linguistic Biomarker Dashboard")
+    st.caption(
+        "Computed from your timed/typed responses using standard NLP feature-extraction techniques "
+        "(spaCy/NLTK), compared against rough, literature-informed reference ranges — "
+        "**not a clinically derived normative dataset.** See README for the limitation."
+    )
+
+    profile = st.session_state.profile
+    bands = profile["bands"]
+
+    band_icon = {"below_typical_range": "🔻", "within_typical_range": "✅", "above_typical_range": "🔺"}
+    band_label = {"below_typical_range": "Below typical range", "within_typical_range": "Within typical range",
+                  "above_typical_range": "Above typical range"}
+
+    cols = st.columns(3)
+    metrics = [
+        ("Semantic fluency", profile["semantic_fluency"]["unique_valid_count"], "unique animals named",
+         bands["semantic_fluency"]),
+        ("Phonemic fluency", profile["phonemic_fluency"]["unique_valid_count"], "unique F-words named",
+         bands["phonemic_fluency"]),
+        ("Lexical diversity (MATTR)", profile["lexical_diversity"]["mattr"], "vocabulary variety score",
+         bands["lexical_diversity"]),
+        ("Idea density (approx.)", profile["idea_density"]["approx_idea_density_per_10_words"], "propositions per 10 words",
+         bands["idea_density"]),
+        ("Syntactic complexity", profile["syntactic_complexity"]["subordination_index"], "subordinate clauses / sentence",
+         bands["syntactic_complexity"]),
+        ("Delayed recall", f"{profile['delayed_recall']['correct_count']}/{profile['delayed_recall']['total_targets']}",
+         "words correctly recalled", None),
+    ]
+    for i, (label, value, sublabel, band) in enumerate(metrics):
+        with cols[i % 3]:
+            badge = f"{band_icon.get(band,'')} {band_label.get(band,'')}" if band else ""
+            st.metric(label, value, help=sublabel)
+            if badge:
+                st.caption(badge)
+
+    with st.expander("📋 See full extracted profile (raw output)"):
+        st.json(profile)
+
+    with st.expander("ℹ️ What do these markers mean? (with citations)"):
+        st.markdown("""
+- **Semantic fluency** (animal naming): one of the most-replicated early markers in the
+  cognitive-aging literature; semantic fluency has been shown to decline faster than letter
+  fluency specifically in those at elevated Alzheimer's risk (PMC7403823).
+- **Lexical diversity (MATTR)**: a length-corrected measure of vocabulary variety; reduced
+  lexical diversity is associated with reduced vocabulary access in spontaneous speech.
+- **Idea density**: a simplified, POS-based approximation of the propositional idea-density
+  measure made famous by the Nun Study, where idea density in essays written in early
+  adulthood predicted Alzheimer's pathology roughly 60 years later (Snowdon et al., 1996;
+  PMC11852352).
+- **Syntactic complexity**: simplified sentence structure can reflect increased cognitive load
+  during language production.
+- **Delayed recall**: a classic episodic memory measure, included here in simplified form.
+
+Full citation list is in `README.md`.
+        """)
+
+    st.warning(
+        "Reference ranges shown above are **rough, literature-informed approximations**, "
+        "not a validated clinical normative dataset. A single below-range score on a short, "
+        "typed task says very little on its own — performance varies widely with mood, "
+        "fatigue, typing speed, and unfamiliarity with the task.",
+        icon="⚠️",
+    )
+
+    col_a, col_b = st.columns(2)
+    with col_a:
+        if st.button("← Retake assessment"):
+            st.session_state.stage = 1
+            st.session_state.task_idx = 0
+            st.session_state.responses = {}
+            st.session_state.profile = None
+            for t in TASK_FLOW:
+                reset_task_timer(t)
+            st.rerun()
+    with col_b:
+        if st.button("Continue to personalized coaching →", type="primary"):
+            st.session_state.stage = 3
+            st.rerun()
+
+# ---------------------------------------------------------------------------
+# STAGE 3 — Grounded Prevention Coaching (RAG)
+# ---------------------------------------------------------------------------
+elif st.session_state.stage == 3:
+    st.header("Stage 3 · Personalized Prevention Coaching")
+    st.caption(
+        "Retrieves relevant brain-health prevention literature based on your Stage 2 profile, "
+        "then generates a grounded summary — with a faithfulness check against the retrieved sources."
+    )
+
+    api_key = os.environ.get("GROQ_API_KEY")
+    if not api_key:
+        st.error(
+            "No `GROQ_API_KEY` found in the environment. Set it (e.g. as a HuggingFace "
+            "Spaces secret) to generate coaching text — see README.md for setup.",
+            icon="🔑",
+        )
+    else:
+        if st.session_state.coaching is None:
+            with st.spinner("Retrieving relevant literature and generating your summary..."):
+                retriever = LiteratureRetriever()
+                retrieved = build_retrieved_context(retriever, st.session_state.profile)
+                coaching = generate_coaching(st.session_state.profile, retrieved)
+                faithfulness = check_faithfulness(coaching["text"], coaching["sources_used"])
+                st.session_state.coaching = {"coaching": coaching, "faithfulness": faithfulness,
+                                              "retriever_backend": retriever.backend}
+
+        result = st.session_state.coaching
+        st.markdown("### Your personalized summary")
+        st.markdown(result["coaching"]["text"])
+
+        if result["coaching"]["sources_used"]:
+            with st.expander(f"📚 Sources used ({len(result['coaching']['sources_used'])})"):
+                for s in result["coaching"]["sources_used"]:
+                    st.markdown(f"**{s['title']}**  \n*{s['source']}*  \n{s['summary']}  \n[Link]({s['url']})")
+                    st.markdown("---")
+
+        with st.expander("✅ Faithfulness check (RAGAS-style evaluation)"):
+            f = result["faithfulness"]
+            score = f.get("faithfulness_score")
+            if score is not None:
+                st.metric("Faithfulness score", f"{score:.0%}",
+                          help="Fraction of generated claims directly supported by retrieved sources")
+            for c in f.get("claims", []):
+                icon = {"SUPPORTED": "✅", "PARTIAL": "🟡", "UNSUPPORTED": "❌"}.get(c["verdict"], "❓")
+                st.write(f"{icon} **{c['verdict']}** — {c['claim']}")
+                st.caption(c.get("reason", ""))
+            st.caption(f"Retrieval backend used: `{result['retriever_backend']}`")
+
+    st.markdown("---")
+    col_a, col_b = st.columns(2)
+    with col_a:
+        if st.button("← Back to dashboard"):
+            st.session_state.stage = 2
+            st.rerun()
+    with col_b:
+        if st.button("Start over"):
+            st.session_state.stage = 1
+            st.session_state.task_idx = 0
+            st.session_state.responses = {}
+            st.session_state.profile = None
+            st.session_state.coaching = None
+            for t in TASK_FLOW:
+                reset_task_timer(t)
+            st.rerun()

biomarkers.py

@@ -0,0 +1,350 @@
+"""
+biomarkers.py — Stage 2 of NeuroLens: Linguistic Biomarker Extraction
+
+Computes computational approximations of linguistic markers documented in the
+cognitive-aging / dementia literature from short, typed language samples.
+
+IMPORTANT: These are simplified, non-clinical approximations of published
+research measures, computed from typed text rather than clinically
+standardized administration and scoring. They are NOT validated diagnostic
+or clinical assessment tools. See README.md for citations and limitations.
+
+Design notes:
+- Uses spaCy if a model is installed (richer dependency-based syntax metrics).
+  Falls back to NLTK POS tagging if spaCy/its model isn't available, so the
+  pipeline still runs in restricted environments. On deployment (HF Spaces),
+  install en_core_web_sm for the full feature set.
+"""
+
+import re
+import math
+from collections import Counter
+
+import nltk
+from nltk.tokenize import word_tokenize, sent_tokenize
+
+# Ensure required NLTK resources are present (no-op if already downloaded)
+for pkg in ["punkt_tab", "punkt", "averaged_perceptron_tagger_eng",
+            "averaged_perceptron_tagger", "stopwords"]:
+    try:
+        nltk.data.find(pkg)
+    except LookupError:
+        try:
+            nltk.download(pkg, quiet=True)
+        except Exception:
+            pass
+
+# Try to load spaCy for richer syntactic parsing; degrade gracefully if unavailable.
+_SPACY_NLP = None
+try:
+    import spacy
+    try:
+        _SPACY_NLP = spacy.load("en_core_web_sm")
+    except OSError:
+        _SPACY_NLP = None
+except ImportError:
+    _SPACY_NLP = None
+
+FILLER_WORDS = {"um", "uh", "er", "erm", "hmm", "uhh", "umm"}
+
+SUBORDINATING_MARKERS = {
+    "because", "although", "though", "since", "unless", "while", "whereas",
+    "if", "when", "whenever", "after", "before", "until", "as", "that",
+    "which", "who", "whom", "whose", "where", "so that", "even though",
+}
+
+# Curated, non-exhaustive validity lists for fluency-task scoring.
+# In production these should be swapped for a larger lexical resource;
+# kept compact here for transparency and easy auditing.
+COMMON_ANIMALS = set("""
+dog cat horse cow pig sheep goat chicken duck goose turkey rabbit mouse rat
+squirrel deer elk moose bear wolf fox lion tiger leopard cheetah elephant
+giraffe zebra hippopotamus hippo rhinoceros rhino monkey gorilla chimpanzee
+chimp orangutan kangaroo koala panda camel llama alpaca otter beaver raccoon
+skunk badger hedgehog bat whale dolphin shark seal walrus penguin eagle hawk
+owl falcon vulture sparrow robin crow raven pigeon parrot peacock ostrich
+flamingo swan pelican woodpecker hummingbird snake lizard turtle tortoise
+crocodile alligator frog toad salamander newt fish trout salmon tuna catfish
+goldfish shrimp crab lobster octopus squid jellyfish starfish bee wasp ant
+butterfly moth spider scorpion beetle ladybug grasshopper cricket mosquito
+fly cockroach worm snail slug donkey mule buffalo bison antelope gazelle
+hyena jaguar puma lynx wolverine porcupine armadillo sloth anteater platypus
+mole shrew weasel ferret mink stoat chinchilla guinea pig hamster gerbil
+parakeet canary cockatoo toucan stork heron crane ibis kiwi emu cassowary
+iguana gecko chameleon python cobra viper boa anaconda komodo dragon manatee
+narwhal orca dugong stingray eel pufferfish swordfish marlin barracuda
+""".split())
+
+
+def _tokenize_words(text: str):
+    return [w for w in word_tokenize(text.lower()) if re.search(r"[a-z]", w)]
+
+
+def _pos_tags(text: str):
+    tokens = word_tokenize(text)
+    return nltk.pos_tag(tokens)
+
+
+# ---------------------------------------------------------------------------
+# 1. Lexical diversity — TTR and MATTR (Covington & McFall, 2010)
+# ---------------------------------------------------------------------------
+def lexical_diversity(text: str, window: int = 30) -> dict:
+    words = _tokenize_words(text)
+    n = len(words)
+    if n == 0:
+        return {"ttr": 0.0, "mattr": 0.0, "token_count": 0}
+
+    ttr = len(set(words)) / n
+
+    if n <= window:
+        mattr = ttr
+    else:
+        ratios = []
+        for i in range(n - window + 1):
+            seg = words[i:i + window]
+            ratios.append(len(set(seg)) / window)
+        mattr = sum(ratios) / len(ratios)
+
+    return {"ttr": round(ttr, 3), "mattr": round(mattr, 3), "token_count": n}
+
+
+# ---------------------------------------------------------------------------
+# 2. Semantic / phonemic fluency scoring
+# ---------------------------------------------------------------------------
+def score_semantic_fluency(raw_response: str) -> dict:
+    """Animal-naming (category) fluency: unique valid exemplars generated."""
+    candidates = [w.strip(".,;") for w in re.split(r"[,\s]+", raw_response.lower()) if w.strip(".,;")]
+    seen, valid_unique, intrusions = set(), [], []
+    for w in candidates:
+        if w in seen:
+            continue
+        seen.add(w)
+        if w in COMMON_ANIMALS:
+            valid_unique.append(w)
+        else:
+            intrusions.append(w)
+    return {
+        "unique_valid_count": len(valid_unique),
+        "items": valid_unique,
+        "possible_intrusions": intrusions,  # words not in animal list — not necessarily errors, may just be missing from our list
+    }
+
+
+def score_phonemic_fluency(raw_response: str, target_letter: str) -> dict:
+    """Letter (phonemic) fluency, e.g. words starting with 'F'."""
+    target_letter = target_letter.lower()
+    candidates = [w.strip(".,;") for w in re.split(r"[,\s]+", raw_response.lower()) if w.strip(".,;")]
+    seen, valid_unique, invalid = set(), [], []
+    for w in candidates:
+        if w in seen:
+            continue
+        seen.add(w)
+        if w.startswith(target_letter) and len(w) > 1:
+            valid_unique.append(w)
+        else:
+            invalid.append(w)
+    return {
+        "unique_valid_count": len(valid_unique),
+        "items": valid_unique,
+        "excluded": invalid,
+    }
+
+
+# ---------------------------------------------------------------------------
+# 3. Syntactic complexity — MLU + subordination index
+# ---------------------------------------------------------------------------
+def syntactic_complexity(text: str) -> dict:
+    sentences = [s for s in sent_tokenize(text) if s.strip()]
+    if not sentences:
+        return {"mean_length_utterance": 0.0, "subordination_index": 0.0, "clause_density": 0.0}
+
+    words_per_sentence = [len(_tokenize_words(s)) for s in sentences]
+    mlu = sum(words_per_sentence) / len(sentences)
+
+    total_words = sum(words_per_sentence) or 1
+    sub_count = 0
+    clause_markers = 0
+
+    if _SPACY_NLP is not None:
+        doc = _SPACY_NLP(text)
+        # Dependency-based clause count: finite verbs governed by mark/advcl/relcl/ccomp
+        clause_markers = sum(1 for tok in doc if tok.dep_ in
+                              {"advcl", "relcl", "ccomp", "acl", "csubj"})
+        sub_count = sum(1 for tok in doc if tok.dep_ == "mark")
+    else:
+        text_lower = text.lower()
+        for marker in SUBORDINATING_MARKERS:
+            sub_count += len(re.findall(rf"\b{re.escape(marker)}\b", text_lower))
+        clause_markers = sub_count  # fallback approximation
+
+    subordination_index = sub_count / len(sentences)
+    clause_density = clause_markers / total_words * 100  # clauses per 100 words
+
+    return {
+        "mean_length_utterance": round(mlu, 2),
+        "subordination_index": round(subordination_index, 2),
+        "clause_density": round(clause_density, 2),
+        "engine": "spacy" if _SPACY_NLP is not None else "nltk_fallback",
+    }
+
+
+# ---------------------------------------------------------------------------
+# 4. Disfluency markers
+# ---------------------------------------------------------------------------
+def disfluency_markers(text: str) -> dict:
+    words = _tokenize_words(text)
+    n = len(words) or 1
+
+    filler_count = sum(1 for w in words if w in FILLER_WORDS)
+
+    # immediate word repetitions ("the the", "I I")
+    repetitions = sum(1 for i in range(1, len(words)) if words[i] == words[i - 1])
+
+    # false starts: dash/ellipsis-truncated tokens, common in typed self-correction
+    false_starts = len(re.findall(r"\b\w+-\s", text)) + text.count("...")
+
+    return {
+        "filler_rate_per_100_words": round(filler_count / n * 100, 2),
+        "repetition_rate_per_100_words": round(repetitions / n * 100, 2),
+        "false_start_count": false_starts,
+    }
+
+
+# ---------------------------------------------------------------------------
+# 5. Idea density (approximation of Snowdon & Kemper's propositional idea
+#    density / CPIDR method). True CPIDR uses fine-grained proposition rules;
+#    this is a POS-based approximation: count of content + relational words
+#    that typically introduce a proposition (verbs, adjectives, adverbs,
+#    prepositions, conjunctions) per 10 words.
+# ---------------------------------------------------------------------------
+def idea_density(text: str) -> dict:
+    tagged = _pos_tags(text)
+    words = [w for w, t in tagged if re.search(r"[a-zA-Z]", w)]
+    n = len(words) or 1
+
+    proposition_tags = {
+        "VB", "VBD", "VBG", "VBN", "VBP", "VBZ",   # verbs
+        "JJ", "JJR", "JJS",                          # adjectives
+        "RB", "RBR", "RBS",                          # adverbs
+        "IN",                                          # prepositions/subordinators
+        "CC",                                           # coordinating conjunctions
+    }
+    proposition_count = sum(1 for w, t in tagged if t in proposition_tags)
+
+    density_per_10 = proposition_count / n * 10
+
+    return {
+        "approx_idea_density_per_10_words": round(density_per_10, 2),
+        "proposition_count": proposition_count,
+        "word_count": n,
+        "note": "POS-based approximation of Snowdon & Kemper idea density, not full CPIDR scoring",
+    }
+
+
+# ---------------------------------------------------------------------------
+# 6. Delayed recall scoring
+# ---------------------------------------------------------------------------
+def score_delayed_recall(recalled_text: str, target_words: list) -> dict:
+    recalled = set(_tokenize_words(recalled_text))
+    targets = set(w.lower() for w in target_words)
+    correct = recalled & targets
+    intrusions = recalled - targets
+    return {
+        "correct_count": len(correct),
+        "total_targets": len(targets),
+        "correct_words": sorted(correct),
+        "intrusions": sorted(intrusions),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Rough published / literature-informed comparison baselines.
+# These are NOT a clinically derived normative dataset — they are coarse
+# midpoints drawn from ranges reported across the cited fluency-task
+# literature, intended only to give the user a directional sense of where a
+# score falls. This limitation is stated explicitly in the UI and README.
+# ---------------------------------------------------------------------------
+ROUGH_NORMS = {
+    "semantic_fluency_count": {"low": 13, "mid": 19, "high": 25},
+    "phonemic_fluency_count": {"low": 9, "mid": 14, "high": 19},
+    "mattr": {"low": 0.55, "high": 0.75},
+    "approx_idea_density_per_10_words": {"low": 3.0, "high": 5.5},
+    "subordination_index": {"low": 0.3, "high": 1.2},
+}
+
+
+def percentile_band(value: float, low: float, high: float) -> str:
+    if value < low:
+        return "below_typical_range"
+    if value > high:
+        return "above_typical_range"
+    return "within_typical_range"
+
+
+def build_profile(assessment_responses: dict) -> dict:
+    """
+    assessment_responses keys expected:
+      'semantic_fluency_text', 'phonemic_fluency_text', 'phonemic_letter',
+      'narrative_text', 'recall_response_text', 'recall_target_words'
+    Returns a structured biomarker profile across all four tasks.
+    """
+    sem = score_semantic_fluency(assessment_responses.get("semantic_fluency_text", ""))
+    phon = score_phonemic_fluency(
+        assessment_responses.get("phonemic_fluency_text", ""),
+        assessment_responses.get("phonemic_letter", "F"),
+    )
+    narrative = assessment_responses.get("narrative_text", "")
+    lex = lexical_diversity(narrative)
+    syn = syntactic_complexity(narrative)
+    disf = disfluency_markers(narrative)
+    idea = idea_density(narrative)
+    recall = score_delayed_recall(
+        assessment_responses.get("recall_response_text", ""),
+        assessment_responses.get("recall_target_words", []),
+    )
+
+    norms = ROUGH_NORMS
+    bands = {
+        "semantic_fluency": percentile_band(
+            sem["unique_valid_count"], norms["semantic_fluency_count"]["low"], norms["semantic_fluency_count"]["high"]),
+        "phonemic_fluency": percentile_band(
+            phon["unique_valid_count"], norms["phonemic_fluency_count"]["low"], norms["phonemic_fluency_count"]["high"]),
+        "lexical_diversity": percentile_band(
+            lex["mattr"], norms["mattr"]["low"], norms["mattr"]["high"]),
+        "idea_density": percentile_band(
+            idea["approx_idea_density_per_10_words"],
+            norms["approx_idea_density_per_10_words"]["low"],
+            norms["approx_idea_density_per_10_words"]["high"]),
+        "syntactic_complexity": percentile_band(
+            syn["subordination_index"], norms["subordination_index"]["low"], norms["subordination_index"]["high"]),
+    }
+
+    return {
+        "semantic_fluency": sem,
+        "phonemic_fluency": phon,
+        "lexical_diversity": lex,
+        "syntactic_complexity": syn,
+        "disfluency": disf,
+        "idea_density": idea,
+        "delayed_recall": recall,
+        "bands": bands,
+    }
+
+
+if __name__ == "__main__":
+    # Smoke test
+    sample = {
+        "semantic_fluency_text": "dog cat horse cow pig sheep goat lion tiger bear bear fox wolf",
+        "phonemic_fluency_text": "fox fish fan friend funny fast forest",
+        "phonemic_letter": "F",
+        "narrative_text": (
+            "I wake up around seven and I, um, make coffee while I check my email. "
+            "Then I usually take a shower and get dressed before I head out, although "
+            "sometimes I eat breakfast first if I have time."
+        ),
+        "recall_response_text": "apple table river hammer",
+        "recall_target_words": ["apple", "table", "river", "hammer", "candle", "bicycle"],
+    }
+    import json
+    profile = build_profile(sample)
+    print(json.dumps(profile, indent=2))

data/corpus.json

@@ -0,0 +1,146 @@
+[
+  {
+    "id": "fingers_multidomain_2015",
+    "domain": "cognitive_training",
+    "title": "FINGER: Finnish Geriatric Intervention Study to Prevent Cognitive Impairment and Disability",
+    "source": "Kivipelto et al., The Lancet, 2015; long-term follow-up analyses 2025-2026",
+    "url": "https://www.sciencedirect.com/science/article/pii/S2666245025000091",
+    "summary": "FINGER was the first large randomized controlled trial (n=1,260) to show that a combined two-year program of healthy diet, structured exercise, cognitive training, social activity, and vascular risk management produced meaningfully better cognitive outcomes than general health advice in older adults at risk for decline. The intervention group improved roughly 25% more on a composite cognition score, with gains across executive function, processing speed, and complex memory. The control group's risk of cognitive decline was about 30% higher after two years. Eleven-year follow-up work has found that both intervention participation and the degree of actual lifestyle change achieved were linked to better long-term cognitive trajectories, suggesting durability beyond the active trial period."
+  },
+  {
+    "id": "ww_fingers_global_network",
+    "domain": "cognitive_training",
+    "title": "World-Wide FINGERS network and digital cognitive training tools",
+    "source": "Multiple WW-FINGERS protocol and outcome papers, 2025-2026",
+    "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12762639/",
+    "summary": "The FINGER model has since been adapted by research teams in roughly 70 countries through the World-Wide FINGERS network, with computer-based and app-delivered cognitive training as a consistent component across adaptations such as MIND-ADmini, AgeWell.de, J-MINT, and MIND-CHINA. Across these trials, digital cognitive training is most often combined with physical activity and social engagement components rather than used in isolation, and several programs have demonstrated that internet-delivered, multidomain interventions can produce measurable cognitive and health benefits even without in-person delivery."
+  },
+  {
+    "id": "us_pointer_2025",
+    "domain": "cognitive_training",
+    "title": "U.S. POINTER: testing FINGER-style multidomain intervention in a U.S. population",
+    "source": "U.S. POINTER design and outcome papers, 2025",
+    "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12726325/",
+    "summary": "U.S. POINTER was designed to test whether FINGER's positive cognitive findings generalize to a larger, more diverse American population at risk for cognitive decline. It combined the same core domains -- physical activity, nutritional guidance, cognitive and social engagement, and cardiometabolic risk management -- delivered through a structured, coached, multi-year program, reinforcing that combined lifestyle interventions (rather than any single habit change) tend to outperform single-domain approaches."
+  },
+  {
+    "id": "apoe4_lifestyle_metaanalysis",
+    "domain": "apoe4_genetics",
+    "title": "Effect of ApoE genotype on efficacy of multidomain lifestyle interventions: meta-analysis of FINGER, MAPT, and J-MINT",
+    "source": "Alzheimer's & Dementia, PMC12726239, 2025",
+    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC12726239/",
+    "summary": "Pooling three randomized controlled trials (FINGER, MAPT, and J-MINT, combined n > 3,400), this meta-analysis found that multidomain lifestyle interventions benefited APOE4 carriers at least as much as -- and in some trials more than -- non-carriers, with a statistically significant genotype-by-intervention interaction. Effect sizes were largest in J-MINT (0.36), followed by FINGER (0.15) and MAPT (0.12). This is described as the first robust evidence that carrying the strongest common genetic risk factor for Alzheimer's does not blunt the benefit of lifestyle intervention, and may even amplify it."
+  },
+  {
+    "id": "pensa_apoe4_egcg",
+    "domain": "apoe4_genetics",
+    "title": "PENSA: personalized multimodal lifestyle intervention in APOE4 carriers with subjective cognitive decline",
+    "source": "World-Wide FINGERS PENSA trial, PMC11714784, 2025",
+    "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11714784/",
+    "summary": "PENSA enrolled APOE4 carriers experiencing subjective cognitive decline in a 12-month personalized program of diet, physical activity, and cognitive and social stimulation. Adherence was notably high across components (around 90% for nutrition guidance, over 70% for cognitive training and psychoeducation), and the trial used continuous ecological momentary assessment via mobile tools, illustrating that intensive, personalized multidomain coaching is feasible to sustain in an at-risk genetic population over a full year."
+  },
+  {
+    "id": "apoe4_lifestyle_adi_summary",
+    "domain": "apoe4_genetics",
+    "title": "Healthy lifestyle changes and cognition in APOE4 carriers (FINGER subgroup analysis)",
+    "source": "Solomon et al., JAMA Neurology, 2018; ADDF summary",
+    "url": "https://www.alzdiscovery.org/cognitive-vitality/blog/healthy-lifestyle-changes-may-benefit-cognition-in-older-people-with-apoe4",
+    "summary": "A subgroup analysis of the original FINGER trial focusing on the roughly 360 participants carrying at least one APOE4 allele found that the multidomain lifestyle program improved cognitive function and memory in this genetically at-risk group, similar to or better than the benefit seen in non-carriers. This was an early and influential signal that genetic risk for Alzheimer's is not fixed destiny -- modifiable lifestyle factors still meaningfully shift outcomes."
+  },
+  {
+    "id": "exercise_glymphatic_review",
+    "domain": "exercise_cognitive_reserve",
+    "title": "Physical activity, neuroplasticity, and glymphatic clearance in aging",
+    "source": "Narrative review, MDPI, 2025",
+    "url": "https://www.mdpi.com/2308-3417/10/6/143",
+    "summary": "This review synthesizes evidence that regular physical activity improves executive function, memory, and processing speed in older adults, including those with mild cognitive impairment or APOE4 genetic risk. Mechanistically, exercise is associated with increased BDNF, IGF-1, and irisin signaling that support neuroplasticity, along with improved glymphatic clearance -- the brain's waste-clearance system implicated in amyloid and tau removal -- and reduced neuroinflammation. The authors note physical activity sits within a broader multidomain framework alongside diet, sleep, and social engagement rather than acting as a standalone protective factor."
+  },
+  {
+    "id": "lifestyle_executive_function_memory",
+    "domain": "exercise_cognitive_reserve",
+    "title": "Cognitive reserve, physical activity, diet, and sleep in healthy older adults",
+    "source": "Aging Clinical and Experimental Research, Springer, 2026",
+    "url": "https://link.springer.com/article/10.1007/s40520-026-03362-y",
+    "summary": "In a cross-sectional study of 204 cognitively healthy older adults, researchers examined how cognitive reserve, lifelong physical activity, dietary habits, and sleep duration related independently to executive function and long-term memory, controlling for smoking, alcohol use, and chronic conditions. The work reinforces that these four modifiable factors each contribute distinct, only partially overlapping protective associations with different cognitive domains, supporting a multidomain rather than single-factor approach to brain health maintenance."
+  },
+  {
+    "id": "mind_diet_rct_nejm",
+    "domain": "diet_nutrition",
+    "title": "Randomized controlled trial of the MIND diet for cognitive decline prevention",
+    "source": "New England Journal of Medicine, 2023",
+    "url": "https://www.nejm.org/doi/full/10.1056/NEJMoa2302368",
+    "summary": "This two-site RCT randomized older adults with a family history of dementia, elevated BMI, and a suboptimal baseline diet to either the MIND diet (a hybrid of the Mediterranean and DASH diets emphasizing leafy greens, berries, nuts, olive oil, and whole grains) or a calorie-matched control diet for three years, with cognition assessed via a 12-test battery. It is among the more rigorous diet-cognition trials to date, and findings have been interpreted cautiously by the field -- later commentary (see SMARRT/FINGER caution entry) notes that while plausible, diet-only effects on cognition have been less consistent across studies than combined multidomain interventions."
+  },
+  {
+    "id": "lifestyle_caution_nature",
+    "domain": "diet_nutrition",
+    "title": "Can diet and exercise really prevent Alzheimer's? A field-wide caution",
+    "source": "Nature News Feature, 2025",
+    "url": "https://www.nature.com/articles/d41586-025-02925-9",
+    "summary": "This field overview is useful precisely because it tempers enthusiasm: it notes that while trials like FINGER and SMARRT (a two-year RCT treating modifiable risk factors such as uncontrolled hypertension) show encouraging signals, most positive results come from people with mild cognitive decline rather than established dementia, where intervention effects are far weaker. The MIND diet specifically is flagged as showing initially promising but not consistently replicated effects across subsequent studies, and researchers caution against overstating any single intervention's individual contribution outside a combined, monitored program."
+  },
+  {
+    "id": "lifestyle_asia_west_review",
+    "domain": "diet_nutrition",
+    "title": "Diet and physical activity for dementia risk reduction: Western and Asian approaches compared",
+    "source": "ScienceDirect review, 2025",
+    "url": "https://www.sciencedirect.com/science/article/pii/S2274580724006204",
+    "summary": "This comparative review finds consistent evidence that aerobic, resistance, balance, and dual-task exercise enhance neurogenesis and cerebral blood flow and are linked to delayed cognitive decline across cultural contexts, while dietary approaches differ by region -- structured plans like MIND/Mediterranean diets in Western literature versus traditional mind-body practices (e.g., Tai Chi) and antioxidant- and polyphenol-rich diets in Asian literature. The authors argue that culturally tailored delivery of the same underlying principles (movement plus nutrient-dense, anti-inflammatory eating) likely matters more for adherence than which specific named diet is used."
+  },
+  {
+    "id": "nun_study_idea_density",
+    "domain": "cognitive_reserve_early_life",
+    "title": "Linguistic ability in early life and Alzheimer's disease in late life: the Nun Study",
+    "source": "Snowdon, Kemper, et al., JAMA, 1996; 30-year retrospective, Alzheimer's & Dementia, 2025",
+    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC11852352/",
+    "summary": "In one of the most cited findings in cognitive aging research, autobiographies written by Catholic nuns in their early twenties were scored decades later for 'idea density' -- the number of distinct ideas expressed per ten words. Low idea density in young adulthood was strongly associated with both clinical dementia and confirmed Alzheimer's neuropathology assessed via brain autopsy roughly 60 years later, with researchers reporting they could predict eventual Alzheimer's pathology with notably high accuracy from these early writings alone. Subsequent analysis found this association held even among nuns who did not carry the APOE4 risk allele, suggesting idea density captures something about early-life cognitive and neurological development distinct from genetic risk -- interpreted by the field as evidence that early intellectual engagement contributes to 'cognitive reserve.'"
+  },
+  {
+    "id": "nun_study_exercise_overview",
+    "domain": "cognitive_reserve_early_life",
+    "title": "The Nun Study: lifestyle and longitudinal findings overview",
+    "source": "Nun Study retrospective summaries, 2025-2026",
+    "url": "https://en.wikipedia.org/wiki/Nun_Study",
+    "summary": "Beyond linguistic ability, the Nun Study's homogeneous-lifestyle cohort (678 sisters with similar diets, routines, and no smoking/alcohol confounds) allowed researchers to isolate other protective associations, including that habitual daily exercise was inversely associated with developing Alzheimer's, and that this benefit appeared even among participants who only began exercising later in life -- suggesting it is not solely a marker of pre-existing health but a modifiable protective behavior at any age."
+  },
+  {
+    "id": "semantic_fluency_neurodegeneration",
+    "domain": "cognitive_training",
+    "title": "Semantic fluency decline tracks early Alzheimer's-related brain changes",
+    "source": "PMC7403823, longitudinal cohort study (n=2,261)",
+    "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7403823/",
+    "summary": "Tracking over 2,000 non-demented older adults over several years, this study found that semantic (category) fluency declined fastest specifically in those at elevated Alzheimer's risk -- APOE4 carriers and those who later progressed to MCI or dementia -- while phonemic (letter) fluency stayed comparatively stable except very close to dementia onset. Lower semantic fluency was further linked to faster cortical thinning and reduced metabolic activity in temporal and parietal regions associated with Alzheimer's specifically, distinguishing it from general age-related decline."
+  },
+  {
+    "id": "verbal_fluency_mci_screening",
+    "domain": "cognitive_training",
+    "title": "Verbal fluency as a screening tool for mild cognitive impairment",
+    "source": "PMC9153280, McDonnell et al.",
+    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC9153280/",
+    "summary": "In a sample of 332 participants spanning healthy aging, amnestic MCI, and Alzheimer's disease, both semantic (animal) and phonemic (letter) fluency significantly predicted diagnostic group, but semantic fluency carried substantially more discriminating power. The authors conclude semantic fluency testing is a practically efficient screening signal in time-limited settings, useful for flagging people who may benefit from more thorough follow-up assessment -- though explicitly not a stand-alone diagnostic tool."
+  },
+  {
+    "id": "social_engagement_fingers_component",
+    "domain": "social_engagement",
+    "title": "Social activity as a structural component of multidomain dementia-prevention trials",
+    "source": "Synthesized from FINGER/WW-FINGERS trial designs, 2025",
+    "url": "https://www.alz.org/wwfingers",
+    "summary": "Across FINGER and its global WW-FINGERS adaptations, structured social activity is included as one of the five core intervention domains alongside diet, exercise, cognitive training, and vascular risk management -- not as an afterthought but as a load-bearing component shown to contribute to the roughly 25% greater cognitive improvement seen in intervention versus control groups. Trial designers note social engagement components also improve program adherence and retention, which in turn supports better outcomes across the other domains."
+  },
+  {
+    "id": "digital_tools_social_engagement",
+    "domain": "social_engagement",
+    "title": "Digital delivery of social and cognitive engagement in dementia prevention",
+    "source": "PMC12762639, WW-FINGERS digital tools review, 2025",
+    "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12762639/",
+    "summary": "A review of digital tool adoption across 19 published WW-FINGERS protocols found social engagement components delivered via web platforms and apps (not just in-person) enhanced usability and participation, with the Maintain Your Brain (Australia) trial demonstrating that a fully internet-based multidomain lifestyle program -- including social and cognitive engagement elements -- can still produce measurable cognitive and health benefits, broadening access beyond clinic-based delivery."
+  },
+  {
+    "id": "verbal_fluency_qualitative_clusters",
+    "domain": "cognitive_training",
+    "title": "Qualitative verbal fluency patterns differentiate subjective decline from MCI",
+    "source": "PMC12738191, Cherkasov et al., 2025",
+    "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12738191/",
+    "summary": "Comparing 40 patients with subjective cognitive decline to 51 with diagnosed MCI, this study found that beyond raw fluency word counts, qualitative clustering patterns -- how words group into semantic categories and how often a speaker switches between them -- added meaningful discriminative information beyond the simple count, with significant differences in total fluency score and several cluster-pattern measures between the two groups. This supports analyzing the structure of a fluency response, not just the tally, which NeuroLens's clustering-adjacent metrics (lexical diversity, idea density) aim to approximate in a simplified form."
+  }
+]

rag_engine.py

@@ -0,0 +1,325 @@
+"""
+rag_engine.py — Stage 3 of NeuroLens: Grounded, Personalized Prevention Coaching
+
+Retrieves relevant literature from a curated corpus (data/corpus.json) based on
+which linguistic biomarkers from Stage 2 stood out, then generates a citation-
+grounded coaching summary via Groq's inference API. Includes a RAGAS-style
+faithfulness check on the generated text against the retrieved sources.
+
+Embedding backend:
+- Primary (production / HuggingFace Spaces): sentence-transformers
+  'intfloat/e5-large-v2' + FAISS, for high-quality semantic retrieval.
+- Fallback (offline / restricted environments, e.g. sandboxed dev): scikit-learn
+  TF-IDF + cosine similarity. Automatically used if sentence-transformers or
+  its model weights aren't available, so the pipeline still runs end-to-end
+  without internet access to the Hugging Face Hub.
+
+This keeps app.py agnostic to which backend is active.
+"""
+
+import json
+import os
+from pathlib import Path
+
+import numpy as np
+
+CORPUS_PATH = Path(__file__).parent / "data" / "corpus.json"
+
+# Maps a Stage 2 biomarker "band" finding to the literature domains most
+# relevant to addressing it. Used to bias retrieval toward useful content
+# rather than retrieving purely by surface similarity to assessment text.
+MARKER_TO_DOMAINS = {
+    "semantic_fluency": ["cognitive_training", "social_engagement"],
+    "phonemic_fluency": ["exercise_cognitive_reserve", "diet_nutrition"],
+    "lexical_diversity": ["cognitive_reserve_early_life", "cognitive_training"],
+    "idea_density": ["cognitive_reserve_early_life", "social_engagement"],
+    "syntactic_complexity": ["exercise_cognitive_reserve", "diet_nutrition"],
+}
+
+
+def load_corpus() -> list:
+    with open(CORPUS_PATH, "r") as f:
+        return json.load(f)
+
+
+class LiteratureRetriever:
+    """Embedding-backed retriever over the curated corpus, with automatic
+    fallback to TF-IDF if a neural embedding backend isn't available."""
+
+    def __init__(self):
+        self.corpus = load_corpus()
+        self.texts = [f"{d['title']}. {d['summary']}" for d in self.corpus]
+        self.backend = None
+        self._init_backend()
+
+    def _init_backend(self):
+        # Try neural embeddings first (production path)
+        try:
+            from sentence_transformers import SentenceTransformer
+            model = SentenceTransformer("intfloat/e5-large-v2")
+            # e5 models expect "passage: " / "query: " prefixes for best results
+            passage_texts = [f"passage: {t}" for t in self.texts]
+            embeddings = model.encode(passage_texts, normalize_embeddings=True)
+            self._st_model = model
+            self._embeddings = np.array(embeddings, dtype="float32")
+            self.backend = "sentence-transformers"
+
+            try:
+                import faiss
+                index = faiss.IndexFlatIP(self._embeddings.shape[1])
+                index.add(self._embeddings)
+                self._faiss_index = index
+                self.backend = "sentence-transformers+faiss"
+            except ImportError:
+                self._faiss_index = None
+            return
+        except Exception:
+            pass
+
+        # Fallback: TF-IDF (always available, no network/model download required)
+        from sklearn.feature_extraction.text import TfidfVectorizer
+        self._vectorizer = TfidfVectorizer(stop_words="english")
+        self._tfidf_matrix = self._vectorizer.fit_transform(self.texts)
+        self.backend = "tfidf_fallback"
+
+    def _retrieve_indices(self, query: str, k: int, candidate_idx=None):
+        if self.backend.startswith("sentence-transformers"):
+            q_emb = self._st_model.encode([f"query: {query}"], normalize_embeddings=True)
+            q_emb = np.array(q_emb, dtype="float32")
+            if candidate_idx is not None and len(candidate_idx) > 0:
+                sub_embeddings = self._embeddings[candidate_idx]
+                sims = sub_embeddings @ q_emb[0]
+                order = np.argsort(-sims)[:k]
+                return [candidate_idx[i] for i in order]
+            else:
+                sims = self._embeddings @ q_emb[0]
+                return list(np.argsort(-sims)[:k])
+        else:
+            from sklearn.metrics.pairwise import cosine_similarity
+            q_vec = self._vectorizer.transform([query])
+            if candidate_idx is not None and len(candidate_idx) > 0:
+                sims = cosine_similarity(q_vec, self._tfidf_matrix[candidate_idx]).flatten()
+                order = np.argsort(-sims)[:k]
+                return [candidate_idx[i] for i in order]
+            else:
+                sims = cosine_similarity(q_vec, self._tfidf_matrix).flatten()
+                return list(np.argsort(-sims)[:k])
+
+    def retrieve_for_domains(self, query: str, domains: list, k: int = 2) -> list:
+        candidate_idx = [i for i, d in enumerate(self.corpus) if d["domain"] in domains]
+        idx = self._retrieve_indices(query, k, candidate_idx=candidate_idx if candidate_idx else None)
+        return [self.corpus[i] for i in idx]
+
+
+def select_retrieval_plan(bands: dict) -> dict:
+    """
+    Given Stage 2's `bands` dict (marker -> 'below_typical_range' / 'within_typical_range'
+    / 'above_typical_range'), decide which markers are worth addressing and which
+    literature domains to pull from for each.
+    """
+    plan = {}
+    for marker, band in bands.items():
+        if band == "below_typical_range" and marker in MARKER_TO_DOMAINS:
+            plan[marker] = MARKER_TO_DOMAINS[marker]
+    return plan
+
+
+def build_retrieved_context(retriever: LiteratureRetriever, profile: dict, k_per_marker: int = 2) -> dict:
+    """Returns {marker: [retrieved doc dicts]} for every marker that needs addressing."""
+    plan = select_retrieval_plan(profile["bands"])
+    retrieved = {}
+    queries = {
+        "semantic_fluency": "interventions to improve semantic memory and category fluency",
+        "phonemic_fluency": "exercise and vascular health for executive function and word retrieval",
+        "lexical_diversity": "building cognitive reserve through lifelong learning and engagement",
+        "idea_density": "early and ongoing intellectual engagement and cognitive reserve",
+        "syntactic_complexity": "physical activity and diet supporting language processing and cognitive load",
+    }
+    for marker, domains in plan.items():
+        docs = retriever.retrieve_for_domains(queries.get(marker, marker), domains, k=k_per_marker)
+        retrieved[marker] = docs
+    return retrieved
+
+
+# ---------------------------------------------------------------------------
+# Generation — uses Groq's inference API running an open-weight model
+# (openai/gpt-oss-120b). See README.md for API key setup.
+# ---------------------------------------------------------------------------
+COACHING_SYSTEM_PROMPT = """You are a brain-health research communicator writing for NeuroLens, \
+a non-clinical research demonstration prototype. You write supportive, non-alarmist, \
+evidence-grounded coaching summaries for users based on their performance on simplified \
+cognitive-linguistic tasks.
+
+Strict rules:
+- Only state claims that are directly supported by the provided source excerpts. Do not \
+add outside knowledge or invent statistics.
+- Every factual claim about research findings must be attributable to one of the provided \
+sources; refer to sources naturally (e.g., "Research from the FINGER trial suggests...").
+- Never use clinical or diagnostic language (no "you may have," "this indicates early signs of," \
+"this is a symptom of"). Frame everything as "your results on this task" and "research on this topic," \
+never as a statement about the user's health status.
+- Tone: warm, encouraging, plain-language. No alarmism.
+- End with one sentence reminding the user this is a research prototype, not a clinical assessment.
+- Keep the full response under 250 words.
+- Respond with plain prose only — no markdown headers, no preamble like "Here is your summary."
+"""
+
+GROQ_MODEL = "openai/gpt-oss-120b"  # see README.md for API key setup at console.groq.com
+
+
+def generate_coaching(profile: dict, retrieved_context: dict, client=None, model: str = GROQ_MODEL) -> dict:
+    """
+    Calls Groq's API to generate a grounded coaching summary.
+    `client` should be an instantiated groq.Groq() client; if None, one is
+    created from the GROQ_API_KEY environment variable (see README.md for setup).
+    Returns {"text": str, "sources_used": [...]}
+    """
+    from groq import Groq
+
+    if client is None:
+        client = Groq()
+
+    if not retrieved_context:
+        return {
+            "text": ("Your results across these short tasks fell within the typical ranges used for "
+                      "this demo, so there's no specific area to highlight right now. Maintaining a mix "
+                      "of physical activity, social engagement, varied learning, sleep, and a nutrient-dense "
+                      "diet is broadly supported by the brain-health prevention literature regardless. "
+                      "Reminder: NeuroLens is a research prototype, not a clinical assessment."),
+            "sources_used": [],
+        }
+
+    context_blocks = []
+    all_sources = []
+    for marker, docs in retrieved_context.items():
+        for d in docs:
+            context_blocks.append(
+                f"[Source: {d['title']} — {d['source']}]\n{d['summary']}"
+            )
+            all_sources.append(d)
+
+    context_text = "\n\n".join(context_blocks)
+    markers_text = ", ".join(retrieved_context.keys())
+
+    user_prompt = f"""The user's results on these tasks were below the typical comparison range: {markers_text}.
+
+Here are the relevant source excerpts to ground your response (use ONLY these):
+
+{context_text}
+
+Write the personalized coaching summary now."""
+
+    response = client.chat.completions.create(
+        model=model,
+        max_tokens=1000,
+        reasoning_effort="low",
+        messages=[
+            {"role": "system", "content": COACHING_SYSTEM_PROMPT},
+            {"role": "user", "content": user_prompt},
+        ],
+    )
+
+    text = (response.choices[0].message.content or "").strip()
+    if not text:
+        raise RuntimeError(
+            "Groq returned empty content for coaching generation (model spent its token "
+            "budget on internal reasoning). Try increasing max_tokens further."
+        )
+
+    # de-duplicate sources by id
+    seen_ids = set()
+    unique_sources = []
+    for s in all_sources:
+        if s["id"] not in seen_ids:
+            seen_ids.add(s["id"])
+            unique_sources.append(s)
+
+    return {"text": text, "sources_used": unique_sources}
+
+
+# ---------------------------------------------------------------------------
+# Faithfulness evaluation (RAGAS-style) — also via Groq.
+# ---------------------------------------------------------------------------
+FAITHFULNESS_SYSTEM_PROMPT = """You are a strict fact-checker. You will be given a generated \
+coaching text and a set of source excerpts it was supposed to be grounded in.
+
+Break the generated text into individual factual claims about research findings. For each \
+claim, determine whether it is directly supported by the source excerpts (SUPPORTED), \
+partially supported / overstated relative to the sources (PARTIAL), or not supported / \
+hallucinated (UNSUPPORTED).
+
+Respond ONLY with valid JSON in this exact format — no other text, no markdown code fences, \
+no explanation before or after the JSON:
+{
+  "claims": [
+    {"claim": "...", "verdict": "SUPPORTED|PARTIAL|UNSUPPORTED", "reason": "..."}
+  ],
+  "faithfulness_score": <float between 0 and 1, fraction of claims that are SUPPORTED>
+}
+Keep each "reason" to one short sentence. Do not show your reasoning process — output the
+JSON object directly.
+"""
+
+
+def check_faithfulness(generated_text: str, sources_used: list, client=None, model: str = GROQ_MODEL) -> dict:
+    from groq import Groq
+    import json as _json
+
+    if client is None:
+        client = Groq()
+
+    if not sources_used:
+        return {"claims": [], "faithfulness_score": 1.0, "note": "No sources retrieved; generic fallback text used."}
+
+    source_text = "\n\n".join(f"[{s['title']}]\n{s['summary']}" for s in sources_used)
+
+    user_prompt = f"""GENERATED TEXT:
+{generated_text}
+
+SOURCE EXCERPTS:
+{source_text}
+
+Evaluate faithfulness now. Respond with JSON only."""
+
+    response = client.chat.completions.create(
+        model=model,
+        max_tokens=2000,
+        reasoning_effort="low",
+        messages=[
+            {"role": "system", "content": FAITHFULNESS_SYSTEM_PROMPT},
+            {"role": "user", "content": user_prompt},
+        ],
+        # Note: response_format={"type": "json_object"} is intentionally omitted —
+        # it has been unreliable with this model (see Groq community reports of
+        # json_validate_failed / structured outputs being ignored on gpt-oss-120b).
+        # Plain prompt-based JSON instructions + the parsing below are more robust.
+    )
+    raw = (response.choices[0].message.content or "").strip()
+    if not raw:
+        return {"claims": [], "faithfulness_score": None,
+                "note": "Groq returned empty content (model spent its token budget on internal "
+                        "reasoning). Try increasing max_tokens further."}
+    raw = raw.removeprefix("```json").removeprefix("```").removesuffix("```").strip()
+    try:
+        return _json.loads(raw)
+    except _json.JSONDecodeError:
+        return {"claims": [], "faithfulness_score": None, "note": "Could not parse evaluator output", "raw": raw}
+
+
+if __name__ == "__main__":
+    retriever = LiteratureRetriever()
+    print(f"Backend: {retriever.backend}, corpus size: {len(retriever.corpus)}")
+
+    fake_bands = {
+        "semantic_fluency": "below_typical_range",
+        "phonemic_fluency": "within_typical_range",
+        "lexical_diversity": "within_typical_range",
+        "idea_density": "below_typical_range",
+        "syntactic_complexity": "within_typical_range",
+    }
+    fake_profile = {"bands": fake_bands}
+    context = build_retrieved_context(retriever, fake_profile)
+    for marker, docs in context.items():
+        print(f"\n--- {marker} ---")
+        for d in docs:
+            print(" ", d["title"])

requirements.txt

@@ -1,3 +1,10 @@
-altair
-pandas
-streamlit
+streamlit>=1.38
+streamlit-autorefresh>=1.0
+groq>=0.13
+spacy>=3.7
+https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl
+nltk>=3.9
+scikit-learn>=1.5
+numpy>=1.26
+sentence-transformers>=3.0
+faiss-cpu>=1.8