feat: deploy Phase 1 architecture fixes and Phase 2 dataset collection & evaluation infrastructure

MEXAR_Phase1_Architecture_Fixes.md

@@ -0,0 +1,309 @@
+# MEXAR Phase 1 — Architecture Fixes to Match the Paper
+
+**Purpose of this document:** hand this to an AI coding agent (Antigravity or similar) working directly in the `Mexar` repository (`github.com/devrajsinh2012/Mexar`). This is Phase 1 only: it closes the gap between what the paper *"Hybrid Retrieval and Claim Verification: An Empirical Study of Constrained RAG for High-Stakes Professional Domains"* claims about the system's architecture, and what the code currently does. No dataset work in this document, that's Phase 2, handled separately.
+
+**Presentation date:** July 30-31, 2026. Target for this phase: 3 days.
+
+**Ground rule for the agent:** every change below must keep the app deployable on the current stack (FastAPI backend on Hugging Face Spaces, Supabase Postgres + pgvector, React frontend on Vercel). Don't introduce services that need infra Dev doesn't already have.
+
+---
+
+## 1.0 Preconditions
+
+- Add to `backend/requirements.txt`:
+  ```
+  scikit-learn>=1.4.0
+  spacy>=3.7.0
+  ```
+- After install, download the small English model: `python -m spacy download en_core_web_sm`. Add this as a step in `Dockerfile` so it survives Hugging Face Spaces rebuilds.
+- Verify the pgvector extension version on the Supabase project supports HNSW (needs pgvector >= 0.5.0). Run `SELECT extversion FROM pg_extension WHERE extname = 'vector';`. If below 0.5.0, run `ALTER EXTENSION vector UPDATE;` before touching the index migration in 1.2.
+
+---
+
+## 1.1 Domain Guardrail: replace the heuristic with real TF-IDF + NER + Jaccard
+
+**Why:** Paper Section III-A defines the guardrail as TF-IDF term weighting (Eq. 1), NER-extracted domain entities, and Jaccard similarity between query and signature (Eq. 2). The current `_check_guardrail()` in `backend/modules/reasoning_engine.py` is a hand-tuned fuzzy-string-match heuristic with arbitrary bonus weights and no TF-IDF or NER anywhere. This needs to become the real thing.
+
+**New file: `backend/utils/domain_signature.py`**
+
+```python
+"""
+MEXAR - Domain Signature Construction
+Implements Section III-A: TF-IDF lexical signature + NER entity signature.
+"""
+from typing import List, Dict, Set, Tuple
+from sklearn.feature_extraction.text import TfidfVectorizer
+import spacy
+import logging
+
+logger = logging.getLogger(__name__)
+_nlp = None
+
+def get_nlp():
+    global _nlp
+    if _nlp is None:
+        _nlp = spacy.load("en_core_web_sm")
+    return _nlp
+
+# Entity labels relevant to professional domains (medical/legal/financial)
+RELEVANT_ENTITY_LABELS = {"ORG", "LAW", "PRODUCT", "GPE", "PERSON", "NORP", "EVENT", "FAC"}
+
+def build_tfidf_signature(documents: List[str], top_n: int = 100, tau_tf: float = 0.0) -> List[Tuple[str, float]]:
+    """
+    Eq. 1: w(t, D) = tf(t, D) * log(|D| / |{d in D : t in d}|)
+    Returns top_n (term, weight) pairs above tau_tf, sorted descending by weight.
+    """
+    if not documents or len(documents) < 2:
+        # TF-IDF needs a corpus of multiple "documents" to compute IDF meaningfully.
+        # If compiling from a single uploaded file, split it into paragraph-level
+        # pseudo-documents so IDF is not degenerate.
+        documents = _split_into_pseudo_documents(documents[0]) if documents else []
+
+    vectorizer = TfidfVectorizer(
+        max_features=2000,
+        stop_words="english",
+        ngram_range=(1, 2),
+        min_df=1,
+    )
+    matrix = vectorizer.fit_transform(documents)
+    scores = matrix.sum(axis=0).A1
+    terms = vectorizer.get_feature_names_out()
+    pairs = [(t, float(s)) for t, s in zip(terms, scores) if s > tau_tf]
+    pairs.sort(key=lambda x: x[1], reverse=True)
+    return pairs[:top_n]
+
+def _split_into_pseudo_documents(text: str, chunk_size: int = 500) -> List[str]:
+    words = text.split()
+    return [" ".join(words[i:i+chunk_size]) for i in range(0, len(words), chunk_size)] or [text]
+
+def extract_ner_entities(documents: List[str], tau_ent: int = 2, max_docs_to_scan: int = 200) -> List[str]:
+    """
+    Extracts named entities appearing in at least tau_ent documents.
+    Capped at max_docs_to_scan for latency (spaCy is not free at compile time).
+    """
+    nlp = get_nlp()
+    doc_freq: Dict[str, int] = {}
+    for doc_text in documents[:max_docs_to_scan]:
+        seen_in_this_doc: Set[str] = set()
+        for ent in nlp(doc_text[:5000]).ents:  # cap per-doc length for speed
+            if ent.label_ in RELEVANT_ENTITY_LABELS:
+                key = ent.text.lower().strip()
+                if key and key not in seen_in_this_doc:
+                    seen_in_this_doc.add(key)
+        for key in seen_in_this_doc:
+            doc_freq[key] = doc_freq.get(key, 0) + 1
+    return [k for k, v in doc_freq.items() if v >= tau_ent]
+
+def build_domain_signature(
+    documents: List[str],
+    tau_tf: float = 0.0,
+    tau_ent: int = 2,
+    top_n_lexical: int = 100,
+) -> Dict[str, List]:
+    """Returns Sigma = Sigma_lex ∪ Sigma_ent as described in Section III-A, kept separate for inspection."""
+    lexical_pairs = build_tfidf_signature(documents, top_n=top_n_lexical, tau_tf=tau_tf)
+    lexical_terms = [t for t, _ in lexical_pairs]
+    entities = extract_ner_entities(documents, tau_ent=tau_ent)
+    return {
+        "lexical": lexical_terms,
+        "lexical_weights": {t: w for t, w in lexical_pairs},
+        "entities": entities,
+        "combined": sorted(set(lexical_terms) | set(e for e in entities)),
+    }
+```
+
+**Modify `backend/modules/knowledge_compiler.py`:**
+Replace `_extract_domain_signature()` so it calls `build_domain_signature()` from the new module, passing in the raw per-source-document text list (not the flattened `text_context` string, we need per-document granularity for TF-IDF's document-frequency term to mean anything). Store the result's `combined` list in the existing `domain_signature` JSONB column, and add the `lexical_weights` dict to a **new** JSONB column (see migration below) so the guardrail can compute a weighted Jaccard rather than a flat one if useful later.
+
+**New migration** (Alembic or raw SQL depending on how `backend/migrations` is structured, check `backend/migrations/README.md` for the convention already in use):
+```sql
+ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_signature_weights JSONB;
+ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_entities JSONB;
+```
+
+**Rewrite `_check_guardrail()` in `backend/modules/reasoning_engine.py`:**
+Delete the fuzzy-match/bonus-score implementation entirely. Replace with real Eq. 2:
+
+```python
+def _check_guardrail(self, query: str, domain_signature: List[str], prompt_analysis: Dict[str, Any]) -> Tuple[bool, float]:
+    """Eq. 2: sim(q, Sigma) = |Phi(q) ∩ Sigma| / |Phi(q)|"""
+    from utils.domain_signature import get_nlp
+    nlp = get_nlp()
+    query_doc = nlp(query.lower())
+    # Phi(q): lemmatized content tokens, excluding stopwords/punctuation
+    phi_q = {
+        tok.lemma_ for tok in query_doc
+        if not tok.is_stop and not tok.is_punct and len(tok.text) > 2
+    }
+    sigma = set(s.lower() for s in (domain_signature or []))
+
+    if not phi_q:
+        return False, 0.0
+
+    intersection = phi_q & sigma
+    score = len(intersection) / len(phi_q)
+
+    is_in_domain = score >= self.DOMAIN_SIMILARITY_THRESHOLD
+    logger.info(f"Guardrail: sim={score:.3f}, |Phi(q)|={len(phi_q)}, |intersection|={len(intersection)}, in_domain={is_in_domain}")
+    return is_in_domain, score
+```
+
+**Tune `DOMAIN_SIMILARITY_THRESHOLD`:** the current 0.05 was calibrated against the *old* heuristic scorer, it will not transfer to real Jaccard similarity, which tends to produce lower absolute values on short queries. Don't guess this: after wiring the new guardrail, run `backend/evaluation/guardrail_analysis.py` against a small hand-labeled set of ~20 in-domain and ~20 out-of-domain queries per existing test_data domain, sweep the threshold from 0.05 to 0.4 in steps of 0.05, and pick the value that maximizes F1. Record that sweep, it becomes supporting evidence for Table IV in the paper instead of a guessed constant.
+
+**Acceptance test:** for the medical test agent, a query like "what is the recommended treatment for stage 2 hypertension" should be accepted; a query like "how do I file a small claims lawsuit" should be rejected. Log both the old and new guardrail score during a transition period to sanity-check before removing the old code path.
+
+---
+
+## 1.2 Vector index: IVFFlat → HNSW
+
+**Why:** Paper Section III-B explicitly claims HNSW indexing. The current migration (`backend/migrations/hybrid_search_function.sql`, lines ~98-101) creates an IVFFlat index instead.
+
+**New migration file** `backend/migrations/switch_to_hnsw_index.sql`:
+```sql
+-- Drop the old IVFFlat index
+DROP INDEX IF EXISTS idx_document_chunks_embedding;
+
+-- Create HNSW index to match paper Section III-B
+CREATE INDEX IF NOT EXISTS idx_document_chunks_embedding_hnsw
+ON document_chunks USING hnsw (embedding vector_cosine_ops)
+WITH (m = 16, ef_construction = 64);
+
+COMMENT ON INDEX idx_document_chunks_embedding_hnsw IS
+  'HNSW index for cosine similarity search, m=16, ef_construction=64 (Section III-B)';
+```
+Apply via `Supabase:apply_migration` (or however existing migrations are run, check for a runner script referenced in `backend/migrations/README.md`) rather than editing the old file, so history is preserved.
+
+**Note for the agent:** HNSW build time scales worse than IVFFlat with corpus size, but at the 500-600 document / few-thousand-chunk scale planned for the presentation, this is a non-issue. No further tuning needed.
+
+**Acceptance test:** `\d document_chunks` in psql (or `SELECT indexdef FROM pg_indexes WHERE tablename = 'document_chunks';`) should show the HNSW index, not IVFFlat. Re-run a handful of existing queries and confirm retrieval still returns coherent top-5 chunks (HNSW is approximate, results may reorder slightly versus IVFFlat, that's expected and fine).
+
+---
+
+## 1.3 RRF weighting: reconcile code and paper
+
+**Why:** Paper Eq. 5 is unweighted RRF. `backend/migrations/hybrid_search_function.sql` applies a 0.6 semantic / 0.4 keyword weighting inside the SQL function.
+
+**Decision for this project (recommended, don't re-litigate):** keep the 0.6/0.4 weighting in code, it's a reasonable and defensible design choice, and update the paper instead. This is less risky than ripping out a tuned weighting under time pressure. The agent does not need to touch `hybrid_search_function.sql` for this item, just flag it in a "paper edits needed" note for later.
+
+If Dev instead decides to match the paper exactly, the one-line change is setting `semantic_weight := 0.5; keyword_weight := 0.5;` in that SQL function, but this is **not** the default action, don't do it unless explicitly told.
+
+---
+
+## 1.4 Faithfulness verification: make DeBERTa-v3 NLI the real, primary signal
+
+**Why:** this is the most important fix. Paper Section III-C and the entire Related Work argument against LLM-judging-LLM circularity claim the production system verifies claims with "an NLI model (DeBERTa-v3-large fine-tuned on MNLI)" using per-document max entailment (Eq. 6). In the actual code:
+- The primary confidence-driving call in `reasoning_engine.reason()` (Step 6) is `self.faithfulness_scorer.score(answer, context)`, which is `FaithfulnessScorer` in `backend/utils/faithfulness.py`, an LLM (Groq) extracting claims and then Groq again answering YES/NO. That's exactly the circularity the paper argues against.
+- The actual NLI model in the repo, `BartNLIScorer`, uses `facebook/bart-large-mnli`, not DeBERTa-v3-large, and it's computed only as a side "reviewer feedback" score (`bart_nli_result` in `reason()`), it never feeds the confidence calculation.
+- The NLI scorer concatenates all retrieved context into a single premise string, it does not implement the per-document max in Eq. 6.
+
+**Model choice:** use `cross-encoder/nli-deberta-v3-large` from Hugging Face. This is a DeBERTa-v3-large architecture fine-tuned specifically for NLI (SNLI + MultiNLI), it is the standard, citable checkpoint people mean when they say "DeBERTa-v3-large fine-tuned on MNLI" in RAG faithfulness papers, and it loads cleanly through `sentence-transformers`'s `CrossEncoder` class, which is already a dependency.
+
+**Modify `backend/utils/faithfulness.py`:** replace the `BartNLIScorer` class with:
+
+```python
+class DebertaNLIScorer:
+    """
+    Primary faithfulness verifier per Section III-C.
+    Per-claim, per-document max entailment probability (Eq. 6),
+    aggregated as the fraction of claims exceeding tau_ent (Eq. 7).
+    """
+    LABEL_MAP = {0: "contradiction", 1: "entailment", 2: "neutral"}  # verify against model card at load time
+    TAU_ENT = 0.7
+
+    def __init__(self):
+        self._model = None
+
+    @property
+    def model(self):
+        if self._model is None:
+            from sentence_transformers import CrossEncoder
+            logger.info("Loading cross-encoder/nli-deberta-v3-large...")
+            self._model = CrossEncoder("cross-encoder/nli-deberta-v3-large")
+            logger.info("DeBERTa-v3 NLI model loaded.")
+        return self._model
+
+    def score(self, answer: str, context_documents: List[str]) -> FaithfulnessResult:
+        """
+        context_documents: list of individual retrieved chunk texts (NOT concatenated),
+        so per-document max in Eq. 6 is meaningful.
+        """
+        import re
+        import numpy as np
+
+        if not answer or not context_documents:
+            return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
+
+        sentences = [s.strip() for s in re.split(r'(?<=[.!?])\s+', answer) if len(s.strip()) > 15][:10]
+        if not sentences:
+            return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
+
+        supported = 0
+        unsupported = []
+
+        for sentence in sentences:
+            pairs = [(doc[:2000], sentence) for doc in context_documents]
+            scores = self.model.predict(pairs)  # shape: (n_docs, 3) logits/probs per label
+            probs = self._softmax(scores)
+            entailment_idx = 1  # confirm this matches the model's actual label order at integration time
+            max_entailment = float(np.max(probs[:, entailment_idx]))
+            if max_entailment > self.TAU_ENT:
+                supported += 1
+            else:
+                unsupported.append(sentence)
+
+        score = supported / len(sentences)
+        return FaithfulnessResult(
+            score=round(score, 3),
+            total_claims=len(sentences),
+            supported_claims=supported,
+            unsupported_claims=unsupported[:5],
+        )
+
+    @staticmethod
+    def _softmax(x):
+        import numpy as np
+        e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
+        return e_x / e_x.sum(axis=-1, keepdims=True)
+```
+
+**Important integration detail for the agent:** verify `cross-encoder/nli-deberta-v3-large`'s actual `id2label` mapping from its `config.json` on first load and print it to logs, cross-encoder NLI checkpoints don't always use the same label order. Don't hardcode `entailment_idx = 1` blindly, confirm it, this is exactly the kind of silent bug that would quietly invalidate the reported faithfulness numbers.
+
+**Modify `backend/modules/reasoning_engine.py`, `reason()` method:**
+- Rename the old `FaithfulnessScorer` usage. Keep the class in the codebase (don't delete it), but relabel its role: it becomes the **"LLM-as-judge baseline"** used for baseline comparison tables (this is genuinely useful, RAGAS-style LLM self-eval is a legitimate baseline to compare against in Table I/III, just not the production signal).
+- Swap Step 6 to call `self.deberta_nli_scorer.score(answer, [c.content for c in top_chunks])` (passing the list of individual chunks, not the concatenated `context` string) as the **primary** `faithfulness_result` used in `_calculate_confidence()` and in the returned `explainability` payload.
+- Keep computing the old LLM-based score too, but store it under a clearly-labeled `"llm_judge_baseline_score"` key in `explainability.confidence_breakdown`, not as the headline faithfulness number.
+
+**Acceptance test:** feed a known-hallucinated answer (write one manually, e.g. an answer that states a fabricated statistic not present in any retrieved chunk) through the pipeline and confirm the DeBERTa scorer flags it as unsupported. Feed a fully-grounded answer and confirm it scores above `TAU_ENT`. Compare against the LLM-judge baseline score on the same pair, they should usually agree, but if they diverge a lot on the same input, that's worth investigating before trusting either for the paper's numbers.
+
+---
+
+## 1.5 Wiring check
+
+After 1.1-1.4 are done, re-run the existing `backend/evaluation/baseline_runner.py` and `backend/evaluation/benchmark_runner.py` against the current small `test_data/` samples (not for final numbers, just as a smoke test) and confirm nothing throws. Fix any interface mismatches from the `context: str` → `context_documents: List[str]` signature change in the faithfulness scorer, other call sites (`evaluation/metrics.py`, `_run_baseline()` in `reasoning_engine.py`) pass a concatenated string today and will need the same list-of-chunks treatment.
+
+---
+
+## Summary of files touched in Phase 1
+
+| File | Change |
+|---|---|
+| `backend/requirements.txt` | add scikit-learn, spacy |
+| `backend/Dockerfile` | add spacy model download step |
+| `backend/utils/domain_signature.py` | **new** — TF-IDF + NER signature builder |
+| `backend/modules/knowledge_compiler.py` | `_extract_domain_signature()` rewritten to use new module |
+| `backend/modules/reasoning_engine.py` | `_check_guardrail()` rewritten to real Jaccard; Step 6 rewired to DeBERTa scorer |
+| `backend/utils/faithfulness.py` | `BartNLIScorer` replaced with `DebertaNLIScorer` (per-doc max, Eq. 6/7) |
+| `backend/migrations/switch_to_hnsw_index.sql` | **new** — HNSW index migration |
+| `backend/migrations/*` (new file) | add `domain_signature_weights`, `domain_entities` columns |
+| `backend/evaluation/*` | update call sites for the `context: str` → `context_documents: List[str]` signature change |
+
+---
+
+## What Phase 1 does NOT cover (handled in the separate Phase 2 document)
+
+- Fetching real medical/legal/financial documents
+- Building the query sets
+- Running the actual evaluation to generate paper numbers
+- Rewriting the paper's dataset section
+
+Phase 1 is architecture only. Once this is done and verified against the small existing `test_data/` samples, move to the Phase 2 document to bring in real data at the ~500-600 document scale.

MEXAR_Phase2_Dataset_Collection.md

@@ -0,0 +1,291 @@
+# MEXAR Phase 2 — Real Datasets at Reduced, Disclosed Scale
+
+**Purpose of this document:** hand this to an AI coding agent (Antigravity or similar) working directly in the `Mexar` repository (`github.com/devrajsinh2012/Mexar`). This is Phase 2 only: it brings in real documents and real queries so the numbers in the paper are reproducible. Assumes Phase 1 (architecture fixes: TF-IDF/NER guardrail, HNSW index, DeBERTa faithfulness scorer) is already done and verified.
+
+**Presentation date:** July 30-31, 2026. Target for this phase: days 4-11 (about a week).
+
+---
+
+## 2.0 Scope decision (already agreed with Dev)
+
+Target **500-600 total documents** across the three domains for the presentation, not the 5,226 stated in the current paper draft. This is a resource-constrained pilot, and the paper's dataset section (IV-A) gets rewritten to say so honestly. Query sets are also reduced and single-annotator (Dev/Om/Jay), not the 900+600 two-expert-annotated set currently described. This is a normal, defensible thing to disclose in a paper, and far safer than presenting numbers nobody can reproduce live.
+
+**Target split:**
+
+| Domain | Target docs | Source |
+|---|---|---|
+| Medical | ~220 | PubMed Central OA subset |
+| Legal | ~150 | CourtListener (Free Law Project) |
+| Financial | ~130 | SEC EDGAR full-text search |
+| **Total** | **~500** | |
+
+**Important for the agent:** these fetch scripts hit external APIs (PubMed E-utilities, CourtListener, SEC EDGAR) that are outside a typical restricted sandbox's allowed network domains. Run these scripts in Dev's own local dev environment or a CI runner with normal internet access, not inside a network-restricted container.
+
+---
+
+## 2.1 Medical corpus — PubMed Central OA
+
+**New script:** `backend/scripts/fetch_pubmed.py`
+
+- Use the NCBI E-utilities API (`esearch` + `efetch`), no key required for low-volume use, but request one from NCBI if fetching more than 3 requests/sec.
+- Query terms per subdomain, run three separate searches and cap results:
+  - `"cardiology treatment guideline"[Title/Abstract] AND "open access"[filter]`
+  - `"oncology treatment protocol"[Title/Abstract] AND "open access"[filter]`
+  - `"internal medicine diagnostic criteria"[Title/Abstract] AND "open access"[filter]`
+- Pull ~75 docs per subdomain (≈220 total).
+- Save each as plain text (abstract + body if available via the OA subset bulk XML) into `test_data/medical_real/`, one file per document, named by PMCID, e.g. `PMC1234567.txt`.
+- Respect NCBI's rate limits (3 req/sec without an API key, 10/sec with one) and identify yourself via the `email` parameter on every request, per NCBI's usage policy.
+- Log a manifest file `test_data/medical_real/manifest.json` mapping PMCID → title → subdomain → source URL, this becomes supporting material for the paper's dataset section and makes the corpus auditable.
+
+**Skeleton:**
+```python
+"""
+MEXAR - PubMed Central OA fetcher for the medical corpus.
+Run this locally (not in a network-restricted sandbox).
+"""
+import requests
+import time
+import json
+import os
+
+NCBI_EMAIL = "your_email@example.com"  # required by NCBI usage policy
+OUTPUT_DIR = "test_data/medical_real"
+SEARCH_TERMS = {
+    "cardiology": '"cardiology treatment guideline"[Title/Abstract] AND "open access"[filter]',
+    "oncology": '"oncology treatment protocol"[Title/Abstract] AND "open access"[filter]',
+    "internal_medicine": '"internal medicine diagnostic criteria"[Title/Abstract] AND "open access"[filter]',
+}
+DOCS_PER_SUBDOMAIN = 75
+
+def esearch(term, retmax):
+    url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
+    params = {"db": "pmc", "term": term, "retmax": retmax, "retmode": "json", "email": NCBI_EMAIL}
+    r = requests.get(url, params=params)
+    r.raise_for_status()
+    return r.json()["esearchresult"]["idlist"]
+
+def efetch_fulltext(pmc_id):
+    url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi"
+    params = {"db": "pmc", "id": pmc_id, "rettype": "full", "retmode": "xml", "email": NCBI_EMAIL}
+    r = requests.get(url, params=params)
+    r.raise_for_status()
+    return r.text  # parse out body text with an XML parser before saving as plain text
+
+def main():
+    os.makedirs(OUTPUT_DIR, exist_ok=True)
+    manifest = []
+    for subdomain, term in SEARCH_TERMS.items():
+        ids = esearch(term, DOCS_PER_SUBDOMAIN)
+        for pmc_id in ids:
+            time.sleep(0.34)  # stay under 3 req/sec
+            try:
+                xml_text = efetch_fulltext(pmc_id)
+                # TODO: parse XML body text out with e.g. lxml, strip tags, save plain text
+                plain_text = xml_text  # placeholder, replace with real extraction
+                out_path = os.path.join(OUTPUT_DIR, f"PMC{pmc_id}.txt")
+                with open(out_path, "w", encoding="utf-8") as f:
+                    f.write(plain_text)
+                manifest.append({"pmc_id": pmc_id, "subdomain": subdomain, "path": out_path})
+            except Exception as e:
+                print(f"Failed to fetch PMC{pmc_id}: {e}")
+    with open(os.path.join(OUTPUT_DIR, "manifest.json"), "w") as f:
+        json.dump(manifest, f, indent=2)
+    print(f"Fetched {len(manifest)} medical documents.")
+
+if __name__ == "__main__":
+    main()
+```
+The agent should replace the placeholder XML-to-text extraction with a real parser (e.g. `lxml` or `BeautifulSoup`) that pulls the `<body>` text out of the PMC XML format and strips markup, don't save raw XML as the "plain text" corpus.
+
+---
+
+## 2.2 Legal corpus — CourtListener
+
+**New script:** `backend/scripts/fetch_courtlistener.py`
+
+- Use the CourtListener REST API (`https://www.courtlistener.com/api/rest/v4/`), requires a free account and API token (set as `COURTLISTENER_TOKEN` env var).
+- Pull opinions/case summaries across contract law, IP, and corporate governance search terms, plus a handful of statutory/regulatory text pages if accessible via their `opinions` and `search` endpoints.
+- Target ~50 docs per subdomain (≈150 total).
+- Save the opinion text (or summary if full text is very long, truncate to a reasonable chunking size, e.g. first 3000 words) into `test_data/legal_real/`, one file per case, named by CourtListener opinion ID.
+- Same manifest pattern as 2.1: `test_data/legal_real/manifest.json` with case name, subdomain, court, date, source URL.
+
+**Skeleton:**
+```python
+"""
+MEXAR - CourtListener fetcher for the legal corpus.
+Requires COURTLISTENER_TOKEN env var. Run locally.
+"""
+import requests
+import os
+import json
+
+TOKEN = os.environ["COURTLISTENER_TOKEN"]
+HEADERS = {"Authorization": f"Token {TOKEN}"}
+BASE_URL = "https://www.courtlistener.com/api/rest/v4"
+OUTPUT_DIR = "test_data/legal_real"
+SEARCH_TERMS = {
+    "contract_law": "breach of contract damages",
+    "intellectual_property": "patent infringement trademark",
+    "corporate_governance": "fiduciary duty shareholder derivative",
+}
+DOCS_PER_SUBDOMAIN = 50
+
+def search_opinions(query, count):
+    url = f"{BASE_URL}/search/"
+    params = {"q": query, "type": "o", "order_by": "score desc"}
+    results = []
+    while len(results) < count:
+        r = requests.get(url, headers=HEADERS, params=params)
+        r.raise_for_status()
+        data = r.json()
+        results.extend(data["results"])
+        if not data.get("next"):
+            break
+        url = data["next"]
+        params = {}
+    return results[:count]
+
+def main():
+    os.makedirs(OUTPUT_DIR, exist_ok=True)
+    manifest = []
+    for subdomain, term in SEARCH_TERMS.items():
+        opinions = search_opinions(term, DOCS_PER_SUBDOMAIN)
+        for op in opinions:
+            text = op.get("plain_text") or op.get("snippet", "")
+            if not text:
+                continue
+            op_id = op.get("id") or op.get("cluster_id")
+            out_path = os.path.join(OUTPUT_DIR, f"case_{op_id}.txt")
+            with open(out_path, "w", encoding="utf-8") as f:
+                f.write(text[:15000])  # cap length
+            manifest.append({
+                "id": op_id, "subdomain": subdomain,
+                "case_name": op.get("caseName", ""), "path": out_path,
+            })
+    with open(os.path.join(OUTPUT_DIR, "manifest.json"), "w") as f:
+        json.dump(manifest, f, indent=2)
+    print(f"Fetched {len(manifest)} legal documents.")
+
+if __name__ == "__main__":
+    main()
+```
+The agent should verify the actual current CourtListener v4 search response schema (field names like `plain_text` vs `snippet` vs a separate opinion-text endpoint may have changed) before trusting this skeleton verbatim, API responses drift over time.
+
+---
+
+## 2.3 Financial corpus — SEC EDGAR
+
+**New script:** `backend/scripts/fetch_secedgar.py`
+
+- Use SEC EDGAR's full-text search API (`https://efts.sec.gov/LATEST/search-index`) and the regular filing index to pull excerpts from 10-K filings and select analyst/accounting-standard documents.
+- **Mandatory:** SEC requires a descriptive `User-Agent` header identifying the requester (name + email) on every request, or you get blocked. Format: `"YourName YourEmail@example.com"`.
+- Target ~130 docs, mixing filing sections (MD&A, risk factors) rather than entire 10-Ks, which are too long and would dominate the corpus disproportionately relative to medical/legal document sizes.
+- Save into `test_data/financial_real/`, named by CIK + accession number.
+- Same manifest pattern.
+
+**Skeleton:**
+```python
+"""
+MEXAR - SEC EDGAR fetcher for the financial corpus.
+Run locally. Requires a descriptive User-Agent per SEC's fair access policy.
+"""
+import requests
+import os
+import json
+import time
+
+USER_AGENT = "Devrajsinh Gohil djgohil2012@gmail.com"  # SEC requires this format
+HEADERS = {"User-Agent": USER_AGENT}
+OUTPUT_DIR = "test_data/financial_real"
+SEARCH_TERMS = ["risk factors", "management discussion analysis", "GAAP IFRS accounting standards"]
+DOCS_PER_TERM = 45  # ~130 total across 3 terms
+
+def full_text_search(query, count):
+    url = "https://efts.sec.gov/LATEST/search-index?q=%22{}%22&forms=10-K".format(query.replace(" ", "+"))
+    r = requests.get(url, headers=HEADERS)
+    r.raise_for_status()
+    hits = r.json().get("hits", {}).get("hits", [])
+    return hits[:count]
+
+def main():
+    os.makedirs(OUTPUT_DIR, exist_ok=True)
+    manifest = []
+    for term in SEARCH_TERMS:
+        hits = full_text_search(term, DOCS_PER_TERM)
+        for hit in hits:
+            time.sleep(0.2)  # be polite to SEC's rate limits
+            source = hit.get("_source", {})
+            cik = source.get("ciks", ["unknown"])[0]
+            accession = source.get("adsh", "unknown")
+            # TODO: fetch actual filing document text via the filing index URL,
+            # this skeleton only captures search-hit metadata, not full text.
+            excerpt = source.get("display_names", [""])[0]
+            out_path = os.path.join(OUTPUT_DIR, f"{cik}_{accession}.txt")
+            with open(out_path, "w", encoding="utf-8") as f:
+                f.write(excerpt)
+            manifest.append({"cik": cik, "accession": accession, "term": term, "path": out_path})
+    with open(os.path.join(OUTPUT_DIR, "manifest.json"), "w") as f:
+        json.dump(manifest, f, indent=2)
+    print(f"Fetched {len(manifest)} financial documents.")
+
+if __name__ == "__main__":
+    main()
+```
+This one needs the most work from the agent: the full-text search endpoint returns hit metadata, not the actual filing body text. The agent needs to follow through to the actual filing document (via the CIK/accession-number-derived URL under `https://www.sec.gov/Archives/edgar/data/{cik}/{accession}/`) and extract the relevant section text (MD&A, risk factors) from the filing HTML, rather than saving just the search snippet.
+
+---
+
+## 2.4 Ingestion format
+
+Each fetch script should output files in whatever format `backend/modules/data_validator.py`'s parsers already expect, check the `.txt`/`.json` handling there first, don't invent a new schema. Simplest path: one `.txt` file per document, plain text body, filename as a stable ID, so the existing agent-compilation flow (`KnowledgeCompiler.compile()`) can ingest them exactly like a user-uploaded file batch, no special-casing needed in the app itself.
+
+---
+
+## 2.5 Query set authoring
+
+Create `test_data/query_sets/{medical,legal,financial}_queries.json` with this schema:
+```json
+[
+  {
+    "query": "What is the first-line treatment for stage 2 hypertension?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": ["PMC1234567"],
+    "annotator": "devrajsinh"
+  }
+]
+```
+- ~40-50 in-domain queries per domain (authored by whoever on the team is most comfortable with that domain, split the load across Dev/Om/Jay).
+- ~25-35 out-of-domain queries per domain-pair (medical→legal, legal→financial, etc.) for guardrail testing, reusing the structure already implied by Table IV in the paper, just at a smaller count.
+- Single annotator per query is fine at this scale, note this explicitly as a limitation instead of claiming inter-annotator agreement you didn't actually measure.
+
+---
+
+## 2.6 Suggested paper text for Section IV-A
+
+Drafted now so it's ready when the paper gets updated in Phase 4:
+
+> *"Due to resource constraints, this study evaluates on a reduced corpus of approximately 500 documents (220 medical, 150 legal, 130 financial) sourced from PubMed Central Open Access, CourtListener, and SEC EDGAR respectively, with 120-150 in-domain queries and approximately 90 out-of-domain queries authored by the paper's co-authors. We report this as a pilot-scale evaluation and note that a larger, multi-annotator study is needed to confirm these findings generalize at production scale."*
+
+Adjust the exact counts once the fetch scripts finish and real numbers are known.
+
+---
+
+## Summary of new files in Phase 2
+
+| File | Purpose |
+|---|---|
+| `backend/scripts/fetch_pubmed.py` | pulls ~220 medical docs |
+| `backend/scripts/fetch_courtlistener.py` | pulls ~150 legal docs |
+| `backend/scripts/fetch_secedgar.py` | pulls ~130 financial docs |
+| `test_data/medical_real/`, `test_data/legal_real/`, `test_data/financial_real/` | fetched corpora + manifests |
+| `test_data/query_sets/*.json` | authored query sets with domain/expected-source labels |
+
+---
+
+## What Phase 2 does NOT cover (handled later)
+
+- Running `baseline_runner.py`, `benchmark_runner.py`, `guardrail_analysis.py`, `statistical_tests.py` against this real data to produce actual paper numbers, that's Phase 3.
+- Rewriting Tables I-V and Figures 2-4 in the paper, and rehearsing the live demo, that's Phase 4.
+
+Once Phase 1 (architecture) and Phase 2 (data) are both done, the evaluation scripts already in `backend/evaluation/` should be able to run against real inputs with minimal further changes, since Phase 1 kept their function signatures stable.

README.md

@@ -1 +1,418 @@
-# Mexar Space
+---
+title: Mexar
+emoji: 🧠
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+pinned: false
+license: mit
+---
+# MEXAR Ultimate 🧠
+
+**Multimodal Explainable AI Reasoning Assistant**
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![React 18](https://img.shields.io/badge/react-18-61dafb.svg)](https://reactjs.org/)
+[![FastAPI](https://img.shields.io/badge/fastapi-0.109-009688.svg)](https://fastapi.tiangolo.com/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+[![Deployed](https://img.shields.io/badge/status-live-brightgreen.svg)](https://mexar.vercel.app)
+
+> Create domain-specific intelligent agents from your data with transparent, explainable AI responses using RAG (Retrieval-Augmented Generation) with source attribution and faithfulness scoring.
+
+**🚀 Live Demo**: [https://mexar.vercel.app](https://mexar.vercel.app)  
+**📡 Backend API**: [https://devrajsinh2012-mexar.hf.space](https://devrajsinh2012-mexar.hf.space)
+
+---
+
+## ✨ Key Features
+
+| Feature | Description |
+|---------|-------------|
+| 🔍 **Hybrid Search** | Combines semantic (vector) + keyword search with RRF fusion for optimal retrieval |
+| 🎯 **Cross-Encoder Reranking** | Improves retrieval precision using sentence-transformers |
+| 📊 **Source Attribution** | Inline citations `[1]`, `[2]` linking answers to source data |
+| ✅ **Faithfulness Scoring** | Measures how well answers are grounded in retrieved context |
+| 🗣️ **Multimodal Input** | Audio (Whisper), Images (Vision), Video support |
+| 🔐 **Domain Guardrails** | Prevents hallucinations outside knowledge base |
+| 🔊 **Text-to-Speech** | ElevenLabs + Web Speech API integration |
+| 📁 **5 File Types** | CSV, PDF, DOCX, JSON, TXT |
+
+---
+
+## 🏗️ Architecture
+
+```
+┌────────────────────────────────────────────────────────────────────────────┐
+│                         MEXAR Ultimate Stack                               │
+├────────────────────────────────────────────────────────────────────────────┤
+│                                                                            │
+│            ┌────────────────────────────────────────────┐                  │
+│            │        React Frontend (Vercel)             │                  │
+│            └────────────────────────────────────────────┘                  │
+│                               │                                            │
+│                               ▼                                            │
+│            ┌────────────────────────────────────────────┐                  │
+│            │     FastAPI Backend (Hugging Face Spaces)  │                  │
+│            └────────────────────────────────────────────┘                  │
+│                               │                                            │
+│                               ▼                                            │
+│   ┌────────────────────────────────────────────────────────────────────┐   │
+│   │                 Core Intelligence Layer                            │   │
+│   │                                                                    │   │
+│   │   🔄 Data Validator        (PDF / DOCX / CSV / TXT / JSON)         │   │
+│   │   🤖 Prompt Analyzer       (LLM-based Intent Parsing)              │   │
+│   │   📦 Knowledge Compiler    (FastEmbed + Chunking)                  │   │
+│   │   🧠 Reasoning Engine                                              │   │
+│   │      ├─ Hybrid Search       (Dense + Sparse)                       │   │
+│   │      ├─ Cross-Encoder       (Re-ranking)                           │   │
+│   │      ├─ Source Attribution  (Citation Tracking)                    │   │
+│   │      └─ Faithfulness Scorer (Hallucination Control)                │   │
+│   │                                                                    │   │
+│   └─────────────────────────────────────��──────────────────────────────┘   │
+│                               │                                            │
+│                               ▼                                            │
+│   ┌────────────────────────────────────────────────────────────────────┐   │
+│   │               External Services Layer                              │   │
+│   │                                                                    │   │
+│   │   🗄 Supabase   → PostgreSQL + pgvector + File Storage             │   │
+│   │   ⚡ Groq API   → LLM Inference + Whisper + Vision                │   │
+│   │   🔊 ElevenLabs → Text-to-Speech                                  │   │
+│   │                                                                    │   │
+│   └────────────────────────────────────────────────────────────────────┘   │
+│                                                                            │
+└────────────────────────────────────────────────────────────────────────────┘
+
+```
+
+---
+
+## 🚀 Quick Start
+
+### Prerequisites
+
+- **Python 3.9+** with pip
+- **Node.js 18+** with npm
+- **PostgreSQL** with `pgvector` extension (or use Supabase)
+- **Groq API Key** - Get free at [console.groq.com](https://console.groq.com)
+
+### Local Development
+
+#### 1. Backend Setup
+
+```bash
+cd backend
+
+# Create virtual environment
+python -m venv venv
+
+# Activate (Windows)
+.\venv\Scripts\activate
+# Activate (macOS/Linux)
+source venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Configure environment
+cp .env.example .env
+# Edit .env and add your API keys
+
+# Run server
+python -m uvicorn main:app --reload --host 0.0.0.0 --port 8000
+```
+
+**Backend will run at**: [http://localhost:8000](http://localhost:8000)
+
+#### 2. Frontend Setup
+
+```bash
+cd frontend
+
+# Install dependencies
+npm install
+
+# Start development server
+npm start
+```
+
+**Frontend will run at**: [http://localhost:3000](http://localhost:3000)
+
+---
+
+## 📈 Evaluation Workflows
+
+The scripts in `backend/evaluation` support baseline comparison, guardrail checks, benchmark runs, and ablation studies.
+
+Run from project root:
+
+```bash
+cd backend
+
+# Baseline comparison: MEXAR vs CRAG vs RAPTOR
+python evaluation/baseline_runner.py
+
+# Backbone comparison (restores original backbone after completion)
+python evaluation/backbone_comparison.py
+
+# Guardrail boundary query analysis
+python evaluation/guardrail_analysis.py
+
+# Benchmark dataset run (all rows by default) + save report
+python evaluation/benchmark_runner.py --dataset-path ../test_data/medqa_sample.json --agent-name medical_agent --output evaluation_outputs/medqa_report.json
+
+# Quick benchmark smoke test
+python evaluation/benchmark_runner.py --dataset-path ../test_data/medqa_sample.json --agent-name medical_agent --max-samples 25
+
+# McNemar significance helper
+python evaluation/statistical_tests.py
+```
+
+Notes:
+- Faithfulness values are read from `explainability.confidence_breakdown.faithfulness` when available.
+- Benchmark reports include per-query status and aggregate summary metrics.
+
+---
+
+## 📁 Project Structure
+
+```
+mexar_ultimate/
+├── backend/                    # FastAPI Backend
+│   ├── api/                   # REST API endpoints
+│   │   ├── auth.py           # Authentication (JWT)
+│   │   ├── agents.py         # Agent CRUD
+│   │   ├── chat.py           # Chat + multimodal
+│   │   ├── compile.py        # Knowledge compilation
+│   │   └── websocket.py      # Real-time updates
+│   ├── core/                  # Core configuration
+│   │   ├── config.py         # Settings
+│   │   ├── database.py       # SQLAlchemy setup
+│   │   └── security.py       # JWT handling
+│   ├── models/                # Database models
+│   │   ├── user.py           # User model
+│   │   ├── agent.py          # Agent + CompilationJob
+│   │   ├── chunk.py          # DocumentChunk (pgvector)
+│   │   └── conversation.py   # Chat history
+│   ├── modules/               # Core AI modules
+│   │   ├── data_validator.py # File parsing
+│   │   ├── prompt_analyzer.py # Domain extraction
+│   │   ├── knowledge_compiler.py # Vector embeddings
+│   │   ├── reasoning_engine.py # RAG pipeline
+│   │   └── explainability.py # UI formatting
+│   ├── utils/                 # Utilities
+│   │   ├── groq_client.py    # Groq API wrapper
+│   │   ├── hybrid_search.py  # RRF search fusion
+│   │   ├── reranker.py       # Cross-encoder
+│   │   ├── faithfulness.py   # Claim verification
+│   │   └── source_attribution.py # Citation extraction
+│   ├── main.py               # FastAPI entry point
+│   └── requirements.txt      # Python dependencies
+│
+├── frontend/                  # React Frontend
+│   ├── src/
+│   │   ├── pages/            # React pages
+│   │   │   ├── Landing.jsx   # Home page
+│   │   │   ├── Login.jsx     # Authentication
+│   │   │   ├── Dashboard.jsx # User dashboard
+│   │   │   ├── AgentCreation.jsx # Create agent
+│   │   │   ├── CompilationProgress.jsx # Build progress
+│   │   │   └── Chat.jsx      # Chat interface
+│   │   ├── components/       # Reusable UI
+│   │   ├── contexts/         # React contexts
+│   │   ├── api/              # API client
+│   │   └── App.jsx           # Main component
+│   ├── package.json          # Node dependencies
+│   └── vercel.json           # Vercel config
+│
+├── Dockerfile                 # Docker config for HF Spaces
+└── README.md                  # This file
+```
+---
+### 📸 Screenshot
+<img width="1863" height="918" alt="image" src="https://github.com/user-attachments/assets/75ff2c32-4c90-4e65-ab5f-10c94bad2e65" />
+<img width="330" height="330" alt="image" src="https://github.com/user-attachments/assets/2d9846c9-b002-4f05-b632-6dc1c8c58ad1" />
+<img width="330" height="330" alt="image" src="https://github.com/user-attachments/assets/810de78d-4725-494e-94b5-e8aa71b8b7fd" />
+<img width="330" height="330" alt="image" src="https://github.com/user-attachments/assets/155ea162-d8b9-4b45-acf9-42afc47c8498" />
+<img width="330" height="330" alt="image" src="https://github.com/user-attachments/assets/cadf46a8-c679-4a76-a58d-a956f9cb9cc1" />
+<img width="330" height="330" alt="image" src="https://github.com/user-attachments/assets/598b7925-8001-44a7-ae34-6fddb85b7531" />
+<img width="330" height="330" alt="image" src="https://github.com/user-attachments/assets/e9e99b9d-05b0-4ae5-9990-ed9c9cc124c6" />
+
+---
+
+## 🌐 Deployment
+
+### Current Deployment (Free Tier)
+
+- **Frontend**: Vercel - [https://mexar.vercel.app](https://mexar.vercel.app)
+- **Backend**: Hugging Face Spaces - [https://devrajsinh2012-mexar.hf.space](https://devrajsinh2012-mexar.hf.space)
+- **Database**: Supabase (PostgreSQL with pgvector)
+- **Storage**: Supabase Storage
+- **Total Cost**: $0/month
+
+### Deploy Your Own Instance
+
+#### Deploy Backend to Hugging Face Spaces
+
+1. Fork this repository
+2. Create a new Space at [huggingface.co/new-space](https://huggingface.co/new-space)
+3. Select **Docker** as SDK
+4. Connect your GitHub repository
+5. Add Repository Secrets:
+   - `GROQ_API_KEY`
+   - `DATABASE_URL`
+   - `SUPABASE_URL`
+   - `SUPABASE_KEY`
+   - `SECRET_KEY`
+   - `FRONTEND_URL`
+
+#### Deploy Frontend to Vercel
+
+1. Import repository at [vercel.com](https://vercel.com)
+2. Set **Root Directory** to `frontend`
+3. Add Environment Variable:
+   - `REACT_APP_API_URL` = Your HF Spaces URL
+
+---
+
+## 🔧 Environment Variables
+
+### Backend (`backend/.env`)
+
+```env
+# Required: Get from console.groq.com
+GROQ_API_KEY=your_groq_api_key_here
+
+# Supabase Database
+DATABASE_URL=postgresql://user:password@host:5432/database
+
+# JWT Security
+SECRET_KEY=generate-a-secure-random-key
+
+# Supabase Storage
+SUPABASE_URL=https://your-project.supabase.co
+SUPABASE_KEY=your_supabase_service_role_key
+
+# Optional: ElevenLabs TTS
+ELEVENLABS_API_KEY=your_elevenlabs_api_key_here
+
+# Frontend URL for CORS
+FRONTEND_URL=https://mexar.vercel.app
+```
+
+### Frontend (`frontend/.env`)
+
+```env
+# Backend API URL
+REACT_APP_API_URL=https://your-backend.hf.space
+```
+
+---
+
+## 🔍 API Documentation
+
+Once the backend is running, interactive API docs are available at:
+
+- **Swagger UI**: `http://localhost:8000/docs`
+- **ReDoc**: `http://localhost:8000/redoc`
+
+### Key Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| POST | `/api/auth/register` | Register new user |
+| POST | `/api/auth/login` | Login (returns JWT) |
+| GET | `/api/agents/` | List all agents |
+| POST | `/api/compile/` | Start agent compilation |
+| GET | `/api/compile/{name}/status` | Check compilation status |
+| POST | `/api/chat/` | Send message to agent |
+| POST | `/api/chat/multimodal` | Send with audio/image |
+
+---
+
+## 🧪 Technologies
+
+### Backend
+- **FastAPI** - Modern async Python web framework
+- **SQLAlchemy** - ORM for PostgreSQL
+- **pgvector** - Vector similarity search
+- **FastEmbed** - Local embedding generation (BAAI/bge-small-en-v1.5)
+- **sentence-transformers** - Cross-encoder reranking
+- **Groq API** - LLM (Llama 3.1/3.3), Whisper (audio), Vision (images)
+
+### Frontend
+- **React 18** - UI framework
+- **Material-UI (MUI)** - Component library
+- **React Router** - Navigation
+- **Axios** - HTTP client
+
+### External Services
+- **Supabase** - Managed PostgreSQL + Storage
+- **Groq** - Fast AI inference (LPU architecture)
+- **ElevenLabs** - Text-to-Speech (optional)
+
+---
+
+## 📊 How It Works
+
+### 1. Agent Creation Flow
+```
+User uploads files → DataValidator parses content
+                  → PromptAnalyzer extracts domain & keywords
+                  → KnowledgeCompiler creates embeddings
+                  → Stored in pgvector database
+```
+
+### 2. Query Processing Flow
+```
+User query → Domain Guardrail check
+          → Hybrid Search (semantic + keyword)
+          → Cross-Encoder Reranking (top 5 results)
+          → LLM Generation with retrieved context
+          → Source Attribution (extract citations)
+          → Faithfulness Scoring (verify grounding)
+          → Explainability Formatting
+```
+
+### 3. Confidence Calculation
+Confidence score is calculated from:
+- **Retrieval Quality** (35%) - Relevance of retrieved chunks
+- **Rerank Score** (30%) - Cross-encoder confidence
+- **Faithfulness** (25%) - Answer grounding in context
+- **Base Floor** (10%) - For in-domain queries
+
+---
+
+## ⚠️ Known Limitations (Free Tier)
+
+1. **Cold Start Delay**: First request after 15 min idle takes 45-90 seconds
+2. **Model Download**: Initial startup takes 3-5 minutes (FastEmbed caching)
+3. **Groq Rate Limits**: 30 requests/min, 14,400/day (free tier)
+4. **Concurrent Users**: 1-2 recommended on free tier (2GB RAM limit)
+5. **Ephemeral Storage**: HF Spaces `/tmp` data lost on restart (Supabase used for persistence)
+
+**Production Migration**: Upgrade to paid tiers for ~$54/month (persistent instances, higher limits)
+
+---
+
+## 🙏 Acknowledgments
+
+- [Groq](https://groq.com) - Fast AI inference with LPU technology
+- [Supabase](https://supabase.com) - PostgreSQL + Storage platform
+- [FastEmbed](https://github.com/qdrant/fastembed) - Lightweight embeddings library
+- [sentence-transformers](https://www.sbert.net) - Reranking models
+- [Hugging Face](https://huggingface.co) - Free ML model hosting
+
+---
+
+## 🤝 Contribution
+
+This project is built by Devrajsinh Gohil and Jay Nasit under the guidance of Prof OM Prakash Suthar.
+
+**Built with ❤️ using modern AI technologies**
+
+---
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+

backend/.env.example

@@ -40,3 +40,22 @@ ELEVENLABS_API_KEY=your_elevenlabs_api_key_here
 # ===========================================
 # For development only, production uses Supabase Storage
 STORAGE_PATH=./data/storage
+
+# ===========================================
+# PHASE 2: Dataset Collection Scripts
+# ===========================================
+# Required for backend/scripts/fetch_courtlistener.py
+# Free account + token at: https://www.courtlistener.com/sign-in/
+COURTLISTENER_TOKEN=your_courtlistener_api_token_here
+
+# Required for backend/scripts/fetch_pubmed.py
+# NCBI usage policy requires a valid email address on every request.
+NCBI_EMAIL=your_email@example.com
+
+# Optional: NCBI API key raises rate limit from 3 req/sec → 10 req/sec.
+# Register at: https://www.ncbi.nlm.nih.gov/account/
+NCBI_API_KEY=
+
+# Required for backend/scripts/fetch_secedgar.py
+# SEC fair access policy: format must be "FirstName LastName email@example.com"
+SEC_USER_AGENT=YourName your@email.com

backend/Dockerfile

@@ -13,6 +13,9 @@ RUN apt-get update && apt-get install -y \
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
 
+# Download spaCy model for domain guardrail NER (Section III-A)
+RUN python -m spacy download en_core_web_sm
+
 # Copy application code
 COPY . .
 

backend/core/config.py

@@ -15,7 +15,8 @@ class Config:
     
     # AI Services
     GROQ_API_KEY = os.getenv("GROQ_API_KEY")
-    LLM_BACKBONE = os.getenv("LLM_BACKBONE", "llama3")  # Options: llama3, mixtral, gemma
+    LLM_BACKBONE = os.getenv("LLM_BACKBONE", "openai/gpt-oss-120b")  # Options: openai/gpt-oss-120b, llama3, mixtral, gemma
+    GROQ_MODEL = os.getenv("GROQ_MODEL", os.getenv("LLM_MODEL", "openai/gpt-oss-120b"))
     
     # Storage
     STORAGE_PATH = os.getenv("STORAGE_PATH", "./data/storage")

backend/evaluation/metrics.py

@@ -8,22 +8,35 @@ from typing import Any, Dict, Optional
 
 sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from utils.faithfulness import FaithfulnessScorer, BartNLIScorer, FActScoreCompat
+from utils.faithfulness import FaithfulnessScorer, DebertaNLIScorer, FActScoreCompat
 
 
 class MetricsRunner:
     def __init__(self):
         self.faith_scorer = FaithfulnessScorer()
-        self.bart_nli = BartNLIScorer()
+        # Primary NLI scorer — DeBERTa-v3 (Section III-C, Eq. 6/7)
+        self.deberta_nli = DebertaNLIScorer()
         self.factscore = FActScoreCompat()
 
-    def evaluate_all(self, answer: str, context: str) -> Dict[str, float]:
-        faith_res = self.faith_scorer.score(answer, context)
-        bart_res = self.bart_nli.score(answer, context)
-        fact_res = self.factscore.score(answer, context)
+    def evaluate_all(self, answer: str, context_documents: List[str]) -> Dict[str, float]:
+        """
+        Evaluate faithfulness using all available scorers.
+
+        Args:
+            answer: LLM-generated answer.
+            context_documents: List of individual retrieved chunk texts.
+                               DeBERTa scorer requires individual docs (Eq. 6).
+                               FaithfulnessScorer receives them joined as a single string.
+        """
+        # FaithfulnessScorer (LLM-judge baseline) still takes a single concatenated string
+        context_str = "\n\n---\n\n".join(context_documents)
+        faith_res = self.faith_scorer.score(answer, context_str)
+        # DeBERTa NLI (primary) takes individual chunk texts
+        deberta_res = self.deberta_nli.score(answer, context_documents)
+        fact_res = self.factscore.score(answer, context_str)
         return {
-            "faithfulness": faith_res.score,
-            "bart_nli": bart_res.score,
+            "faithfulness_llm_judge": faith_res.score,
+            "deberta_nli": deberta_res.score,
             "factscore": fact_res.score,
         }
 

backend/migrations/README.md

@@ -34,14 +34,17 @@ Reciprocal Rank Fusion (RRF) algorithm.
    ```
    - Should return one row
 
-## Alternative: Run from Command Line (Optional)
+## Recommended: Automated Python Migration Runner
 
-If you have `psql` installed:
+Run all migrations automatically using your `DATABASE_URL` configured in `backend/.env`:
 
 ```bash
-psql "postgresql://postgres.xmfcidiwovxuihrkfzps:Yogiji@20122004@aws-1-ap-south-1.pooler.supabase.com:5432/postgres" -f migrations/hybrid_search_function.sql
+python scripts/run_migrations.py
 ```
 
+## Alternative: Run manually via Supabase Dashboard
+
+
 ## What Gets Created
 
 - **Function**: `hybrid_search(vector, text, integer, integer)` 

backend/migrations/add_domain_signature_columns.sql

@@ -0,0 +1,17 @@
+-- MEXAR Phase 1 - Architecture Fix 1.1
+-- Adds domain_signature_weights and domain_entities columns to agents table.
+--
+-- domain_signature_weights: JSONB dict of {term: tfidf_weight} from TF-IDF lexical signature (Eq. 1)
+-- domain_entities:          JSONB list of NER entity strings extracted from corpus (Section III-A)
+--
+-- Apply via Supabase SQL Editor (copy-paste and Run).
+-- See backend/migrations/README.md for instructions.
+
+ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_signature_weights JSONB;
+ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_entities JSONB;
+
+COMMENT ON COLUMN agents.domain_signature_weights IS
+  'TF-IDF lexical weights dict {term: weight} for domain signature (Section III-A, Eq. 1)';
+
+COMMENT ON COLUMN agents.domain_entities IS
+  'NER-extracted domain entities list for domain signature (Section III-A)';

backend/migrations/switch_to_hnsw_index.sql

@@ -0,0 +1,27 @@
+-- MEXAR Phase 1 - Architecture Fix 1.2
+-- Switches the document_chunks embedding index from IVFFlat to HNSW.
+-- Paper Section III-B explicitly claims HNSW indexing.
+--
+-- Pre-requisite: pgvector >= 0.5.0 required for HNSW.
+-- Verify with: SELECT extversion FROM pg_extension WHERE extname = 'vector';
+-- If below 0.5.0, run: ALTER EXTENSION vector UPDATE;
+--
+-- Apply via Supabase SQL Editor (copy-paste and Run).
+-- See backend/migrations/README.md for instructions.
+
+-- Drop the old IVFFlat index created in hybrid_search_function.sql
+DROP INDEX IF EXISTS idx_document_chunks_embedding;
+
+-- Create HNSW index to match paper Section III-B
+-- m=16: number of bi-directional links per node (connectivity)
+-- ef_construction=64: size of dynamic candidate list during index construction (recall/speed trade-off)
+CREATE INDEX IF NOT EXISTS idx_document_chunks_embedding_hnsw
+ON document_chunks USING hnsw (embedding vector_cosine_ops)
+WITH (m = 16, ef_construction = 64);
+
+COMMENT ON INDEX idx_document_chunks_embedding_hnsw IS
+  'HNSW index for cosine similarity search, m=16, ef_construction=64 (Section III-B)';
+
+-- Verification query — run after applying to confirm:
+-- SELECT indexdef FROM pg_indexes WHERE tablename = 'document_chunks';
+-- Should show 'hnsw', not 'ivfflat'.

backend/models/agent.py

@@ -18,6 +18,9 @@ class Agent(Base):
     # All metadata stored in Supabase (no filesystem)
     domain_keywords = Column(JSONB, nullable=True)
     domain_signature = Column(JSONB, nullable=True)
+    # Phase 1 fix 1.1: TF-IDF weights and NER entities stored separately for inspection
+    domain_signature_weights = Column(JSONB, nullable=True)  # {term: tfidf_weight} (Eq. 1)
+    domain_entities = Column(JSONB, nullable=True)            # NER entity list (Section III-A)
     prompt_analysis = Column(JSONB, nullable=True)
     knowledge_graph_json = Column(JSONB, nullable=True)
     compilation_stats = Column(JSONB, nullable=True)

backend/modules/knowledge_compiler.py

@@ -10,6 +10,7 @@ from typing import Dict, List, Any, Optional
 from pathlib import Path
 
 from utils.groq_client import get_groq_client, GroqClient
+from utils.domain_signature import build_domain_signature
 from fastembed import TextEmbedding
 from core.database import SessionLocal
 from models.agent import Agent
@@ -102,7 +103,9 @@ class KnowledgeCompiler:
             self._save_agent(
                 agent_name=agent_name,
                 text_context=text_context,
-                domain_signature=domain_signature,
+                domain_signature=domain_signature["combined"],
+                domain_signature_weights=domain_signature["lexical_weights"],
+                domain_entities=domain_signature["entities"],
                 system_prompt=system_prompt,
                 prompt_analysis=prompt_analysis,
                 stats=stats
@@ -116,7 +119,7 @@ class KnowledgeCompiler:
             self._update_progress("complete", 100, "Compilation complete!")
             
             return {
-                "domain_signature": domain_signature,
+                "domain_signature": domain_signature["combined"],
                 "stats": stats,
                 "agent_path": str(self.data_dir / agent_name)
             }
@@ -200,33 +203,47 @@ class KnowledgeCompiler:
         self,
         parsed_data: List[Dict[str, Any]],
         prompt_analysis: Dict[str, Any]
-    ) -> List[str]:
+    ) -> Dict:
         """
-        Extract domain signature keywords for guardrail checking.
+        Build domain signature using TF-IDF (Eq. 1) + NER entities (Section III-A).
+
+        Extracts per-file raw text to preserve per-document granularity needed for
+        TF-IDF's document-frequency term to be meaningful. Returns the full signature
+        dict from build_domain_signature() including 'combined', 'lexical_weights',
+        and 'entities' keys.
         """
-        # Start with analyzed keywords (highest priority)
-        domain_keywords = prompt_analysis.get("domain_keywords", [])
-        signature = list(domain_keywords)
-        
-        # Add domain and sub-domains
-        domain = prompt_analysis.get("domain", "")
-        if domain and domain not in signature:
-            signature.insert(0, domain)
-        
-        for sub_domain in prompt_analysis.get("sub_domains", []):
-            if sub_domain and sub_domain.lower() not in [s.lower() for s in signature]:
-                signature.append(sub_domain)
-        
-        # Extract column headers from structured data
+        # Collect per-file text documents (NOT a single concatenated string)
+        documents: List[str] = []
         for file_data in parsed_data:
-            if file_data.get("data") and isinstance(file_data["data"], list):
-                if file_data["data"] and isinstance(file_data["data"][0], dict):
-                    for key in file_data["data"][0].keys():
-                        clean_key = key.lower().strip().replace("_", " ")
-                        if clean_key and clean_key not in [s.lower() for s in signature]:
-                            signature.append(clean_key)
-        
-        return signature[:80]  # Limit for efficiency
+            text = (
+                file_data.get("text") or
+                file_data.get("content") or
+                ""
+            )
+            # For structured data (CSV/JSON), build a text representation per file
+            if not text and file_data.get("data") and isinstance(file_data["data"], list):
+                rows = file_data["data"]
+                text = " ".join(
+                    " ".join(str(v) for v in row.values() if v is not None)
+                    for row in rows
+                    if isinstance(row, dict)
+                )
+            if text and text.strip():
+                documents.append(text)
+
+        # Build TF-IDF + NER domain signature
+        signature = build_domain_signature(
+            documents=documents,
+            tau_tf=0.0,
+            tau_ent=2,
+            top_n_lexical=100,
+        )
+
+        logger.info(
+            f"Domain signature: {len(signature['combined'])} combined terms "
+            f"({len(signature['lexical'])} lexical + {len(signature['entities'])} NER entities)"
+        )
+        return signature
     
     def _calculate_stats(
         self,
@@ -249,11 +266,13 @@ class KnowledgeCompiler:
         agent_name: str,
         text_context: str,
         domain_signature: List[str],
+        domain_signature_weights: Dict,
+        domain_entities: List[str],
         system_prompt: str,
         prompt_analysis: Dict[str, Any],
         stats: Dict[str, Any]
     ):
-        """Save agent artifacts to filesystem."""
+        """Save agent artifacts to filesystem and Supabase DB."""
         agent_dir = self.data_dir / agent_name
         agent_dir.mkdir(parents=True, exist_ok=True)
         
@@ -267,12 +286,30 @@ class KnowledgeCompiler:
             "system_prompt": system_prompt,
             "prompt_analysis": prompt_analysis,
             "domain_signature": domain_signature,
+            "domain_signature_weights": domain_signature_weights,
+            "domain_entities": domain_entities,
             "stats": stats,
             "created_at": self._get_timestamp()
         }
         with open(agent_dir / "metadata.json", "w", encoding="utf-8") as f:
             json.dump(metadata, f, indent=2, ensure_ascii=False)
         
+        # Persist new domain_signature_weights and domain_entities to Supabase DB
+        try:
+            with SessionLocal() as db:
+                agent = db.query(Agent).filter(Agent.name == agent_name).first()
+                if agent:
+                    agent.domain_signature = domain_signature
+                    agent.domain_signature_weights = domain_signature_weights
+                    agent.domain_entities = domain_entities
+                    db.commit()
+                    logger.info(
+                        f"Persisted domain_signature ({len(domain_signature)} terms), "
+                        f"domain_signature_weights, and domain_entities for agent '{agent_name}'."
+                    )
+        except Exception as e:
+            logger.warning(f"Could not persist domain signature to DB: {e}")
+        
         logger.info(f"Agent saved to: {agent_dir}")
     
     def _get_timestamp(self) -> str:

backend/modules/reasoning_engine.py

@@ -9,14 +9,13 @@ import logging
 from typing import Dict, List, Any, Optional, Tuple
 from pathlib import Path
 import networkx as nx
-from difflib import SequenceMatcher
 import numpy as np
 
 from utils.groq_client import get_groq_client, GroqClient
 from utils.hybrid_search import HybridSearcher
 from utils.reranker import Reranker
 from utils.source_attribution import SourceAttributor
-from utils.faithfulness import FaithfulnessScorer, BartNLIScorer
+from utils.faithfulness import FaithfulnessScorer, DebertaNLIScorer
 from fastembed import TextEmbedding
 from core.database import SessionLocal
 from models.agent import Agent
@@ -68,8 +67,10 @@ class ReasoningEngine:
         self.searcher = HybridSearcher(self.embedding_model) if self.embedding_model else None
         self.reranker = Reranker()
         self.attributor = SourceAttributor(self.embedding_model)
+        # Primary faithfulness: DeBERTa-v3 NLI (Section III-C, Eq. 6/7)
+        self.deberta_nli_scorer = DebertaNLIScorer()
+        # Baseline faithfulness: LLM-as-judge (kept for paper Table I/III comparison)
         self.faithfulness_scorer = FaithfulnessScorer()
-        self.bart_nli_scorer = BartNLIScorer()
         
         # Cache for loaded agents
         self._agent_cache: Dict[str, Dict] = {}
@@ -154,10 +155,15 @@ class ReasoningEngine:
         attribution = self.attributor.attribute(answer, top_chunks, chunk_embeddings)
         
         # Step 6: Faithfulness Scoring
-        faithfulness_result = self.faithfulness_scorer.score(answer, context)
+        # PRIMARY: DeBERTa-v3 NLI per-document max entailment (Eq. 6/7, Section III-C)
+        # Passes individual chunk texts (NOT concatenated) so per-doc max is meaningful.
+        faithfulness_result = self.deberta_nli_scorer.score(
+            answer, [c.content for c in top_chunks]
+        )
         
-        # Independent NLI Baseline Scoring (for reviewer feedback)
-        bart_nli_result = self.bart_nli_scorer.score(answer, context)
+        # BASELINE (LLM-as-judge): kept for paper Table I/III comparison
+        # Stored under a clearly-labelled key, NOT used as the headline faithfulness number.
+        llm_faithfulness_result = self.faithfulness_scorer.score(answer, context)
         
         # Step 7: Calculate Confidence
         top_similarity = rrf_scores[0] if rrf_scores else 0
@@ -166,7 +172,7 @@ class ReasoningEngine:
         confidence = self._calculate_confidence(
             top_similarity=top_similarity,
             rerank_score=top_rerank,
-            faithfulness=faithfulness_result.score
+            faithfulness=faithfulness_result.score  # DeBERTa is primary signal
         )
         
         # Step 8: Build Explainability
@@ -178,7 +184,7 @@ class ReasoningEngine:
             rerank_scores=rerank_scores,
             attribution=attribution,
             faithfulness=faithfulness_result,
-            bart_nli_result=bart_nli_result,
+            llm_faithfulness_result=llm_faithfulness_result,
             confidence=confidence,
             domain_score=domain_score
         )
@@ -228,65 +234,47 @@ class ReasoningEngine:
         domain_signature: List[str],
         prompt_analysis: Dict[str, Any]
     ) -> Tuple[bool, float]:
-        """Check if query matches the domain."""
-        query_lower = query.lower()
-        query_words = set(query_lower.split())
-        
-        matches = 0
-        bonus_matches = 0
-        
-        # Check domain match
-        domain = prompt_analysis.get("domain", "")
-        if domain.lower() in query_lower:
-            bonus_matches += 3
-        
-        # Check sub-domains
-        for sub_domain in prompt_analysis.get("sub_domains", []):
-            if sub_domain.lower() in query_lower:
-                bonus_matches += 2
-        
-        # Check domain keywords
-        for keyword in prompt_analysis.get("domain_keywords", []):
-            if keyword.lower() in query_lower:
-                bonus_matches += 1.5
-        
-        # Check signature keywords with fuzzy matching
-        signature_lower = [kw.lower() for kw in (domain_signature or [])]
-        
-        for word in query_words:
-            if len(word) < 3:
-                continue
-            for kw in signature_lower[:100]:
-                if self._fuzzy_match(word, kw) > 0.75:
-                    matches += 1
-                    break
-                if word in kw or kw in word:
-                    matches += 0.5
-                    break
-        
-        # Calculate score
-        max_possible = max(1, min(len(query_words), 10))
-        base_score = matches / max_possible
-        bonus_score = min(0.5, bonus_matches * 0.1)
-        score = min(1.0, base_score + bonus_score)
-        
-        if bonus_matches >= 1:
-            score = max(score, 0.2)
-        
+        """
+        Eq. 2: sim(q, Sigma) = |Phi(q) ∩ Sigma| / |Phi(q)|
+
+        Replaces the old fuzzy-string heuristic with real Jaccard similarity
+        over spaCy-lemmatized content tokens (Section III-A).
+
+        Phi(q): lemmatized content tokens from query, excluding stopwords/punctuation
+                and tokens shorter than 3 characters.
+        Sigma:  domain_signature (combined lexical + NER terms from build_domain_signature)
+        """
+        from utils.domain_signature import get_nlp
+        nlp = get_nlp()
+
+        query_doc = nlp(query.lower())
+
+        # Phi(q): lemmatized meaningful tokens
+        phi_q = {
+            tok.lemma_
+            for tok in query_doc
+            if not tok.is_stop and not tok.is_punct and len(tok.text) > 2
+        }
+
+        sigma = set(s.lower() for s in (domain_signature or []))
+
+        if not phi_q:
+            logger.info("Guardrail: phi_q is empty — query has no content tokens, rejecting.")
+            return False, 0.0
+
+        intersection = phi_q & sigma
+        score = len(intersection) / len(phi_q)
+
         is_in_domain = score >= self.DOMAIN_SIMILARITY_THRESHOLD
-        
-        # Analyze guardrail false-accept rate: Log boundary queries (close to threshold)
-        if 0.05 <= score < 0.15:
-            logger.info(f"GUARDRAIL_BOUNDARY_ACCEPT: score={score:.2f}, query='{query}' - Check for false positive")
-        
-        logger.info(f"Guardrail: score={score:.2f}, matches={matches}, bonus={bonus_matches}, in_domain={is_in_domain}")
-        
+
+        logger.info(
+            f"Guardrail (Eq. 2): sim={score:.3f}, |Phi(q)|={len(phi_q)}, "
+            f"|Sigma|={len(sigma)}, |intersection|={len(intersection)}, "
+            f"in_domain={is_in_domain}"
+        )
+
         return is_in_domain, score
     
-    def _fuzzy_match(self, s1: str, s2: str) -> float:
-        """Calculate fuzzy match ratio."""
-        return SequenceMatcher(None, s1, s2).ratio()
-    
     def _generate_answer(
         self,
         query: str,
@@ -378,7 +366,7 @@ IMPORTANT INSTRUCTIONS:
         rerank_scores: List[float],
         attribution,
         faithfulness,
-        bart_nli_result,
+        llm_faithfulness_result,
         confidence: float,
         domain_score: float
     ) -> Dict[str, Any]:
@@ -401,8 +389,10 @@ IMPORTANT INSTRUCTIONS:
                 "domain_relevance": f"{domain_score*100:.0f}%",
                 "retrieval_quality": f"{rrf_scores[0]*100:.1f}%" if rrf_scores else "N/A",
                 "rerank_score": f"{rerank_scores[0]:.2f}" if rerank_scores else "N/A",
+                # Primary faithfulness: DeBERTa-v3 NLI (Section III-C)
                 "faithfulness": f"{faithfulness.score*100:.0f}%",
-                "bart_nli_score": f"{bart_nli_result.score*100:.0f}%" if bart_nli_result else "N/A",
+                # Baseline: LLM-as-judge (for paper Table I/III comparison only)
+                "llm_judge_baseline_score": f"{llm_faithfulness_result.score*100:.0f}%" if llm_faithfulness_result else "N/A",
                 "claims_supported": f"{faithfulness.supported_claims}/{faithfulness.total_claims}"
             },
             "unsupported_claims": faithfulness.unsupported_claims[:3],

backend/requirements.txt

@@ -1,5 +1,9 @@
 # MEXAR Phase 1 - Backend Dependencies
 
+# Domain Guardrail (Section III-A: TF-IDF + NER + Jaccard)
+scikit-learn>=1.4.0
+spacy>=3.7.0
+
 # Web Framework
 fastapi==0.109.0
 uvicorn[standard]==0.27.0

backend/scripts/__init__.py

@@ -0,0 +1,5 @@
+"""
+MEXAR Dataset Collection Scripts — Phase 2
+Fetches real-world corpora from PubMed Central, CourtListener, and SEC EDGAR.
+Run these scripts locally (not in a network-restricted sandbox).
+"""

backend/scripts/fetch_courtlistener.py

@@ -0,0 +1,294 @@
+"""
+MEXAR - CourtListener fetcher for the legal corpus.
+
+Fetches ~150 judicial opinions from the CourtListener REST API v4 across three
+legal subdomains (contract law, intellectual property, corporate governance),
+saves opinion text as plain .txt files, and writes a manifest.json.
+
+Usage:
+    python backend/scripts/fetch_courtlistener.py
+
+Requirements:
+    pip install requests
+
+Environment:
+    COURTLISTENER_TOKEN — API token from https://www.courtlistener.com/sign-in/
+                          (free account required)
+
+Run this locally (not in a network-restricted sandbox).
+
+Notes on CourtListener v4 API schema:
+  The v4 /search/ endpoint returns a different schema from earlier versions.
+  This script uses the /search/?type=o endpoint which returns opinion cluster
+  results.  Full opinion text must often be fetched via a separate
+  /api/rest/v4/opinions/{id}/ call — this script handles that two-step flow.
+  See https://www.courtlistener.com/help/api/rest/v4/
+"""
+import argparse
+import json
+import logging
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Dict, List, Optional
+
+import requests
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+)
+logger = logging.getLogger(__name__)
+
+TOKEN: Optional[str] = os.environ.get("COURTLISTENER_TOKEN")
+BASE_URL = "https://www.courtlistener.com/api/rest/v4"
+OUTPUT_DIR = Path("test_data/legal_real")
+
+SEARCH_TERMS: Dict[str, str] = {
+    "contract_law": "breach of contract damages",
+    "intellectual_property": "patent infringement trademark",
+    "corporate_governance": "fiduciary duty shareholder derivative",
+}
+DOCS_PER_SUBDOMAIN: int = 50
+
+# CourtListener is not publicly rate-limited, but be polite.
+REQUEST_DELAY: float = 0.3  # seconds between requests
+
+# Maximum characters to save per document (avoid multi-MB opinions dominating
+# the corpus relative to shorter medical/financial docs).
+MAX_CHARS: int = 15_000
+
+
+# ---------------------------------------------------------------------------
+# API helpers
+# ---------------------------------------------------------------------------
+
+
+def _headers() -> dict:
+    if not TOKEN:
+        raise EnvironmentError(
+            "COURTLISTENER_TOKEN environment variable is not set.  "
+            "Get a free token at https://www.courtlistener.com/sign-in/"
+        )
+    return {"Authorization": f"Token {TOKEN}"}
+
+
+def _get(url: str, params: dict = None) -> dict:
+    """GET with auth, raise on HTTP errors, return parsed JSON."""
+    resp = requests.get(url, headers=_headers(), params=params or {}, timeout=30)
+    resp.raise_for_status()
+    return resp.json()
+
+
+# ---------------------------------------------------------------------------
+# Search & text retrieval
+# ---------------------------------------------------------------------------
+
+
+def search_opinions(query: str, count: int) -> List[dict]:
+    """
+    Search CourtListener for opinion clusters matching `query`.
+
+    Paginates through results until `count` items are collected or
+    there are no more pages.
+
+    Args:
+        query: Full-text search string.
+        count: Maximum number of results to return.
+
+    Returns:
+        List of result dicts from the CourtListener search API.
+    """
+    url = f"{BASE_URL}/search/"
+    params: dict = {
+        "q": query,
+        "type": "o",          # opinions
+        "order_by": "score desc",
+    }
+    collected: List[dict] = []
+
+    while len(collected) < count:
+        try:
+            data = _get(url, params)
+        except requests.HTTPError as exc:
+            logger.error("Search request failed: %s", exc)
+            break
+
+        results = data.get("results", [])
+        collected.extend(results)
+
+        next_url = data.get("next")
+        if not next_url:
+            break
+        # Paginate: use next URL directly (already contains all params).
+        url = next_url
+        params = {}
+        time.sleep(REQUEST_DELAY)
+
+    return collected[:count]
+
+
+def _extract_text_from_result(result: dict) -> str:
+    """
+    Pull opinion text from a search result dict.
+
+    CourtListener v4 search results for opinions may include:
+      - 'snippet'       — highlighted search snippet (short)
+      - 'plain_text'    — full plain text of the opinion (if indexed)
+
+    If neither is present (common for older opinions), fall back to
+    fetching the opinion's detail page via its cluster URL.
+    """
+    # Try direct text fields first (fast path).
+    for field in ("plain_text", "text", "snippet"):
+        val = result.get(field, "")
+        if val and len(val) > 200:
+            return val
+
+    # Slow path: fetch the first opinion document from the cluster.
+    cluster_id = result.get("cluster_id") or result.get("id")
+    if not cluster_id:
+        return ""
+
+    time.sleep(REQUEST_DELAY)
+    try:
+        cluster_data = _get(f"{BASE_URL}/clusters/{cluster_id}/")
+        sub_opinions = cluster_data.get("sub_opinions", [])
+        for opinion_url in sub_opinions:
+            # opinion_url is typically an absolute URL.
+            time.sleep(REQUEST_DELAY)
+            try:
+                op_data = _get(opinion_url)
+                for field in ("plain_text", "html_with_citations", "html", "xml_harvard"):
+                    text = op_data.get(field, "") or ""
+                    if len(text) > 200:
+                        # Strip HTML tags if necessary.
+                        if "<" in text:
+                            text = _strip_html(text)
+                        return text
+            except Exception as exc:
+                logger.debug("Opinion fetch error %s: %s", opinion_url, exc)
+    except Exception as exc:
+        logger.debug("Cluster fetch error %s: %s", cluster_id, exc)
+
+    return ""
+
+
+def _strip_html(html: str) -> str:
+    """Very lightweight HTML tag stripper using the stdlib only."""
+    import re
+    text = re.sub(r"<[^>]+>", " ", html)
+    text = re.sub(r"&[a-z]+;", " ", text)
+    text = re.sub(r" {2,}", " ", text)
+    return text.strip()
+
+
+# ---------------------------------------------------------------------------
+# Main fetch loop
+# ---------------------------------------------------------------------------
+
+
+def fetch_subdomain(
+    subdomain: str,
+    query: str,
+    n: int,
+    output_dir: Path,
+) -> List[Dict]:
+    """
+    Collect up to `n` opinion texts for `subdomain` and write them to disk.
+
+    Args:
+        subdomain:  Label string used in the manifest.
+        query:      CourtListener search query.
+        n:          Target document count.
+        output_dir: Directory to write .txt files into.
+
+    Returns:
+        List of manifest entry dicts for successfully saved documents.
+    """
+    logger.info("Subdomain '%s': searching '%s' …", subdomain, query)
+    results = search_opinions(query, n)
+    logger.info("  Retrieved %d search results.", len(results))
+
+    entries: List[Dict] = []
+    for result in results:
+        # Cluster / opinion ID.
+        op_id = result.get("cluster_id") or result.get("id")
+        if not op_id:
+            continue
+
+        text = _extract_text_from_result(result)
+        if len(text) < 200:
+            logger.debug("Skipping %s — insufficient text (%d chars).", op_id, len(text))
+            continue
+
+        text = text[:MAX_CHARS]
+        out_path = output_dir / f"case_{op_id}.txt"
+        out_path.write_text(text, encoding="utf-8")
+
+        entries.append(
+            {
+                "id": op_id,
+                "subdomain": subdomain,
+                "case_name": result.get("caseName") or result.get("case_name", ""),
+                "court": result.get("court", ""),
+                "date_filed": result.get("dateFiled") or result.get("date_filed", ""),
+                "source_url": (
+                    result.get("absolute_url")
+                    or f"https://www.courtlistener.com/?q=id:{op_id}"
+                ),
+                "path": str(out_path),
+                "char_count": len(text),
+            }
+        )
+        logger.info("  Saved case_%s (%d chars)", op_id, len(text))
+
+    logger.info("  Subdomain '%s': %d documents saved.", subdomain, len(entries))
+    return entries
+
+
+def main(docs_per_subdomain: int = DOCS_PER_SUBDOMAIN) -> None:
+    """Entry point."""
+    if not TOKEN:
+        sys.exit(
+            "Error: COURTLISTENER_TOKEN environment variable is not set.\n"
+            "Get a free API token at https://www.courtlistener.com/sign-in/"
+        )
+
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    manifest: List[Dict] = []
+
+    for subdomain, query in SEARCH_TERMS.items():
+        entries = fetch_subdomain(subdomain, query, docs_per_subdomain, OUTPUT_DIR)
+        manifest.extend(entries)
+
+    manifest_path = OUTPUT_DIR / "manifest.json"
+    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")
+    logger.info(
+        "Done.  %d legal documents fetched.  Manifest → %s",
+        len(manifest),
+        manifest_path,
+    )
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Fetch CourtListener opinions for the MEXAR legal corpus."
+    )
+    parser.add_argument(
+        "--docs-per-subdomain",
+        type=int,
+        default=DOCS_PER_SUBDOMAIN,
+        help="Documents to fetch per legal subdomain (default: %(default)s).",
+    )
+    args = parser.parse_args()
+    main(docs_per_subdomain=args.docs_per_subdomain)

backend/scripts/fetch_pubmed.py

@@ -0,0 +1,333 @@
+"""
+MEXAR - PubMed Central OA fetcher for the medical corpus.
+
+Pulls ~220 documents from PubMed Central Open Access across three medical
+subdomains (cardiology, oncology, internal medicine), parses the full-text XML
+into plain text, and writes one .txt file per document into
+test_data/medical_real/, along with a manifest.json for auditing.
+
+Usage:
+    python backend/scripts/fetch_pubmed.py
+
+Requirements:
+    pip install requests lxml
+
+Environment:
+    NCBI_EMAIL  — your email address (required by NCBI usage policy)
+    NCBI_API_KEY — optional; raises rate limit from 3 req/sec to 10 req/sec
+
+Run this locally (not in a network-restricted sandbox).
+"""
+import argparse
+import json
+import logging
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Dict, List, Optional
+
+import requests
+
+try:
+    from lxml import etree
+except ImportError:
+    sys.exit(
+        "lxml is required — run: pip install lxml"
+    )
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+)
+logger = logging.getLogger(__name__)
+
+# NCBI email is mandatory per NCBI E-utilities usage policy.
+# Override with the NCBI_EMAIL environment variable or the --email CLI flag.
+DEFAULT_NCBI_EMAIL: str = os.getenv("NCBI_EMAIL", "your_email@example.com")
+NCBI_API_KEY: Optional[str] = os.getenv("NCBI_API_KEY")  # optional; raises rate cap to 10/s
+
+# One .txt per doc, one manifest.json — relative to repo root.
+OUTPUT_DIR = Path("test_data/medical_real")
+
+# Three subdomains × 75 docs ≈ 220 total (some may fail or be empty).
+SEARCH_TERMS: Dict[str, str] = {
+    "cardiology": (
+        '"cardiology treatment guideline"[Title/Abstract] AND "open access"[filter]'
+    ),
+    "oncology": (
+        '"oncology treatment protocol"[Title/Abstract] AND "open access"[filter]'
+    ),
+    "internal_medicine": (
+        '"internal medicine diagnostic criteria"[Title/Abstract] AND "open access"[filter]'
+    ),
+}
+DOCS_PER_SUBDOMAIN: int = 75
+
+# NCBI rate limits: 3 req/sec without key, 10/sec with key.
+REQUEST_DELAY_NO_KEY: float = 0.35   # ≈ 2.9 req/sec — safe without key
+REQUEST_DELAY_WITH_KEY: float = 0.12  # ≈ 8.3 req/sec — safe with key
+
+NCBI_BASE = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
+
+# ---------------------------------------------------------------------------
+# NCBI E-utilities helpers
+# ---------------------------------------------------------------------------
+
+
+def _ncbi_params(extra: dict, email: str) -> dict:
+    """Return base params dict merged with extra, including optional API key."""
+    params = {"email": email, **extra}
+    if NCBI_API_KEY:
+        params["api_key"] = NCBI_API_KEY
+    return params
+
+
+def esearch(term: str, retmax: int, email: str) -> List[str]:
+    """
+    Run an E-search against the PMC database and return a list of PMC IDs.
+
+    Args:
+        term:   NCBI search query string.
+        retmax: Maximum number of IDs to retrieve.
+        email:  Requester email (required by NCBI).
+
+    Returns:
+        List of PMC ID strings (numeric, without the 'PMC' prefix).
+    """
+    url = f"{NCBI_BASE}/esearch.fcgi"
+    params = _ncbi_params(
+        {"db": "pmc", "term": term, "retmax": retmax, "retmode": "json"},
+        email,
+    )
+    resp = requests.get(url, params=params, timeout=30)
+    resp.raise_for_status()
+    return resp.json()["esearchresult"]["idlist"]
+
+
+def efetch_xml(pmc_id: str, email: str) -> bytes:
+    """
+    Fetch the full-text XML for a single PMC article.
+
+    Args:
+        pmc_id: Numeric PMC ID string (without the 'PMC' prefix).
+        email:  Requester email.
+
+    Returns:
+        Raw XML bytes from the PMC OA full-text endpoint.
+    """
+    url = f"{NCBI_BASE}/efetch.fcgi"
+    params = _ncbi_params(
+        {"db": "pmc", "id": pmc_id, "rettype": "full", "retmode": "xml"},
+        email,
+    )
+    resp = requests.get(url, params=params, timeout=60)
+    resp.raise_for_status()
+    return resp.content
+
+
+# ---------------------------------------------------------------------------
+# XML → plain-text extraction
+# ---------------------------------------------------------------------------
+
+# Tags in the JATS PMC XML that contain useful body text.
+_BODY_TAGS = {"abstract", "body", "sec", "p", "title"}
+
+
+def _iter_text(element) -> str:
+    """
+    Recursively collect all text inside an lxml element, with
+    paragraph-level whitespace to keep the output readable.
+    """
+    parts = []
+    if element.text:
+        parts.append(element.text.strip())
+    for child in element:
+        child_text = _iter_text(child)
+        if child_text:
+            # Add a blank line between sections/paragraphs.
+            sep = "\n\n" if child.tag in {"sec", "p", "title", "abstract"} else " "
+            parts.append(sep + child_text)
+        if child.tail:
+            parts.append(child.tail.strip())
+    return " ".join(p for p in parts if p)
+
+
+def xml_to_plain_text(xml_bytes: bytes) -> str:
+    """
+    Parse PMC full-text XML and extract human-readable plain text.
+
+    Pulls text from <abstract>, <body>, and nested <sec>/<p> elements,
+    strips all XML markup, and returns a clean, whitespace-normalised string.
+
+    Args:
+        xml_bytes: Raw XML bytes as returned by efetch.
+
+    Returns:
+        Plain-text string.  Empty string if parsing fails.
+    """
+    try:
+        root = etree.fromstring(xml_bytes)
+    except etree.XMLSyntaxError as exc:
+        logger.warning("XML parse error: %s", exc)
+        return ""
+
+    sections: List[str] = []
+    # Pull <article-title>, <abstract>, and <body> in document order.
+    for tag in ("article-title", "abstract", "body"):
+        for element in root.iter(tag):
+            text = _iter_text(element).strip()
+            if text:
+                sections.append(text)
+
+    plain = "\n\n".join(sections)
+    # Collapse excessive whitespace.
+    import re
+    plain = re.sub(r" {2,}", " ", plain)
+    plain = re.sub(r"\n{3,}", "\n\n", plain)
+    return plain.strip()
+
+
+def _get_article_meta(xml_bytes: bytes) -> Dict[str, str]:
+    """Extract title and journal name from parsed PMC XML for the manifest."""
+    meta: Dict[str, str] = {"title": "", "journal": ""}
+    try:
+        root = etree.fromstring(xml_bytes)
+        title_el = root.find(".//article-title")
+        if title_el is not None:
+            meta["title"] = (_iter_text(title_el) or "").strip()
+        journal_el = root.find(".//journal-title")
+        if journal_el is not None:
+            meta["journal"] = (journal_el.text or "").strip()
+    except Exception:
+        pass
+    return meta
+
+
+# ---------------------------------------------------------------------------
+# Main fetch loop
+# ---------------------------------------------------------------------------
+
+
+def fetch_subdomain(
+    subdomain: str,
+    term: str,
+    n: int,
+    output_dir: Path,
+    email: str,
+    delay: float,
+) -> List[Dict]:
+    """
+    Search for and download up to `n` PMC articles matching `term`.
+
+    Args:
+        subdomain:  Label string used in the manifest.
+        term:       NCBI search query.
+        n:          Target number of documents.
+        output_dir: Directory to write .txt files into.
+        email:      NCBI requester email.
+        delay:      Seconds to sleep between requests.
+
+    Returns:
+        List of manifest entry dicts for successfully fetched documents.
+    """
+    logger.info("Subdomain '%s': searching for %d docs …", subdomain, n)
+    try:
+        ids = esearch(term, n, email)
+    except Exception as exc:
+        logger.error("esearch failed for '%s': %s", subdomain, exc)
+        return []
+
+    logger.info("  Found %d IDs; fetching full text …", len(ids))
+    entries: List[Dict] = []
+
+    for pmc_id in ids:
+        time.sleep(delay)
+        try:
+            xml_bytes = efetch_xml(pmc_id, email)
+            plain = xml_to_plain_text(xml_bytes)
+        except requests.HTTPError as exc:
+            logger.warning("HTTP error for PMC%s: %s", pmc_id, exc)
+            continue
+        except Exception as exc:
+            logger.warning("Failed PMC%s: %s", pmc_id, exc)
+            continue
+
+        if len(plain) < 200:
+            logger.debug("PMC%s: text too short (%d chars), skipping.", pmc_id, len(plain))
+            continue
+
+        out_path = output_dir / f"PMC{pmc_id}.txt"
+        out_path.write_text(plain, encoding="utf-8")
+
+        meta = _get_article_meta(xml_bytes)
+        entries.append(
+            {
+                "pmc_id": pmc_id,
+                "subdomain": subdomain,
+                "title": meta["title"],
+                "journal": meta["journal"],
+                "source_url": f"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC{pmc_id}/",
+                "path": str(out_path),
+                "char_count": len(plain),
+            }
+        )
+        logger.info("  Saved PMC%s (%d chars)", pmc_id, len(plain))
+
+    logger.info("  Subdomain '%s': %d documents saved.", subdomain, len(entries))
+    return entries
+
+
+def main(email: str = DEFAULT_NCBI_EMAIL, docs_per_subdomain: int = DOCS_PER_SUBDOMAIN) -> None:
+    """
+    Entry point.  Iterates over all three medical subdomains and writes results.
+    """
+    if email == "your_email@example.com":
+        logger.warning(
+            "Using placeholder email — set the NCBI_EMAIL env var or pass --email. "
+            "NCBI may throttle requests without a valid email."
+        )
+
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    delay = REQUEST_DELAY_WITH_KEY if NCBI_API_KEY else REQUEST_DELAY_NO_KEY
+    manifest: List[Dict] = []
+
+    for subdomain, term in SEARCH_TERMS.items():
+        entries = fetch_subdomain(subdomain, term, docs_per_subdomain, OUTPUT_DIR, email, delay)
+        manifest.extend(entries)
+
+    manifest_path = OUTPUT_DIR / "manifest.json"
+    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")
+    logger.info(
+        "Done.  %d medical documents fetched.  Manifest → %s",
+        len(manifest),
+        manifest_path,
+    )
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Fetch PubMed Central OA articles for the MEXAR medical corpus."
+    )
+    parser.add_argument(
+        "--email",
+        default=DEFAULT_NCBI_EMAIL,
+        help="Your email address (required by NCBI usage policy).",
+    )
+    parser.add_argument(
+        "--docs-per-subdomain",
+        type=int,
+        default=DOCS_PER_SUBDOMAIN,
+        help="Number of documents to fetch per subdomain (default: %(default)s).",
+    )
+    args = parser.parse_args()
+    main(email=args.email, docs_per_subdomain=args.docs_per_subdomain)

backend/scripts/fetch_secedgar.py

@@ -0,0 +1,467 @@
+"""
+MEXAR - SEC EDGAR fetcher for the financial corpus.
+
+Fetches ~130 10-K filing sections (MD&A, Risk Factors) from SEC EDGAR using
+the full-text search API, follows through to the actual EDGAR filing documents,
+extracts the relevant section text, and saves one .txt file per document plus
+a manifest.json.
+
+Usage:
+    python backend/scripts/fetch_secedgar.py
+
+Requirements:
+    pip install requests beautifulsoup4 lxml
+
+SEC Fair Access Policy:
+    Every request MUST include a descriptive User-Agent header in the format:
+    "FirstName LastName email@example.com"
+    without this, SEC will block your IP.
+
+    Override with the SEC_USER_AGENT environment variable.
+
+Run this locally (not in a network-restricted sandbox).
+"""
+import argparse
+import json
+import logging
+import os
+import re
+import sys
+import time
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+from urllib.parse import urljoin
+
+import requests
+
+try:
+    from bs4 import BeautifulSoup
+except ImportError:
+    sys.exit("beautifulsoup4 is required — run: pip install beautifulsoup4 lxml")
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+)
+logger = logging.getLogger(__name__)
+
+# SEC requires a descriptive User-Agent on every request.
+# Format: "Name Email" — use yours or set SEC_USER_AGENT env var.
+_DEFAULT_USER_AGENT = "Devrajsinh Gohil djgohil2012@gmail.com"
+USER_AGENT: str = os.getenv("SEC_USER_AGENT", _DEFAULT_USER_AGENT)
+
+HEADERS: Dict[str, str] = {
+    "User-Agent": USER_AGENT,
+    "Accept-Encoding": "gzip, deflate",
+    "Host": "efts.sec.gov",
+}
+EDGAR_HEADERS: Dict[str, str] = {
+    "User-Agent": USER_AGENT,
+    "Accept-Encoding": "gzip, deflate",
+}
+
+OUTPUT_DIR = Path("test_data/financial_real")
+
+# Search terms and per-term document targets (~130 total).
+SEARCH_CONFIG: List[Tuple[str, str, int]] = [
+    ("risk_factors",              "risk factors",                     45),
+    ("management_discussion",     "management discussion and analysis", 45),
+    ("accounting_standards",      "GAAP IFRS accounting standards",   40),
+]
+
+# Maximum characters per saved document — keep consistent with medical/legal sizes.
+MAX_CHARS: int = 15_000
+
+# Be polite to SEC servers.
+REQUEST_DELAY: float = 0.25  # seconds
+
+# EDGAR full-text search endpoint.
+EFTS_URL = "https://efts.sec.gov/LATEST/search-index"
+EDGAR_ARCHIVES = "https://www.sec.gov/Archives/edgar/data"
+
+# ---------------------------------------------------------------------------
+# SEC EDGAR API helpers
+# ---------------------------------------------------------------------------
+
+
+def _get(url: str, params: dict = None, extra_headers: dict = None) -> requests.Response:
+    """GET with SEC-compliant headers and error handling."""
+    h = {**EDGAR_HEADERS}
+    if extra_headers:
+        h.update(extra_headers)
+    resp = requests.get(url, headers=h, params=params or {}, timeout=30)
+    resp.raise_for_status()
+    return resp
+
+
+def full_text_search(query: str, count: int) -> List[dict]:
+    """
+    Query the SEC EDGAR full-text search API for 10-K filings.
+
+    The EFTS search endpoint returns hit metadata only — not the filing body.
+    Each hit contains the accession number and CIK needed to locate the filing
+    document in EDGAR archives.
+
+    Args:
+        query: Search phrase (e.g. "risk factors").
+        count: Maximum number of hits to return.
+
+    Returns:
+        List of _source dicts from the EFTS hits array.
+    """
+    # EFTS search uses query-string encoding.
+    params = {
+        "q": f'"{query}"',
+        "forms": "10-K",
+        "dateRange": "custom",
+        "startdt": "2020-01-01",
+        "enddt": "2024-12-31",
+    }
+    try:
+        resp = _get(
+            EFTS_URL,
+            params=params,
+            extra_headers={"Host": "efts.sec.gov"},
+        )
+    except Exception as exc:
+        logger.error("EFTS search failed for '%s': %s", query, exc)
+        return []
+
+    data = resp.json()
+    hits = data.get("hits", {}).get("hits", [])
+    logger.info("  EFTS returned %d hits for '%s'", len(hits), query)
+    return [h.get("_source", {}) for h in hits[:count]]
+
+
+# ---------------------------------------------------------------------------
+# Filing document text extraction
+# ---------------------------------------------------------------------------
+
+def _accession_to_path(accession: str) -> str:
+    """Convert '0001234567-23-012345' → '000123456723012345'."""
+    return accession.replace("-", "")
+
+
+def _get_filing_index(cik: str, accession: str) -> Optional[dict]:
+    """
+    Fetch the filing index JSON from EDGAR to find the primary document URL.
+
+    Returns the parsed index data dict, or None on failure.
+    """
+    cik_int = str(int(cik)) if cik.isdigit() else cik
+    acc_path = _accession_to_path(accession)
+    url = (
+        f"https://data.sec.gov/submissions/CIK{cik_int.zfill(10)}.json"
+    )
+    # Use the filing index endpoint instead.
+    index_url = (
+        f"https://www.sec.gov/cgi-bin/browse-edgar"
+        f"?action=getcompany&CIK={cik_int}&type=10-K&dateb=&owner=include&count=1&search_text="
+    )
+    # Directly hit the accession-number based index.
+    direct_url = (
+        f"https://www.sec.gov/Archives/edgar/data/{cik_int}/{acc_path}/{accession}-index.json"
+    )
+    try:
+        resp = _get(direct_url)
+        return resp.json()
+    except Exception:
+        pass
+    # Fallback: try the .htm index.
+    try:
+        htm_url = (
+            f"https://www.sec.gov/Archives/edgar/data/{cik_int}/{acc_path}/{accession}-index.htm"
+        )
+        resp = _get(htm_url)
+        # Parse the index page for the primary document link.
+        soup = BeautifulSoup(resp.text, "lxml")
+        return {"_html": resp.text, "_base": f"https://www.sec.gov/Archives/edgar/data/{cik_int}/{acc_path}/"}
+    except Exception as exc:
+        logger.debug("Filing index fetch failed for %s/%s: %s", cik, accession, exc)
+        return None
+
+
+def _find_primary_document_url(cik: str, accession: str, index_data: dict) -> Optional[str]:
+    """
+    Given filing index data, return the URL of the primary 10-K HTML document.
+    """
+    acc_path = _accession_to_path(accession)
+    base = f"https://www.sec.gov/Archives/edgar/data/{cik}/{acc_path}/"
+
+    # JSON index format (modern filings).
+    if "documents" in index_data:
+        for doc in index_data.get("documents", []):
+            doc_type = doc.get("type", "").upper()
+            if doc_type in ("10-K", "10-K/A"):
+                filename = doc.get("filename") or doc.get("documentName", "")
+                if filename:
+                    return base + filename
+
+    # HTML index fallback.
+    if "_html" in index_data:
+        soup = BeautifulSoup(index_data["_html"], "lxml")
+        for row in soup.find_all("tr"):
+            cells = row.find_all("td")
+            if len(cells) >= 3:
+                doc_type = cells[3].get_text(strip=True) if len(cells) > 3 else ""
+                if "10-K" in doc_type:
+                    link = cells[2].find("a")
+                    if link and link.get("href"):
+                        href = link["href"]
+                        if href.startswith("/"):
+                            return "https://www.sec.gov" + href
+                        return href
+        # If no 10-K row found, grab any .htm link from the table.
+        for link in soup.select("table a[href]"):
+            href = link["href"]
+            if href.endswith(".htm") or href.endswith(".html"):
+                if href.startswith("/"):
+                    return "https://www.sec.gov" + href
+                return href
+
+    return None
+
+
+def _extract_section(html: str, section: str) -> str:
+    """
+    Extract a named section (e.g. 'risk factors', 'management discussion')
+    from a 10-K HTML document.
+
+    Strategy:
+    1. Parse with BeautifulSoup.
+    2. Find the first heading element (<b>, <strong>, <h2>, <h3>, <p>) whose
+       text closely matches the section name.
+    3. Collect all subsequent paragraph text until the next major heading.
+
+    Falls back to returning the first 15,000 chars of the full body text if
+    no matching section heading is found.
+    """
+    soup = BeautifulSoup(html, "lxml")
+
+    # Remove script/style noise.
+    for tag in soup(["script", "style", "meta", "link"]):
+        tag.decompose()
+
+    section_lower = section.lower()
+    # Headings to search in.
+    heading_candidates = soup.find_all(
+        ["b", "strong", "h1", "h2", "h3", "h4", "p"],
+    )
+
+    section_start = None
+    for el in heading_candidates:
+        text = el.get_text(" ", strip=True).lower()
+        if section_lower in text and len(text) < 200:
+            section_start = el
+            break
+
+    if section_start is None:
+        # No section heading found — return the full text (truncated).
+        full_text = soup.get_text(" ", strip=True)
+        return re.sub(r" {2,}", " ", full_text)
+
+    # Collect text from siblings/following elements until next major heading.
+    parts: List[str] = []
+    total = 0
+    for sibling in section_start.find_all_next():
+        tag_name = sibling.name or ""
+        text = sibling.get_text(" ", strip=True)
+        if not text:
+            continue
+
+        # Stop at the next major heading that looks like a different section.
+        if tag_name in ("h1", "h2", "h3"):
+            break
+        if tag_name in ("b", "strong", "p") and len(text) < 200:
+            text_lower = text.lower()
+            # Check for common 10-K section heading patterns.
+            if re.match(r"(item\s+\d|part\s+[iv]+)", text_lower):
+                break
+
+        parts.append(text)
+        total += len(text)
+        if total >= MAX_CHARS:
+            break
+
+    extracted = " ".join(parts)
+    extracted = re.sub(r" {2,}", " ", extracted)
+    return extracted.strip()
+
+
+def fetch_and_extract(source: dict, query_label: str) -> Tuple[str, str, int]:
+    """
+    Follow through from an EFTS hit's metadata to the actual filing text.
+
+    Args:
+        source:      _source dict from an EFTS hit.
+        query_label: Human-readable label for the search term (used for
+                     section extraction hints).
+
+    Returns:
+        (text, filing_url, char_count)  — text is empty string on failure.
+    """
+    cik_list = source.get("ciks", [])
+    cik = str(cik_list[0]) if cik_list else source.get("cik", "")
+    accession = source.get("adsh", "")
+
+    if not cik or not accession:
+        return "", "", 0
+
+    acc_path = _accession_to_path(accession)
+    filing_url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{acc_path}/"
+
+    time.sleep(REQUEST_DELAY)
+
+    # Step 1: Get filing index to find the primary document.
+    index_data = _get_filing_index(cik, accession)
+    if not index_data:
+        logger.debug("No index for %s / %s", cik, accession)
+        return "", filing_url, 0
+
+    doc_url = _find_primary_document_url(cik, acc_path, index_data)
+    if not doc_url:
+        logger.debug("Could not locate primary doc for %s / %s", cik, accession)
+        return "", filing_url, 0
+
+    time.sleep(REQUEST_DELAY)
+
+    # Step 2: Fetch the 10-K document.
+    try:
+        resp = _get(doc_url)
+        html = resp.text
+    except Exception as exc:
+        logger.debug("Failed to fetch filing document %s: %s", doc_url, exc)
+        return "", filing_url, 0
+
+    # Step 3: Extract the relevant section.
+    section_hint = query_label.replace("_", " ")
+    text = _extract_section(html, section_hint)
+    if len(text) > MAX_CHARS:
+        text = text[:MAX_CHARS]
+
+    return text, doc_url, len(text)
+
+
+# ---------------------------------------------------------------------------
+# Main fetch loop
+# ---------------------------------------------------------------------------
+
+
+def fetch_term(
+    label: str,
+    query: str,
+    n: int,
+    output_dir: Path,
+) -> List[Dict]:
+    """
+    Search EDGAR for `n` filings matching `query`, extract section text, save.
+
+    Args:
+        label:      Short label for the search term (used in filename + manifest).
+        query:      EDGAR full-text search phrase.
+        n:          Target document count.
+        output_dir: Directory to write .txt files into.
+
+    Returns:
+        List of manifest entry dicts.
+    """
+    logger.info("Term '%s': searching EDGAR for %d docs …", label, n)
+    sources = full_text_search(query, n)
+
+    entries: List[Dict] = []
+    seen = set()
+
+    for source in sources:
+        cik_list = source.get("ciks", [])
+        cik = str(cik_list[0]) if cik_list else source.get("cik", "")
+        accession = source.get("adsh", "")
+
+        if not cik or not accession:
+            continue
+
+        doc_key = f"{cik}_{accession}"
+        if doc_key in seen:
+            continue
+        seen.add(doc_key)
+
+        text, doc_url, char_count = fetch_and_extract(source, label)
+        if char_count < 200:
+            logger.debug("Skipping %s — insufficient text (%d chars).", doc_key, char_count)
+            continue
+
+        out_path = output_dir / f"{doc_key}.txt"
+        out_path.write_text(text, encoding="utf-8")
+
+        company_names = source.get("display_names", [])
+        company = company_names[0] if company_names else source.get("entity_name", "")
+
+        entries.append(
+            {
+                "cik": cik,
+                "accession": accession,
+                "company": company,
+                "term": label,
+                "filing_date": source.get("period_of_report", ""),
+                "source_url": doc_url,
+                "path": str(out_path),
+                "char_count": char_count,
+            }
+        )
+        logger.info("  Saved %s (%d chars)", doc_key, char_count)
+
+    logger.info("  Term '%s': %d documents saved.", label, len(entries))
+    return entries
+
+
+def main() -> None:
+    """Entry point."""
+    if USER_AGENT == _DEFAULT_USER_AGENT:
+        logger.warning(
+            "Using default User-Agent '%s'. "
+            "Set the SEC_USER_AGENT env var to 'YourName your@email.com' "
+            "to comply with SEC fair access policy.",
+            USER_AGENT,
+        )
+
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    manifest: List[Dict] = []
+
+    for label, query, target_n in SEARCH_CONFIG:
+        entries = fetch_term(label, query, target_n, OUTPUT_DIR)
+        manifest.extend(entries)
+
+    manifest_path = OUTPUT_DIR / "manifest.json"
+    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")
+    logger.info(
+        "Done.  %d financial documents fetched.  Manifest → %s",
+        len(manifest),
+        manifest_path,
+    )
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Fetch SEC EDGAR 10-K sections for the MEXAR financial corpus."
+    )
+    parser.add_argument(
+        "--user-agent",
+        default=None,
+        help=(
+            "SEC-compliant User-Agent string ('FirstName LastName email@example.com'). "
+            "Overrides the SEC_USER_AGENT environment variable."
+        ),
+    )
+    args = parser.parse_args()
+    if args.user_agent:
+        USER_AGENT = args.user_agent
+        HEADERS["User-Agent"] = USER_AGENT
+        EDGAR_HEADERS["User-Agent"] = USER_AGENT
+    main()

backend/scripts/run_migrations.py

@@ -0,0 +1,92 @@
+import os
+import sys
+import psycopg2
+from dotenv import load_dotenv
+
+if sys.stdout.encoding.lower() != 'utf-8':
+    sys.stdout.reconfigure(encoding='utf-8')
+
+load_dotenv(os.path.join(os.path.dirname(__file__), "..", ".env"))
+
+db_url = os.getenv("DATABASE_URL")
+if not db_url:
+    raise ValueError("DATABASE_URL environment variable is not set in backend/.env")
+
+print("Connecting to database to apply migrations...")
+conn = psycopg2.connect(db_url)
+conn.autocommit = True
+cur = conn.cursor()
+
+migrations = [
+    (
+        "Phase 1.1: Add domain_signature_weights & domain_entities columns to agents table",
+        """
+        ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_signature_weights JSONB;
+        ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_entities JSONB;
+        COMMENT ON COLUMN agents.domain_signature_weights IS
+          'TF-IDF lexical weights dict {term: weight} for domain signature (Section III-A, Eq. 1)';
+        COMMENT ON COLUMN agents.domain_entities IS
+          'NER-extracted domain entities list for domain signature (Section III-A)';
+        """
+    ),
+    (
+        "Phase 1.2: Switch vector index on document_chunks from IVFFlat to HNSW",
+        """
+        DROP INDEX IF EXISTS idx_document_chunks_embedding;
+        DROP INDEX IF EXISTS chunks_embedding_idx;
+        CREATE INDEX IF NOT EXISTS idx_document_chunks_embedding_hnsw
+        ON document_chunks USING hnsw (embedding vector_cosine_ops)
+        WITH (m = 16, ef_construction = 64);
+        COMMENT ON INDEX idx_document_chunks_embedding_hnsw IS
+          'HNSW index for cosine similarity search, m=16, ef_construction=64 (Section III-B)';
+        """
+    ),
+    (
+        "Add preferences column to users table",
+        """
+        ALTER TABLE users ADD COLUMN IF NOT EXISTS preferences JSON DEFAULT '{}';
+        """
+    )
+]
+
+for name, sql in migrations:
+    print(f"\nApplying migration: {name}...")
+    try:
+        cur.execute(sql)
+        print(f"[SUCCESS] {name}")
+    except Exception as e:
+        print(f"[ERROR] applying {name}: {e}")
+
+print("\n--- Migration Verification ---")
+
+# Verify agents table columns
+cur.execute("SELECT column_name FROM information_schema.columns WHERE table_name = 'agents';")
+agent_cols = [r[0] for r in cur.fetchall()]
+print("Agents columns:", agent_cols)
+assert "domain_signature_weights" in agent_cols, "domain_signature_weights missing from agents!"
+assert "domain_entities" in agent_cols, "domain_entities missing from agents!"
+print("  [OK] agents.domain_signature_weights present")
+print("  [OK] agents.domain_entities present")
+
+# Verify document_chunks index
+cur.execute("SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'document_chunks';")
+chunk_indexes = cur.fetchall()
+print("\nDocument Chunks indexes:")
+hnsw_found = False
+for idx_name, idx_def in chunk_indexes:
+    print(f"  - {idx_name}: {idx_def}")
+    if "hnsw" in idx_def.lower():
+        hnsw_found = True
+
+assert hnsw_found, "HNSW index missing on document_chunks!"
+print("  [OK] HNSW index verified on document_chunks.embedding")
+
+# Verify users preferences column
+cur.execute("SELECT column_name FROM information_schema.columns WHERE table_name = 'users';")
+user_cols = [r[0] for r in cur.fetchall()]
+assert "preferences" in user_cols, "preferences missing from users!"
+print("  [OK] users.preferences column present")
+
+cur.close()
+conn.close()
+print("\nALL MIGRATIONS COMPLETED AND VERIFIED SUCCESSFULLY!")

backend/utils/domain_signature.py

@@ -0,0 +1,187 @@
+"""
+MEXAR - Domain Signature Construction
+Implements Section III-A: TF-IDF lexical signature + NER entity signature.
+
+build_domain_signature() produces Sigma = Sigma_lex ∪ Sigma_ent, kept separate
+for inspection. The 'combined' list is stored in the domain_signature JSONB column
+on agents; 'lexical_weights' is stored in domain_signature_weights for optional
+weighted Jaccard use later.
+"""
+from typing import List, Dict, Set, Tuple
+from sklearn.feature_extraction.text import TfidfVectorizer
+import spacy
+import logging
+
+logger = logging.getLogger(__name__)
+
+# Lazy-loaded spaCy model (avoid paying load cost at import time)
+_nlp = None
+
+# Entity labels relevant to professional domains (medical / legal / financial)
+RELEVANT_ENTITY_LABELS = {"ORG", "LAW", "PRODUCT", "GPE", "PERSON", "NORP", "EVENT", "FAC"}
+
+
+def get_nlp():
+    """Lazy-load and cache the spaCy en_core_web_sm model."""
+    global _nlp
+    if _nlp is None:
+        try:
+            _nlp = spacy.load("en_core_web_sm")
+            logger.info("spaCy en_core_web_sm loaded for domain guardrail NER.")
+        except OSError:
+            logger.error(
+                "spaCy model 'en_core_web_sm' not found. "
+                "Run: python -m spacy download en_core_web_sm"
+            )
+            raise
+    return _nlp
+
+
+def build_tfidf_signature(
+    documents: List[str],
+    top_n: int = 100,
+    tau_tf: float = 0.0,
+) -> List[Tuple[str, float]]:
+    """
+    Eq. 1: w(t, D) = tf(t, D) * log(|D| / |{d in D : t in d}|)
+
+    Returns top_n (term, weight) pairs above tau_tf, sorted descending by weight.
+
+    Args:
+        documents: List of per-source document texts.
+        top_n: Maximum number of terms to return.
+        tau_tf: Minimum TF-IDF weight threshold.
+
+    Returns:
+        List of (term, weight) tuples sorted by weight descending.
+    """
+    if not documents:
+        return []
+
+    # TF-IDF requires a corpus of multiple documents so IDF is non-degenerate.
+    # If only a single document is provided (e.g. one uploaded file), split it
+    # into paragraph-level pseudo-documents.
+    if len(documents) < 2:
+        documents = _split_into_pseudo_documents(documents[0]) if documents[0] else []
+        if not documents:
+            return []
+
+    vectorizer = TfidfVectorizer(
+        max_features=2000,
+        stop_words="english",
+        ngram_range=(1, 2),  # unigrams + bigrams for better domain specificity
+        min_df=1,
+    )
+    try:
+        matrix = vectorizer.fit_transform(documents)
+    except ValueError as e:
+        logger.warning(f"TF-IDF vectorizer failed: {e}")
+        return []
+
+    # Sum TF-IDF weights across all documents for each term
+    scores = matrix.sum(axis=0).A1
+    terms = vectorizer.get_feature_names_out()
+
+    pairs = [(t, float(s)) for t, s in zip(terms, scores) if s > tau_tf]
+    pairs.sort(key=lambda x: x[1], reverse=True)
+    return pairs[:top_n]
+
+
+def _split_into_pseudo_documents(text: str, chunk_size: int = 500) -> List[str]:
+    """Split a single long text into word-chunk pseudo-documents for TF-IDF."""
+    words = text.split()
+    chunks = [" ".join(words[i:i + chunk_size]) for i in range(0, len(words), chunk_size)]
+    return chunks or [text]
+
+
+def extract_ner_entities(
+    documents: List[str],
+    tau_ent: int = 2,
+    max_docs_to_scan: int = 200,
+) -> List[str]:
+    """
+    Extracts named entities (Sigma_ent) appearing in at least tau_ent documents.
+
+    Capped at max_docs_to_scan for latency (spaCy NER is not free at compile time).
+    Per-document de-duplication avoids inflating doc-frequency for repeated mentions.
+
+    Args:
+        documents: List of per-source document texts.
+        tau_ent: Minimum document frequency for an entity to be included.
+        max_docs_to_scan: Cap on number of documents processed (speed/latency trade-off).
+
+    Returns:
+        List of entity strings appearing in >= tau_ent documents.
+    """
+    if not documents:
+        return []
+
+    nlp = get_nlp()
+    doc_freq: Dict[str, int] = {}
+
+    for doc_text in documents[:max_docs_to_scan]:
+        seen_in_this_doc: Set[str] = set()
+        try:
+            for ent in nlp(doc_text[:5000]).ents:  # cap per-doc chars for speed
+                if ent.label_ in RELEVANT_ENTITY_LABELS:
+                    key = ent.text.lower().strip()
+                    if key and len(key) > 2 and key not in seen_in_this_doc:
+                        seen_in_this_doc.add(key)
+        except Exception as e:
+            logger.warning(f"NER pass failed for a document: {e}")
+            continue
+
+        for key in seen_in_this_doc:
+            doc_freq[key] = doc_freq.get(key, 0) + 1
+
+    return [k for k, v in doc_freq.items() if v >= tau_ent]
+
+
+def build_domain_signature(
+    documents: List[str],
+    tau_tf: float = 0.0,
+    tau_ent: int = 2,
+    top_n_lexical: int = 100,
+) -> Dict[str, object]:
+    """
+    Returns Sigma = Sigma_lex ∪ Sigma_ent (Section III-A).
+
+    The two components are kept separate for inspection and future weighted Jaccard use.
+
+    Args:
+        documents: List of per-source document texts (NOT a single concatenated string).
+        tau_tf: TF-IDF weight threshold for lexical terms.
+        tau_ent: Minimum doc-frequency for NER entities.
+        top_n_lexical: Number of top TF-IDF terms to retain.
+
+    Returns:
+        Dict with keys:
+            - 'lexical': list of top TF-IDF terms
+            - 'lexical_weights': dict of {term: weight}
+            - 'entities': list of NER entities meeting tau_ent threshold
+            - 'combined': sorted list of all unique terms (lexical ∪ entities)
+    """
+    logger.info(
+        f"Building domain signature from {len(documents)} documents "
+        f"(tau_tf={tau_tf}, tau_ent={tau_ent}, top_n={top_n_lexical})"
+    )
+
+    lexical_pairs = build_tfidf_signature(documents, top_n=top_n_lexical, tau_tf=tau_tf)
+    lexical_terms = [t for t, _ in lexical_pairs]
+    lexical_weights = {t: w for t, w in lexical_pairs}
+
+    entities = extract_ner_entities(documents, tau_ent=tau_ent)
+
+    combined = sorted(set(lexical_terms) | set(entities))
+
+    logger.info(
+        f"Domain signature built: {len(lexical_terms)} lexical terms, "
+        f"{len(entities)} NER entities, {len(combined)} combined."
+    )
+
+    return {
+        "lexical": lexical_terms,
+        "lexical_weights": lexical_weights,
+        "entities": entities,
+        "combined": combined,
+    }

backend/utils/faithfulness.py

@@ -213,66 +213,142 @@ def create_faithfulness_scorer() -> FaithfulnessScorer:
     return FaithfulnessScorer()
 
 
-class BartNLIScorer:
+class DebertaNLIScorer:
     """
-    Evaluates faithfulness using a local NLI model (BART-Large-MNLI) 
-    to break the circular evaluation where the generator evaluates itself.
+    Primary faithfulness verifier per Section III-C.
+
+    Uses cross-encoder/nli-deberta-v3-large (DeBERTa-v3-large fine-tuned on MNLI)
+    to break the circular LLM-judges-LLM evaluation pattern.
+
+    Implements:
+    - Eq. 6: per-claim, per-document max entailment probability
+    - Eq. 7: faithfulness score = fraction of claims exceeding TAU_ENT
+
+    score() receives individual retrieved chunk texts (NOT concatenated), so the
+    per-document max in Eq. 6 is meaningful.
     """
+
+    TAU_ENT = 0.7  # Entailment probability threshold (Eq. 7)
+
     def __init__(self):
-        self._pipe = None
-        
+        self._model = None
+        self._entailment_idx: int = 1  # Verified at load time from id2label
+
     @property
-    def pipe(self):
-        if self._pipe is None:
-            import logging
-            logger = logging.getLogger(__name__)
+    def model(self):
+        """Lazy-load CrossEncoder to avoid slow startup when not needed."""
+        if self._model is None:
+            from sentence_transformers import CrossEncoder
+            logger.info("Loading cross-encoder/nli-deberta-v3-large for faithfulness scoring...")
+            self._model = CrossEncoder("cross-encoder/nli-deberta-v3-large")
+
+            # CRITICAL: verify the actual label order from the model's config,
+            # NLI checkpoints do NOT always use the same label ordering.
+            # Do not hardcode entailment_idx=1 without confirming here.
             try:
-                from transformers import pipeline
-                logger.info("Loading BART NLI model...")
-                # 'contradiction' (0), 'neutral' (1), 'entailment' (2)
-                self._pipe = pipeline("text-classification", model="facebook/bart-large-mnli")
-                logger.info("BART NLI loaded.")
-            except ImportError:
-                logger.error("transformers not installed. Cannot use BartNLIScorer.")
-                self._pipe = "UNAVAILABLE"
-        return self._pipe
+                id2label = self._model.model.config.id2label
+                logger.info(f"DeBERTa NLI id2label: {id2label}")
+                # Find the index whose label contains 'entail' (case-insensitive)
+                for idx, label in id2label.items():
+                    if "entail" in label.lower():
+                        self._entailment_idx = int(idx)
+                        break
+                logger.info(
+                    f"DeBERTa-v3 NLI loaded. "
+                    f"Confirmed entailment_idx={self._entailment_idx} "
+                    f"(label='{id2label.get(self._entailment_idx, 'unknown')}')"
+                )
+            except Exception as e:
+                logger.warning(
+                    f"Could not read id2label from DeBERTa config ({e}). "
+                    f"Defaulting entailment_idx=1. Verify this is correct for this checkpoint."
+                )
+        return self._model
 
-    def score(self, answer: str, context: str) -> FaithfulnessResult:
-        if not answer or not context or self.pipe == "UNAVAILABLE":
-            return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
-            
+    def score(self, answer: str, context_documents: List[str]) -> FaithfulnessResult:
+        """
+        Score faithfulness of an answer against a list of individual retrieved chunks.
+
+        Args:
+            answer: LLM-generated answer text.
+            context_documents: List of individual chunk texts (NOT concatenated).
+                               Per-document max in Eq. 6 requires individual documents.
+
+        Returns:
+            FaithfulnessResult with per-sentence entailment-based score.
+        """
         import re
-        sentences = [s.strip() for s in re.split(r'(?<=[.!?])\s+', answer) if len(s.strip()) > 20][:10]
+        import numpy as np
+
+        if not answer or not context_documents:
+            return FaithfulnessResult(
+                score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[]
+            )
+
+        # Split answer into sentences (Eq. 6 operates per-claim / per-sentence)
+        sentences = [
+            s.strip()
+            for s in re.split(r'(?<=[.!?])\s+', answer)
+            if len(s.strip()) > 15
+        ][:10]  # cap at 10 sentences for latency
+
         if not sentences:
-            return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
-            
+            return FaithfulnessResult(
+                score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[]
+            )
+
         supported = 0
-        unsupported = []
-        
+        unsupported_sentences = []
+
         try:
             for sentence in sentences:
-                # Format for bart-large-mnli: premise </s></s> hypothesis
-                input_text = f"{context[:3000]} </s></s> {sentence}"
-                result = self.pipe(input_text, truncation=True, max_length=1024)[0]
-                label = result['label'].lower()
-                # Consider neutral or entailment as supported for broad QA, or strict entailment
-                if 'entail' in label:
+                # Build (premise=doc, hypothesis=sentence) pairs for all docs
+                pairs = [(doc[:2000], sentence) for doc in context_documents]
+
+                # predict() returns raw logits shape: (n_docs, n_labels)
+                raw_scores = self.model.predict(pairs)
+
+                # Apply softmax to get probabilities
+                probs = self._softmax(raw_scores)
+
+                # Eq. 6: max entailment probability across all retrieved documents
+                max_entailment = float(np.max(probs[:, self._entailment_idx]))
+
+                if max_entailment > self.TAU_ENT:
                     supported += 1
                 else:
-                    unsupported.append(sentence)
+                    unsupported_sentences.append(sentence)
+
         except Exception as e:
-            logger.error(f"BART NLI Error: {e}")
-            return FaithfulnessResult(score=0.5, total_claims=len(sentences), supported_claims=0, unsupported_claims=sentences[:5])
+            logger.error(f"DebertaNLIScorer inference failed: {e}")
+            return FaithfulnessResult(
+                score=0.5,
+                total_claims=len(sentences),
+                supported_claims=0,
+                unsupported_claims=sentences[:5],
+            )
 
+        # Eq. 7: faithfulness = fraction of supported claims
         score = supported / len(sentences)
-        logger.info(f"BART NLI Faithfulness: {supported}/{len(sentences)} claims supported ({score*100:.0f}%)")
+        logger.info(
+            f"DeBERTa NLI Faithfulness: {supported}/{len(sentences)} sentences "
+            f"supported (score={score:.3f}, TAU_ENT={self.TAU_ENT})"
+        )
+
         return FaithfulnessResult(
             score=round(score, 3),
             total_claims=len(sentences),
             supported_claims=supported,
-            unsupported_claims=unsupported[:5]
+            unsupported_claims=unsupported_sentences[:5],
         )
 
+    @staticmethod
+    def _softmax(x):
+        """Numerically stable softmax over last axis."""
+        import numpy as np
+        e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
+        return e_x / e_x.sum(axis=-1, keepdims=True)
+
 
 class FActScoreCompat:
     """
@@ -287,3 +363,4 @@ class FActScoreCompat:
         result = self._scorer.score(answer, context)
         logger.info(f"FActScore: {result.score * 100:.1f}% ({result.supported_claims}/{result.total_claims} facts)")
         return result
+

backend/utils/groq_client.py

@@ -9,7 +9,7 @@ from typing import Optional, List, Dict, Any
 from groq import Groq
 from dotenv import load_dotenv
 
-# Load environment variables
+# Load environment variables (.env) from root/backend
 load_dotenv()
 
 
@@ -34,8 +34,9 @@ class GroqClient:
         
         from core.config import settings
         
-        # Model configurations based on LLM_BACKBONE
-        backbone = getattr(settings, "LLM_BACKBONE", "llama3").lower()
+        # Model configurations based on LLM_BACKBONE and environment settings
+        backbone = getattr(settings, "LLM_BACKBONE", "openai/gpt-oss-120b").lower()
+        active_model = getattr(settings, "GROQ_MODEL", os.getenv("GROQ_MODEL", "openai/gpt-oss-120b"))
         
         if backbone == "mixtral":
             self.models = {
@@ -53,13 +54,21 @@ class GroqClient:
                 "vision": "meta-llama/llama-4-scout-17b-16e-instruct",
                 "whisper": "whisper-large-v3"
             }
+        elif backbone == "llama3":
+            self.models = {
+                "chat": "llama-3.1-8b-instant",
+                "advanced": "llama-3.3-70b-versatile",
+                "fast": "llama-3.1-8b-instant",
+                "vision": "meta-llama/llama-4-scout-17b-16e-instruct",
+                "whisper": "whisper-large-v3"
+            }
         else:
             self.models = {
-                "chat": "llama-3.1-8b-instant",  # Primary LLM (fast & conversational)
-                "advanced": "llama-3.3-70b-versatile",  # Advanced reasoning
-                "fast": "llama-3.1-8b-instant",      # Fast responses
-                "vision": "meta-llama/llama-4-scout-17b-16e-instruct",  # Llama 4 Vision model (Jan 2025)
-                "whisper": "whisper-large-v3"        # Audio transcription
+                "chat": active_model,       # Primary LLM (openai/gpt-oss-120b)
+                "advanced": active_model,   # Advanced reasoning
+                "fast": active_model,       # Fast responses
+                "vision": "meta-llama/llama-4-scout-17b-16e-instruct",
+                "whisper": "whisper-large-v3"
             }
     
     def chat_completion(

pyrightconfig.json

@@ -0,0 +1,11 @@
+{
+  "include": [
+    "backend"
+  ],
+  "extraPaths": [
+    "C:/Users/Vasu/AppData/Local/Programs/Python/Python312/Lib/site-packages",
+    "d:/DG Workspace/Mexar-main/backend"
+  ],
+  "pythonVersion": "3.12",
+  "pythonPlatform": "Windows"
+}

test_data/financial_real/README.md

@@ -0,0 +1,16 @@
+# MEXAR Financial Corpus — Real Data Directory
+
+This directory is populated by running:
+
+```bash
+export SEC_USER_AGENT="YourName your@email.com"
+python backend/scripts/fetch_secedgar.py
+```
+
+**Contents (after fetch):**
+- `<CIK>_<accession>.txt` — one plain-text file per 10-K section (MD&A or Risk Factors)
+- `manifest.json` — CIK → accession → company → term → source URL mapping
+
+**Target:** ~130 documents across 3 search terms (risk factors, MD&A, GAAP/IFRS standards)
+
+See `MEXAR_Phase2_Dataset_Collection.md` for full instructions.

test_data/legal_real/README.md

@@ -0,0 +1,18 @@
+# MEXAR Legal Corpus — Real Data Directory
+
+This directory is populated by running:
+
+```bash
+export COURTLISTENER_TOKEN=your_token_here
+python backend/scripts/fetch_courtlistener.py
+```
+
+Get a free CourtListener API token at: https://www.courtlistener.com/sign-in/
+
+**Contents (after fetch):**
+- `case_<id>.txt` — one plain-text file per CourtListener opinion
+- `manifest.json` — opinion ID → case name → subdomain → court → source URL mapping
+
+**Target:** ~150 documents across 3 subdomains (contract law, intellectual property, corporate governance)
+
+See `MEXAR_Phase2_Dataset_Collection.md` for full instructions.

test_data/medical_real/README.md

@@ -0,0 +1,15 @@
+# MEXAR Medical Corpus — Real Data Directory
+
+This directory is populated by running:
+
+```bash
+python backend/scripts/fetch_pubmed.py --email your_email@example.com
+```
+
+**Contents (after fetch):**
+- `PMC<ID>.txt` — one plain-text file per PubMed Central OA article
+- `manifest.json` — PMCID → title → subdomain → source URL mapping
+
+**Target:** ~220 documents across 3 subdomains (cardiology, oncology, internal medicine)
+
+See `MEXAR_Phase2_Dataset_Collection.md` for full instructions.

test_data/query_sets/financial_queries.json

@@ -0,0 +1,250 @@
+[
+  {
+    "query": "What risk factors are typically disclosed in a technology company's 10-K filing under SEC Regulation S-K Item 1A?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "How do companies disclose cybersecurity risks in their annual SEC filings following the 2023 SEC cybersecurity disclosure rules?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "What types of macroeconomic risks are commonly reported as material risk factors in 10-K filings?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain — interest rate, FX, inflation."
+  },
+  {
+    "query": "How do pharmaceutical companies describe clinical trial failure as a risk factor in their annual reports?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "What supply chain risk disclosures are typically included in manufacturing company 10-Ks?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "How do companies disclose geopolitical and sanctions-related risk factors in their SEC filings?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "What liquidity and capital resources risks are companies required to disclose in their 10-K filings?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "How do companies in the energy sector describe commodity price risk in their annual reports?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "What regulatory and compliance risks are disclosed by financial services companies in their 10-K filings?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "How do companies disclose risks related to customer concentration in their annual SEC reports?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Risk factors subdomain."
+  },
+  {
+    "query": "What is included in the Management Discussion and Analysis (MD&A) section of a 10-K, and what SEC rules govern it?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "How do companies describe their revenue recognition policies in the MD&A section of annual reports?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain — ASC 606 disclosures."
+  },
+  {
+    "query": "How is operating cash flow discussed in the MD&A, and what non-GAAP measures are typically explained there?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain — EBITDA, adjusted earnings."
+  },
+  {
+    "query": "What forward-looking statements are typically made in the MD&A section and what safe harbor protections apply?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "How do companies discuss goodwill impairment testing in their MD&A and related disclosures?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "What segment reporting disclosures are included in the MD&A of diversified companies?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "How does the MD&A section address year-over-year changes in gross margin for product companies?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "What disclosure is required in the MD&A regarding off-balance-sheet arrangements and contingent liabilities?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "How do companies disclose critical accounting estimates in the MD&A under SEC Staff Accounting Bulletin guidance?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "What working capital and liquidity disclosures are expected in the MD&A of a growth-stage company?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "MD&A subdomain."
+  },
+  {
+    "query": "What are the key differences between GAAP and IFRS in the treatment of lease accounting under ASC 842 vs IFRS 16?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Accounting standards subdomain."
+  },
+  {
+    "query": "How does IFRS 9 differ from ASC 326 (CECL) in the recognition of expected credit losses?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Accounting standards subdomain."
+  },
+  {
+    "query": "What are the disclosure requirements under GAAP ASC 820 for fair value measurements at Level 3?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Accounting standards subdomain."
+  },
+  {
+    "query": "How does IFRS 15 govern revenue recognition for multi-element software licensing arrangements?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Accounting standards subdomain."
+  },
+  {
+    "query": "What are the required disclosures for pension and post-retirement benefit obligations under ASC 715?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Accounting standards subdomain."
+  },
+  {
+    "query": "How does GAAP treat research and development costs compared to IFRS, and what disclosures are required?",
+    "domain": "financial",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Accounting standards subdomain."
+  },
+  {
+    "query": "What are the first-line treatment guidelines for stage 3 non-small cell lung cancer?",
+    "domain": "financial",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: medical/oncology query posed to financial corpus — should trigger guardrail."
+  },
+  {
+    "query": "How is sepsis diagnosed and what is the initial antibiotic management protocol?",
+    "domain": "financial",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: medical query posed to financial corpus."
+  },
+  {
+    "query": "What are the elements of a valid breach of contract claim under common law?",
+    "domain": "financial",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: legal query posed to financial corpus."
+  },
+  {
+    "query": "How does the doctrine of fair use apply in copyright infringement cases involving parody works?",
+    "domain": "financial",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: intellectual property/legal query posed to financial corpus."
+  },
+  {
+    "query": "What are the ADA diagnostic thresholds for prediabetes and type 2 diabetes using fasting glucose and HbA1c?",
+    "domain": "financial",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: medical query posed to financial corpus."
+  }
+]

test_data/query_sets/legal_queries.json

@@ -0,0 +1,250 @@
+[
+  {
+    "query": "What remedies are available to a non-breaching party when the other party materially breaches a contract?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "How do courts determine whether consequential damages are recoverable for breach of contract under the Hadley v Baxendale rule?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "What is the legal standard for determining whether a contract has been formed by conduct rather than by written agreement?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain — implied-in-fact contracts."
+  },
+  {
+    "query": "Under the UCC Article 2, what are the implied warranties that attach to the sale of goods?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain — implied warranty of merchantability and fitness."
+  },
+  {
+    "query": "How does the doctrine of promissory estoppel differ from consideration as a basis for enforcing a promise?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "What factors do courts weigh when deciding whether to grant specific performance as a remedy for breach of contract?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "How does the parol evidence rule limit the use of extrinsic evidence to interpret a written contract?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "What constitutes anticipatory repudiation of a contract and what rights does it give to the aggrieved party?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "How is the enforceability of a non-compete clause assessed by courts in employment contracts?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "Under force majeure clauses, what events typically excuse performance of a contract obligation?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Contract law subdomain."
+  },
+  {
+    "query": "What is the standard for proving patent infringement under the doctrine of equivalents?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "How does the likelihood of confusion test apply in trademark infringement cases?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain — Polaroid factors."
+  },
+  {
+    "query": "What is the fair use doctrine in copyright law and what are the four statutory factors courts apply?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "How do courts determine whether trade secrets qualify for protection under the Defend Trade Secrets Act?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "What is required to establish willful patent infringement and what enhanced damages can result?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "How does inter partes review (IPR) at the USPTO differ from district court patent litigation?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "What is the legal standard for trademark dilution claims under the Lanham Act?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain — dilution by blurring vs tarnishment."
+  },
+  {
+    "query": "How do courts evaluate obviousness in patent validity challenges under 35 U.S.C. § 103?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain — Graham v. John Deere factors."
+  },
+  {
+    "query": "What are the elements of a successful copyright infringement claim?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "How is secondary liability (contributory and vicarious infringement) established in patent cases?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Intellectual property subdomain."
+  },
+  {
+    "query": "What is the business judgment rule and when does it protect corporate directors from liability?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Corporate governance subdomain."
+  },
+  {
+    "query": "What are the elements of a shareholder derivative suit and what is the demand requirement?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Corporate governance subdomain."
+  },
+  {
+    "query": "How does the duty of loyalty differ from the duty of care for corporate officers and directors?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Corporate governance subdomain."
+  },
+  {
+    "query": "What legal standard applies to board approval of a merger under Delaware's entire fairness review?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Corporate governance subdomain."
+  },
+  {
+    "query": "What is the duty of disclosure (candor) owed by directors to shareholders in a merger transaction?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Corporate governance subdomain."
+  },
+  {
+    "query": "How do courts assess whether executive compensation decisions are protected by the business judgment rule?",
+    "domain": "legal",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Corporate governance subdomain."
+  },
+  {
+    "query": "What is the first-line pharmacotherapy for newly diagnosed type 2 diabetes mellitus?",
+    "domain": "legal",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: medical query posed to legal corpus — should trigger guardrail."
+  },
+  {
+    "query": "What is the recommended chemotherapy protocol for stage III non-small cell lung cancer?",
+    "domain": "legal",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: medical/oncology query posed to legal corpus."
+  },
+  {
+    "query": "How is EBITDA calculated and what does it measure in company financial performance?",
+    "domain": "legal",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: financial query posed to legal corpus."
+  },
+  {
+    "query": "What are the disclosure requirements under IFRS 9 for financial instruments at fair value through profit or loss?",
+    "domain": "legal",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: accounting/financial standards query posed to legal corpus."
+  },
+  {
+    "query": "What is the pathophysiology of heart failure with reduced ejection fraction?",
+    "domain": "legal",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: medical/cardiology query posed to legal corpus."
+  }
+]

test_data/query_sets/medical_queries.json

@@ -0,0 +1,282 @@
+[
+  {
+    "query": "What is the first-line treatment for stage 2 hypertension according to current guidelines?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain. Expected answer: combination antihypertensive therapy (ACE inhibitor/ARB + CCB or thiazide)."
+  },
+  {
+    "query": "What are the recommended beta-blocker choices for patients with heart failure with reduced ejection fraction (HFrEF)?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain."
+  },
+  {
+    "query": "What diagnostic criteria are used to classify atrial fibrillation subtypes?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain — paroxysmal, persistent, long-standing persistent, permanent."
+  },
+  {
+    "query": "What are the ESC or ACC/AHA guideline recommendations for statin therapy in secondary prevention after myocardial infarction?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain."
+  },
+  {
+    "query": "Which anticoagulants are recommended for stroke prevention in non-valvular atrial fibrillation?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain — DOACs vs warfarin."
+  },
+  {
+    "query": "What is the CHADS2-VASc score and how is it used to guide anticoagulation decisions?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain."
+  },
+  {
+    "query": "What is the current guideline-directed medical therapy for stable ischemic heart disease?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain."
+  },
+  {
+    "query": "How is cardiac resynchronization therapy indicated in heart failure patients?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain."
+  },
+  {
+    "query": "What are the diagnostic criteria for heart failure with preserved ejection fraction (HFpEF)?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain."
+  },
+  {
+    "query": "What is the recommended approach to treating ST-elevation myocardial infarction (STEMI) within the first 12 hours?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Cardiology subdomain — primary PCI vs thrombolysis."
+  },
+  {
+    "query": "What is the standard chemotherapy regimen for first-line treatment of metastatic non-small cell lung cancer?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "What are the NCCN guidelines for adjuvant chemotherapy in stage III colon cancer?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "How are PD-L1 expression levels used to select patients for pembrolizumab therapy in lung cancer?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "What treatment protocol is used for BRCA-mutated ovarian cancer in first-line maintenance?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain — PARP inhibitors."
+  },
+  {
+    "query": "What are the standard treatment options for relapsed/refractory diffuse large B-cell lymphoma?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "What are the first-line treatment options for hormone receptor-positive HER2-negative metastatic breast cancer?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "How does tumor mutational burden (TMB) influence immunotherapy selection in solid tumors?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "What is the role of neoadjuvant chemotherapy in locally advanced rectal cancer?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "What are current guidelines for surveillance imaging after complete remission in Hodgkin lymphoma?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "How are CDK4/6 inhibitors used in the treatment of advanced breast cancer?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Oncology subdomain."
+  },
+  {
+    "query": "What are the diagnostic criteria for type 2 diabetes mellitus according to ADA guidelines?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain — fasting glucose, HbA1c, OGTT thresholds."
+  },
+  {
+    "query": "What are the Rome IV diagnostic criteria for irritable bowel syndrome?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "How is community-acquired pneumonia severity classified using the CURB-65 or PSI scoring systems?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "What are the diagnostic criteria for systemic lupus erythematosus (SLE) per the 2019 EULAR/ACR classification?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "What is the initial workup for a patient presenting with unexplained iron deficiency anemia?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "What are the diagnostic criteria for chronic kidney disease (CKD) and how are GFR stages defined?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "How is the diagnosis of pulmonary embolism confirmed, and what anticoagulation is recommended?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "What are the WHO diagnostic criteria for metabolic syndrome?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "How is sepsis defined and what is the SOFA score used for?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain — Sepsis-3 definition."
+  },
+  {
+    "query": "What are the diagnostic criteria for rheumatoid arthritis according to the 2010 ACR/EULAR classification?",
+    "domain": "medical",
+    "is_in_domain": true,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Internal medicine subdomain."
+  },
+  {
+    "query": "What is the best therapeutic approach for a breach of contract involving consequential damages under common law?",
+    "domain": "medical",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: legal query posed to medical corpus — should trigger domain guardrail."
+  },
+  {
+    "query": "How does SEC Regulation S-K require companies to disclose material risk factors in annual reports?",
+    "domain": "medical",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: financial/regulatory query posed to medical corpus."
+  },
+  {
+    "query": "What is the amortization schedule for a 30-year fixed-rate mortgage at 6.5% interest?",
+    "domain": "medical",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: financial query posed to medical corpus."
+  },
+  {
+    "query": "Under the UCC, when does title to goods transfer from seller to buyer in a sale of goods contract?",
+    "domain": "medical",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: legal/UCC query posed to medical corpus."
+  },
+  {
+    "query": "What are the fiduciary duties of a corporate board of directors to shareholders under Delaware law?",
+    "domain": "medical",
+    "is_in_domain": false,
+    "expected_source_docs": [],
+    "annotator": "devrajsinh",
+    "notes": "Out-of-domain: corporate law query posed to medical corpus."
+  }
+]