nsr51324
/

CortexRAG

@@ -19,303 +19,56 @@ license: apache-2.0
 A retrieval-augmented generation (RAG) system for answering medical questions using a curated English medical Question-Answer knowledge base.
-The system combines:
-- Semantic retrieval using Sentence Transformers
-- FAISS vector search
-- Medical query expansion
-- Cross-Encoder reranking
-- Evidence deduplication
-- Confidence gating
-- LLM-based answer generation
-- Retrieval evaluation
-- Confidence-gate evaluation
-- Latency measurement
-> **Important:** This system is intended for research and educational purposes. It is not a medical diagnostic system and should not be used as a substitute for professional medical advice.
 ---
-# 1. Overview
-This project implements a complete Retrieval-Augmented Generation pipeline for medical Question-Answer data.
-Instead of asking an LLM to answer directly from its internal knowledge, the system first retrieves relevant medical evidence from a local knowledge base.
-The pipeline is:
 ```text
 User Question
       │
       ▼
-Query Expansion
       │
       ▼
 Sentence Transformer Embedding
       │
       ▼
-FAISS Vector Search
-      │
-      ▼
-Top-N Candidate Documents
-      │
-      ▼
-Cross-Encoder Reranking
       │
       ▼
-Top-K Evidence
       │
       ▼
 Near-Duplicate Removal
       │
       ▼
-Confidence Gate
-      │
-      ├── Reject → Insufficient Evidence
-      │
       ▼
 LLM Generation
       │
       ▼
-Evidence-Based Answer
-```
----
-# 2. Knowledge Base
-The knowledge base contains approximately:
-```text
-16,384 medical Question-Answer records
-```
-The dataset contains medical questions and corresponding answers, together with categorical information.
-Main columns used by the system:
-```text
-Question
-Answer
-Category
-doc_id
-```
-A unique `doc_id` is assigned to every document:
-```python
-df = df.reset_index(drop=True)
-df["doc_id"] = df.index
-```
-The `doc_id` is later used to identify evidence sources.
----
-# 3. Embedding Model
-The system uses:
-```text
-sentence-transformers/all-MiniLM-L6-v2
-```
-The model converts every medical question into a dense vector representation.
-Embeddings are normalized:
-```python
-question_embeddings = embedder.encode(
-    questions,
-    batch_size=64,
-    show_progress_bar=True,
-    convert_to_numpy=True,
-    normalize_embeddings=True,
-)
-```
-Normalization allows the FAISS Inner Product index to behave as cosine similarity search.
-The resulting embedding matrix is stored as:
-```text
-question_embeddings.npy
-```
----
-# 4. FAISS Retrieval
-FAISS is used for efficient vector similarity search.
-The index is created using:
-```python
-faiss.IndexFlatIP(embedding_dim)
-```
-Because the embeddings are normalized, Inner Product corresponds to cosine similarity.
-The FAISS index contains:
-```text
-16,384 vectors
-```
-The index is saved as:
-```text
-questions.index
-```
-Retrieval works by:
-1. Embedding the user query
-2. Searching the FAISS index
-3. Returning the top-N most similar questions
-Default retrieval configuration:
-```python
-RETRIEVE_TOP_N = 20
-```
----
-# 5. Query Expansion
-The system includes a lightweight medical query expansion layer.
-This helps bridge the gap between everyday language and medical terminology.
-Examples:
-```text
-underactive thyroid
-        ↓
-hypothyroidism
-overactive thyroid
-        ↓
-hyperthyroidism
-high blood sugar
-        ↓
-hyperglycemia
-low blood sugar
-        ↓
-hypoglycemia
-sugar disease
-        ↓
-diabetes
-```
-For example:
-```text
-I suffer from an underactive thyroid gland
-```
-becomes:
-```text
-I suffer from an underactive thyroid gland
-(hypothyroidism)
-```
-The original query is not replaced.
-The medical term is simply appended before generating the embedding.
----
-# 6. Cross-Encoder Reranking
-After FAISS retrieves the initial candidates, a Cross-Encoder is used to rerank them.
-Model:
-```text
-cross-encoder/ms-marco-MiniLM-L-6-v2
-```
-The system retrieves:
-```text
-Top 20
-```
-candidates and reranks them to keep:
-```text
-Top 6
-```
-Configuration:
-```python
-RETRIEVE_TOP_N = 20
-RERANK_TOP_K = 6
-```
-The reranker scores:
-```text
-(query, Answer)
-```
-rather than:
-```text
-(query, Question)
-```
-This is important because the dataset contains multiple records with the same Question but different medical answers.
-Reranking the Answer therefore helps determine which evidence is actually useful for answering the user's question.
----
-# 7. Evidence Deduplication
-Medical datasets can contain multiple answers that are almost identical.
-The system removes near-duplicate answers using:
-```python
-difflib.SequenceMatcher
 ```
-with:
-```python
-threshold = 0.92
-```
-This prevents the LLM from receiving multiple copies of essentially the same evidence.
-Each evidence item contains:
-```text
-doc_id
-question
-answer
-category
-similarity
-rerank_score
-```
----
-# 8. Confidence Gate
-The system does not automatically answer every question.
-Before sending evidence to the LLM, a confidence gate checks whether the retrieved evidence is strong enough.
-Current configuration:
 ```python
 MIN_RERANK_SCORE = -8
@@ -324,873 +77,73 @@ MIN_SUPPORT_COUNT = 2
 SUPPORT_SCORE = -5.0
 ```
-The system accepts the evidence only if:
 1. At least one result exists.
-2. The highest rerank score is above the minimum threshold.
-3. The similarity score is above the minimum similarity floor.
-4. At least two retrieved documents provide sufficient support.
-Otherwise, the system refuses to generate an evidence-based answer.
-Example:
-```text
-Question:
-Why do I feel short of breath and tired when I climb stairs?
-Result:
-No sufficiently relevant evidence.
-System:
-I couldn't find sufficiently relevant medical evidence
-in the knowledge base to answer this question.
-```
-This behavior is important for reducing unsupported answers.
----
-# 9. LLM Generation
-The generation stage uses a Groq-hosted LLM.
-Current model used during development:
-```text
-openai/gpt-oss-20b
-```
-The LLM is explicitly instructed to operate in RAG mode.
-Main generation rules:
-```text
-1. Use only retrieved evidence.
-2. Do not add outside medical knowledge.
-3. Do not guess.
-4. Cite evidence using [doc_id].
-5. Mention when evidence is insufficient.
-6. Do not diagnose users.
-7. Do not prescribe personalized treatment or dosages.
-8. Mention disagreements between sources.
-9. Keep answers concise and organized.
-10. Include a medical-information disclaimer.
 ```
-The API key is NOT stored in this repository.
-It should be provided through an environment variable:
 ```python
 import os
-GROQ_API_KEY = os.environ["GROQ_API_KEY"]
-```
----
-# 10. Example
-## Input
-```text
-What signs might suggest that my thyroid is not producing enough hormones?
-```
-## Retrieved Evidence
-```text
-[15967] Thyroid symptoms
-[13504] Thyroid symptoms
-[10676] Thyroid symptoms
-[13989] What are the symptoms of thyroid disease?
-[13559] Symptoms that indicate a disorder in the thyroid gland
-[12173] How do I know that I have a thyroid problem?
-```
-## Generated Answer
-The system generated an evidence-based response describing symptoms associated with low thyroid hormone production, including:
-- Fatigue
-- Feeling cold
-- Weight gain
-- Constipation
-- Menstrual changes
-- Hair loss
-The final response also included the required medical disclaimer.
----
-# 11. Example of Out-of-Domain Query
-## Input
-```text
-What is the best treatment for a broken leg?
-```
-The retriever returned some medically related documents, mostly concerning diabetic foot injuries.
-However, those documents did not contain appropriate evidence about treating fractures.
-The final system response therefore stated that the available evidence did not contain information about treating a broken leg.
-This demonstrates an important property of the system:
-> Retrieval similarity alone does not guarantee that the retrieved evidence is appropriate for the question.
----
-# 12. Evaluation
-The system was evaluated using multiple complementary evaluation methods.
-These include:
-- Self-retrieval accuracy
-- Recall@K
-- Precision@K
-- F1@K
-- MRR
-- NDCG@K
-- Category sanity check
-- Confidence-gate accuracy
-- Cross-validation threshold tuning
-- Latency evaluation
----
-# 13. Self-Retrieval Evaluation
-A sample of 50 questions was tested.
-Result:
-```text
-Self-retrieval Top-1 Accuracy:
-50 / 50
-= 100%
-```
-However, this metric should NOT be interpreted as proof that the system has 100% real-world retrieval accuracy.
-It mainly verifies that the embedding/index pipeline can retrieve the original record when the exact same question is used.
----
-# 14. Retrieval Evaluation
-The retrieval evaluation used questions that have duplicate Question entries in the dataset.
-This provides multiple possible relevant answers for the same question.
-```text
-Questions with duplicate entries usable for evaluation:
-67
-```
-## Retrieval Only
-```text
-Recall@1     = 0.690
-Precision@1  = 1.000
-F1@1         = 0.745
-Recall@3     = 0.851
-Precision@3  = 0.560
-F1@3         = 0.563
-Recall@5     = 0.899
-Precision@5  = 0.423
-F1@5         = 0.462
-Recall@10    = 0.959
-Precision@10 = 0.286
-F1@10        = 0.349
-MRR          = 1.000
-```
-Number of evaluated queries:
-```text
-150
-```
----
-# 15. Retrieval + Cross-Encoder Evaluation
-The same evaluation was also performed after Cross-Encoder reranking.
-Results:
-```text
-Recall@1     = 0.083
-Precision@1  = 0.113
-F1@1         = 0.093
-Recall@3     = 0.157
-Precision@3  = 0.087
-F1@3         = 0.100
-Recall@5     = 0.222
-Precision@5  = 0.121
-F1@5         = 0.120
-Recall@10    = 0.388
-Precision@10 = 0.135
-F1@10        = 0.153
-MRR          = 0.237
-```
-Number of evaluated queries:
-```text
-150
-```
----
-# 16. Important Evaluation Observation
-The evaluation shows that the Cross-Encoder performed significantly worse under this particular retrieval benchmark.
-This is an important finding rather than something to hide.
-The current reranker was trained for general passage relevance using:
-```text
-cross-encoder/ms-marco-MiniLM-L-6-v2
 ```
-while this evaluation is based on medical Question-Answer records.
-Therefore, the Cross-Encoder may not rank the medical answers in the same way as the manually defined relevance criteria.
-This suggests that the reranking component requires further investigation before being considered production-ready.
-Possible future improvements include:
-- Medical-domain reranker
-- Fine-tuning a Cross-Encoder on medical relevance pairs
-- Using a manually reviewed gold set
-- Better relevance labeling
-- Evaluating reranking separately from retrieval
-- Testing alternative reranking models
----
-# 17. Category Sanity Check
-A category-based sanity check was also performed.
-The goal is to determine whether retrieved documents contain at least one document belonging to the same category as the query's source document.
-## Retrieval + Reranker
-```text
-Category Recall = 0.98
-n = 100
-```
-## Retrieval Only
-```text
-Category Recall = 0.97
-n = 100
-```
-This indicates that the retrieval system generally retrieves documents from the expected medical category.
-However:
-> Category matching is only a supporting diagnostic metric and is NOT treated as the final ground truth for relevance.
----
-# 18. Confidence Gate Evaluation
-A manually constructed evaluation set contained:
-```text
-25 in-domain questions
-25 out-of-domain questions
-```
-Total:
-```text
-50 questions
-```
-The system achieved:
-```text
-Gate Decision Accuracy = 88%
-```
-The system correctly accepted most in-domain questions and rejected most out-of-domain questions.
-However, several false positives were observed.
-Examples included questions about:
-```text
-broken legs
-kidney stones
-sprained ankles
-broken arms
-food poisoning
-```
-These questions sometimes retrieved medically related evidence even though the knowledge base did not contain appropriate evidence.
-This demonstrates why confidence gating and stronger relevance evaluation are necessary.
----
-# 19. Confidence Score Distribution
-For the confidence-gate evaluation:
-### Expected answer = True
-```text
-Mean score = 5.539
-Minimum    = -8.938
-Maximum    = 9.321
-```
-### Expected answer = False
-```text
-Mean score = -6.167
-Minimum    = -11.062
-Maximum    = 1.532
-```
-There is overlap between the two distributions.
-Therefore, a single rerank-score threshold cannot perfectly separate valid and invalid questions.
----
-# 20. Threshold Tuning
-The confidence threshold was evaluated using 5-fold cross-validation.
-Instead of selecting a threshold from one train/test split, the data is divided into five different folds.
-For every fold:
-1. A threshold is optimized on the training portion.
-2. The threshold is evaluated on the held-out fold.
-3. The threshold and test accuracy are recorded.
-4. Mean and standard deviation are calculated.
-This helps determine whether the selected threshold is stable or highly dependent on a small evaluation sample.
-A large threshold standard deviation indicates that the evaluation set is too small or unstable and should be expanded before selecting a production threshold.
----
-# 21. Gold Set
-A semi-automatic gold-set construction process was implemented.
-The process:
-```text
-100 sampled questions
-        │
-        ▼
-Exact duplicate questions
-        │
-        ▼
-Semantic candidates with similarity >= 0.90
-        │
-        ▼
-Manual review
-        │
-        ▼
-Relevant = 1
-Not relevant = 0
-        │
-        ▼
-Final Gold Set
-```
-The initial run generated:
-```text
-100 questions
-16 candidate pairs
-3 questions with semantic candidates
-```
-The generated file:
-```text
-gold_set_for_manual_review.xlsx
-```
-contains candidate pairs that should be manually reviewed.
-For each candidate, the reviewer should enter:
-```text
-1 = relevant
-0 = not relevant
-```
-Only manually approved candidates should be added to the final relevant evidence set.
----
-# 22. Gold Set Evaluation Metrics
-Once the manually reviewed gold set is completed, the system evaluates:
-```text
-Recall@5
-MRR
-NDCG@5
-```
-The evaluation compares:
-```text
-Retriever only
-```
-against:
-```text
-Retriever + Cross-Encoder Reranker
-```
-The manually reviewed gold set is preferred over category-based evaluation because it provides explicit relevance judgments.
----
-# 23. Latency Evaluation
-The complete RAG pipeline was also measured for response latency.
-Example result:
-```text
-Retrieval       = 0.016 sec
-Reranking       = 0.094 sec
-Evidence build  = 0.025 sec
-LLM generation  = 0.858 sec
-Total           = 0.993 sec
-```
-Approximate breakdown:
-```text
-Retrieval       █
-Reranking       █████
-Evidence        ██
-LLM             █████████████████��███████████████████
-```
-The LLM generation stage is the largest contributor to total latency.
-The measured end-to-end latency in this test was approximately:
-```text
-0.99 seconds
-```
-This should not be interpreted as a guaranteed production latency because API/network conditions, hardware, load, and model availability can change.
----
-# 24. Saved Artifacts
-The system uses the following main artifacts:
-```text
-questions.index
-question_embeddings.npy
-README.md
-gold_set_for_manual_review.xlsx
-```
-## questions.index
-FAISS vector index containing the question embeddings.
-## question_embeddings.npy
-NumPy array containing the normalized question embeddings.
-## gold_set_for_manual_review.xlsx
-Human-review file used to create the manually verified relevance set.
----
-# 25. Local Deployment
-The RAG system can be deployed locally as a Python service.
-Recommended architecture:
-```text
-Frontend
-   │
-   ▼
-Backend API
-   │
-   ▼
-Medical RAG Pipeline
-   │
-   ├── FAISS
-   ├── Sentence Transformer
-   ├── Cross Encoder
-   └── LLM API
-```
-The application should load the embedding model and FAISS index once when the service starts.
-They should NOT be reloaded for every user request.
-Example:
-```python
-from sentence_transformers import SentenceTransformer, CrossEncoder
-import faiss
-import numpy as np
-embedder = SentenceTransformer(
-    "sentence-transformers/all-MiniLM-L6-v2"
-)
-reranker = CrossEncoder(
-    "cross-encoder/ms-marco-MiniLM-L-6-v2"
-)
-faiss_index = faiss.read_index(
-    "questions.index"
-)
-question_embeddings = np.load(
-    "question_embeddings.npy"
-)
-```
----
-# 26. Example API Architecture
-A backend can expose an endpoint such as:
-```http
-POST /api/rag/query
-```
-Request:
-```json
-{
-  "question": "What are the symptoms of diabetes?"
-}
-```
-Response:
-```json
-{
-  "answer": "...",
-  "evidence": [
-    {
-      "doc_id": 4793,
-      "similarity": 0.91,
-      "rerank_score": 8.21
-    }
-  ]
-}
-```
-This makes the RAG pipeline independent from the frontend.
----
-# 27. Recommended Project Structure
-```text
-medical-rag/
-│
-├── README.md
-│
-├── artifacts/
-│   ├── questions.index
-│   └── question_embeddings.npy
-│
-├── src/
-│   ├── retrieval.py
-│   ├── reranker.py
-│   ├── query_expansion.py
-│   ├── evidence.py
-│   ├── confidence_gate.py
-│   ├── generation.py
-│   └── rag_pipeline.py
-│
-├── api/
-│   └── app.py
-│
-├── evaluation/
-│   ├── retrieval_evaluation.py
-│   ├── gate_evaluation.py
-│   └── gold_set_evaluation.py
-│
-├── data/
-│   └── gold_set_for_manual_review.xlsx
-│
-└── requirements.txt
-```
----
-# 28. Installation
-Install the main dependencies:
-```bash
-pip install sentence-transformers faiss-cpu numpy pandas openpyxl
-```
-For Groq-based generation:
-```bash
-pip install groq
-```
----
-# 29. Environment Variables
-Never hard-code API keys inside source code.
-Set:
-```bash
-GROQ_API_KEY=your_api_key_here
-```
-Python:
-```python
-import os
-api_key = os.environ["GROQ_API_KEY"]
-```
----
-# 30. Security
-The API key used during development must not be committed to GitHub or Hugging Face.
-Recommended practice:
-```text
-.env
-```
-and:
-```text
-.gitignore
-```
-Example `.gitignore`:
-```text
-.env
-__pycache__/
-*.pyc
-.ipynb_checkpoints/
-```
-If an API key has already been exposed publicly, revoke it and generate a new one.
----
-# 31. Limitations
-This system has several important limitations.
-### 1. Medical domain limitations
-The knowledge base does not necessarily cover every medical condition or clinical scenario.
-### 2. Retrieval limitations
-A semantically similar document is not necessarily an appropriate answer.
-### 3. Reranker limitations
-The current Cross-Encoder was not specifically trained on this medical dataset.
-### 4. Gold-set limitations
-The manually reviewed gold set is still small.
-### 5. Evaluation limitations
-Some evaluation metrics rely on duplicate Question entries rather than a fully manually annotated relevance dataset.
-### 6. LLM limitations
-The generated answer depends on the quality of the retrieved evidence.
-### 7. Clinical safety
-The system should not be used for diagnosis, emergency decisions, prescribing medication, or personalized treatment.
----
-# 32. Future Improvements
-Recommended next steps:
-```text
-1. Complete the manual gold-set annotation.
-2. Increase the number of manually reviewed questions.
-3. Train or evaluate a medical-domain reranker.
-4. Re-evaluate the Cross-Encoder using the manual gold set.
-5. Tune the confidence gate using a larger validation set.
-6. Add citation/source metadata.
-7. Add automated regression tests.
-8. Add monitoring for retrieval failures.
-9. Add API-level authentication and rate limiting.
-10. Deploy the RAG pipeline behind a REST API.
-11. Add a frontend chat interface.
-12. Evaluate hallucination/faithfulness separately from retrieval quality.
-```
----
-# 33. Evaluation Summary
-| Metric | Result |
-|---|---:|
-| Knowledge Base Size | 16,384 records |
-| Self-Retrieval Top-1 | 100% |
-| Retrieval Recall@1 | 69.0% |
-| Retrieval Recall@3 | 85.1% |
-| Retrieval Recall@5 | 89.9% |
-| Retrieval Recall@10 | 95.9% |
-| Retrieval Precision@1 | 100% |
-| Retrieval F1@1 | 74.5% |
-| Retrieval MRR | 1.000 |
-| Category Recall + Reranker | 98% |
-| Category Recall Retrieval Only | 97% |
-| Confidence Gate Accuracy | 88% |
-| Example End-to-End Latency | 0.993 sec |
-> The evaluation numbers above are specific to the current experimental setup and should not be interpreted as general medical accuracy.
----
-# 34. Responsible Use
-This project provides general medical information retrieval and generation.
-It is not:
-- A diagnostic tool
-- A clinical decision-support system
-- A replacement for a physician
-- A prescription system
-- An emergency medical service
-Users should consult qualified healthcare professionals for medical decisions.
----
-# 35. Citation and References
-### Sentence Transformers
-https://www.sbert.net/
-### FAISS
-https://github.com/facebookresearch/faiss
-### Cross-Encoder
-https://www.sbert.net/examples/applications/cross-encoder/README.html
-### Hugging Face Model Cards
-https://huggingface.co/docs/hub/en/model-cards
-### Hugging Face Model Release Checklist
-https://huggingface.co/docs/hub/en/model-release-checklist
----
-# 36. Project Status
-```text
-Status: Research / Prototype
-Retrieval: Implemented
-FAISS: Implemented
-Query Expansion: Implemented
-Cross-Encoder Reranking: Implemented
-Evidence Deduplication: Implemented
-Confidence Gate: Implemented
-LLM Generation: Implemented
-Evaluation: Implemented
-Latency Measurement: Implemented
-Manual Gold Set: In Progress
-Production Deployment: Future Work
-```
----
-# 37. Final Note
-The main objective of this project is not simply to generate medical answers.
-The goal is to build a RAG system that:
-```text
-Retrieves evidence
-       ↓
-Ranks evidence
-       ↓
-Checks confidence
-       ↓
-Generates from evidence
-       ↓
-Provides traceable document IDs
-```
-This design allows the system to be evaluated as a retrieval pipeline independently from the LLM generation layer.

 A retrieval-augmented generation (RAG) system for answering medical questions using a curated English medical Question-Answer knowledge base.
+The system combines semantic retrieval (Sentence Transformers + FAISS), medical query expansion, Cross-Encoder reranking, evidence deduplication, confidence gating, and LLM-based answer generation.
+> **Important:** This system is for research and educational purposes only. It is not a medical diagnostic system and should not replace professional medical advice.
 ---
+## 1. Pipeline
 ```text
 User Question
       │
       ▼
+Query Expansion            (lay terms → medical terms)
       │
       ▼
 Sentence Transformer Embedding
       │
       ▼
+FAISS Vector Search        (Top 20 candidates)
       │
       ▼
+Cross-Encoder Reranking    (Top 6 evidence)
       │
       ▼
 Near-Duplicate Removal
       │
       ▼
+Confidence Gate  ──reject──▶  "Insufficient evidence"
+      │ pass
       ▼
 LLM Generation
       │
       ▼
+Evidence-Based Answer (with doc_id citations)
 ```
+## 2. Knowledge Base
+~16,384 medical Question-Answer records with columns `Question`, `Answer`, `Category`, `doc_id`.
+## 3. Retrieval
+- **Embedding model:** `sentence-transformers/all-MiniLM-L6-v2`, embeddings normalized so FAISS `IndexFlatIP` behaves as cosine similarity.
+- **Query Expansion:** a small lay-term → medical-term dictionary is appended to the query before embedding (e.g. `underactive thyroid → hypothyroidism`, `high blood sugar → hyperglycemia`). The original query is never replaced, only extended.
+- **Reranking:** `cross-encoder/ms-marco-MiniLM-L-6-v2` scores `(query, Answer)` — not `(query, Question)` — since many records share the same question with different answers; this identifies which *evidence* is actually useful.
+- **Evidence deduplication:** near-identical answers are collapsed via `difflib.SequenceMatcher` (threshold 0.92).
+## 4. Confidence Gate
+Evidence is only sent to the LLM if **all** conditions hold:
 ```python
 MIN_RERANK_SCORE = -8
 SUPPORT_SCORE = -5.0
 ```
 1. At least one result exists.
+2. Top rerank score ≥ `MIN_RERANK_SCORE`.
+3. Top similarity ≥ `MIN_SIMILARITY_FLOOR`.
+4. At least `MIN_SUPPORT_COUNT` results score ≥ `SUPPORT_SCORE`.
+Otherwise, the system explicitly refuses to answer rather than guessing.
+## 5. Generation
+LLM: Groq-hosted `openai/gpt-oss-20b`, instructed to answer **only** from retrieved evidence, cite sources by `[doc_id]`, flag disagreement between sources, avoid diagnosis/prescriptions, and include a medical disclaimer. API key is read from `GROQ_API_KEY` (never hard-coded).
+## 6. Example
+**Q:** *What signs might suggest that my thyroid is not producing enough hormones?*
+**Evidence:** 6 thyroid-related records retrieved and reranked.
+**Answer:** Evidence-based summary (fatigue, feeling cold, weight gain, constipation, menstrual changes, hair loss) + disclaimer.
+**Out-of-domain example — Q:** *What is the best treatment for a broken leg?*
+Retrieval returned diabetic-foot documents (lexically similar), but the gate/LLM correctly identified them as inappropriate evidence and refused to answer — demonstrating that similarity alone doesn't guarantee relevance.
+## 7. Evaluation Summary
+| Metric | Result |
+|---|---:|
+| Knowledge Base Size | 16,384 records |
+| Self-Retrieval Top-1 | 100% |
+| Retrieval Recall@1 / @5 / @10 | 69.0% / 89.9% / 95.9% |
+| Retrieval MRR | 1.000 |
+| Retrieval + Reranker Recall@1 / @5 / @10 | 8.3% / 22.2% / 38.8% |
+| Retrieval + Reranker MRR | 0.237 |
+| Category Recall (retrieval / +reranker) | 97% / 98% |
+| Confidence Gate Accuracy | 88% (n=50) |
+| Example End-to-End Latency | 0.99 sec |
+**Key finding:** the general-purpose Cross-Encoder scores notably *worse* than plain retrieval on this duplicate-question-based benchmark. This isn't hidden — it indicates the reranker (trained for general passage relevance) doesn't align well with medical relevance judgments, and needs validation against a manually reviewed gold set before being trusted in production. A semi-automatic gold-set workflow (exact duplicates + semantic candidates ≥0.90 similarity, human-reviewed 1/0 labels) is included for this purpose; Category-match is used only as a secondary sanity check, not ground truth.
+The confidence-gate threshold was tuned via 5-fold cross-validation rather than a single split, to check stability rather than overfit to one small sample.
+## 8. Limitations
+- Knowledge base doesn't cover every condition/scenario.
+- Semantic similarity ≠ appropriate evidence (see broken-leg example).
+- Reranker not fine-tuned on this medical domain yet.
+- Manual gold set is still small — evaluation should be treated as preliminary.
+- Not for diagnosis, emergencies, prescriptions, or personalized treatment.
+## 9. Installation
+```bash
+pip install sentence-transformers faiss-cpu numpy pandas openpyxl groq
 ```
 ```python
 import os
+GROQ_API_KEY = os.environ["GROQ_API_KEY"]  # never hard-code keys
 ```
+## 10. Future Work
+Complete manual gold-set annotation → re-evaluate reranker against it → consider a medical-domain reranker → expand confidence-gate validation set → add citations/monitoring/auth → deploy behind a REST API → evaluate faithfulness/hallucination separately from retrieval quality.
+## 11. Responsible Use
+Not a diagnostic tool, clinical decision-support system, prescription system, or emergency service. Consult a qualified healthcare professional for medical decisions.
+## References
+- [Sentence Transformers](https://www.sbert.net/)
+- [FAISS](https://github.com/facebookresearch/faiss)
+- [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html)