Create README.md

README.md

@@ -0,0 +1,1196 @@
+---
+language:
+  - en
+library_name: sentence-transformers
+tags:
+  - rag
+  - retrieval-augmented-generation
+  - medical
+  - question-answering
+  - semantic-search
+  - faiss
+  - cross-encoder
+  - medical-qa
+pipeline_tag: question-answering
+license: apache-2.0
+---
+
+# Medical RAG System
+
+A retrieval-augmented generation (RAG) system for answering medical questions using a curated English medical Question-Answer knowledge base.
+
+The system combines:
+
+- Semantic retrieval using Sentence Transformers
+- FAISS vector search
+- Medical query expansion
+- Cross-Encoder reranking
+- Evidence deduplication
+- Confidence gating
+- LLM-based answer generation
+- Retrieval evaluation
+- Confidence-gate evaluation
+- Latency measurement
+
+> **Important:** This system is intended for research and educational purposes. It is not a medical diagnostic system and should not be used as a substitute for professional medical advice.
+
+---
+
+# 1. Overview
+
+This project implements a complete Retrieval-Augmented Generation pipeline for medical Question-Answer data.
+
+Instead of asking an LLM to answer directly from its internal knowledge, the system first retrieves relevant medical evidence from a local knowledge base.
+
+The pipeline is:
+
+```text
+User Question
+      │
+      ▼
+Query Expansion
+      │
+      ▼
+Sentence Transformer Embedding
+      │
+      ▼
+FAISS Vector Search
+      │
+      ▼
+Top-N Candidate Documents
+      │
+      ▼
+Cross-Encoder Reranking
+      │
+      ▼
+Top-K Evidence
+      │
+      ▼
+Near-Duplicate Removal
+      │
+      ▼
+Confidence Gate
+      │
+      ├── Reject → Insufficient Evidence
+      │
+      ▼
+LLM Generation
+      │
+      ▼
+Evidence-Based Answer
+```
+
+---
+
+# 2. Knowledge Base
+
+The knowledge base contains approximately:
+
+```text
+16,384 medical Question-Answer records
+```
+
+The dataset contains medical questions and corresponding answers, together with categorical information.
+
+Main columns used by the system:
+
+```text
+Question
+Answer
+Category
+doc_id
+```
+
+A unique `doc_id` is assigned to every document:
+
+```python
+df = df.reset_index(drop=True)
+df["doc_id"] = df.index
+```
+
+The `doc_id` is later used to identify evidence sources.
+
+---
+
+# 3. Embedding Model
+
+The system uses:
+
+```text
+sentence-transformers/all-MiniLM-L6-v2
+```
+
+The model converts every medical question into a dense vector representation.
+
+Embeddings are normalized:
+
+```python
+question_embeddings = embedder.encode(
+    questions,
+    batch_size=64,
+    show_progress_bar=True,
+    convert_to_numpy=True,
+    normalize_embeddings=True,
+)
+```
+
+Normalization allows the FAISS Inner Product index to behave as cosine similarity search.
+
+The resulting embedding matrix is stored as:
+
+```text
+question_embeddings.npy
+```
+
+---
+
+# 4. FAISS Retrieval
+
+FAISS is used for efficient vector similarity search.
+
+The index is created using:
+
+```python
+faiss.IndexFlatIP(embedding_dim)
+```
+
+Because the embeddings are normalized, Inner Product corresponds to cosine similarity.
+
+The FAISS index contains:
+
+```text
+16,384 vectors
+```
+
+The index is saved as:
+
+```text
+questions.index
+```
+
+Retrieval works by:
+
+1. Embedding the user query
+2. Searching the FAISS index
+3. Returning the top-N most similar questions
+
+Default retrieval configuration:
+
+```python
+RETRIEVE_TOP_N = 20
+```
+
+---
+
+# 5. Query Expansion
+
+The system includes a lightweight medical query expansion layer.
+
+This helps bridge the gap between everyday language and medical terminology.
+
+Examples:
+
+```text
+underactive thyroid
+        ↓
+hypothyroidism
+
+overactive thyroid
+        ↓
+hyperthyroidism
+
+high blood sugar
+        ↓
+hyperglycemia
+
+low blood sugar
+        ↓
+hypoglycemia
+
+sugar disease
+        ↓
+diabetes
+```
+
+For example:
+
+```text
+I suffer from an underactive thyroid gland
+```
+
+becomes:
+
+```text
+I suffer from an underactive thyroid gland
+(hypothyroidism)
+```
+
+The original query is not replaced.
+
+The medical term is simply appended before generating the embedding.
+
+---
+
+# 6. Cross-Encoder Reranking
+
+After FAISS retrieves the initial candidates, a Cross-Encoder is used to rerank them.
+
+Model:
+
+```text
+cross-encoder/ms-marco-MiniLM-L-6-v2
+```
+
+The system retrieves:
+
+```text
+Top 20
+```
+
+candidates and reranks them to keep:
+
+```text
+Top 6
+```
+
+Configuration:
+
+```python
+RETRIEVE_TOP_N = 20
+RERANK_TOP_K = 6
+```
+
+The reranker scores:
+
+```text
+(query, Answer)
+```
+
+rather than:
+
+```text
+(query, Question)
+```
+
+This is important because the dataset contains multiple records with the same Question but different medical answers.
+
+Reranking the Answer therefore helps determine which evidence is actually useful for answering the user's question.
+
+---
+
+# 7. Evidence Deduplication
+
+Medical datasets can contain multiple answers that are almost identical.
+
+The system removes near-duplicate answers using:
+
+```python
+difflib.SequenceMatcher
+```
+
+with:
+
+```python
+threshold = 0.92
+```
+
+This prevents the LLM from receiving multiple copies of essentially the same evidence.
+
+Each evidence item contains:
+
+```text
+doc_id
+question
+answer
+category
+similarity
+rerank_score
+```
+
+---
+
+# 8. Confidence Gate
+
+The system does not automatically answer every question.
+
+Before sending evidence to the LLM, a confidence gate checks whether the retrieved evidence is strong enough.
+
+Current configuration:
+
+```python
+MIN_RERANK_SCORE = -8
+MIN_SIMILARITY_FLOOR = 0.40
+MIN_SUPPORT_COUNT = 2
+SUPPORT_SCORE = -5.0
+```
+
+The system accepts the evidence only if:
+
+1. At least one result exists.
+2. The highest rerank score is above the minimum threshold.
+3. The similarity score is above the minimum similarity floor.
+4. At least two retrieved documents provide sufficient support.
+
+Otherwise, the system refuses to generate an evidence-based answer.
+
+Example:
+
+```text
+Question:
+Why do I feel short of breath and tired when I climb stairs?
+
+Result:
+No sufficiently relevant evidence.
+
+System:
+I couldn't find sufficiently relevant medical evidence
+in the knowledge base to answer this question.
+```
+
+This behavior is important for reducing unsupported answers.
+
+---
+
+# 9. LLM Generation
+
+The generation stage uses a Groq-hosted LLM.
+
+Current model used during development:
+
+```text
+openai/gpt-oss-20b
+```
+
+The LLM is explicitly instructed to operate in RAG mode.
+
+Main generation rules:
+
+```text
+1. Use only retrieved evidence.
+2. Do not add outside medical knowledge.
+3. Do not guess.
+4. Cite evidence using [doc_id].
+5. Mention when evidence is insufficient.
+6. Do not diagnose users.
+7. Do not prescribe personalized treatment or dosages.
+8. Mention disagreements between sources.
+9. Keep answers concise and organized.
+10. Include a medical-information disclaimer.
+```
+
+The API key is NOT stored in this repository.
+
+It should be provided through an environment variable:
+
+```python
+import os
+
+GROQ_API_KEY = os.environ["GROQ_API_KEY"]
+```
+
+---
+
+# 10. Example
+
+## Input
+
+```text
+What signs might suggest that my thyroid is not producing enough hormones?
+```
+
+## Retrieved Evidence
+
+```text
+[15967] Thyroid symptoms
+[13504] Thyroid symptoms
+[10676] Thyroid symptoms
+[13989] What are the symptoms of thyroid disease?
+[13559] Symptoms that indicate a disorder in the thyroid gland
+[12173] How do I know that I have a thyroid problem?
+```
+
+## Generated Answer
+
+The system generated an evidence-based response describing symptoms associated with low thyroid hormone production, including:
+
+- Fatigue
+- Feeling cold
+- Weight gain
+- Constipation
+- Menstrual changes
+- Hair loss
+
+The final response also included the required medical disclaimer.
+
+---
+
+# 11. Example of Out-of-Domain Query
+
+## Input
+
+```text
+What is the best treatment for a broken leg?
+```
+
+The retriever returned some medically related documents, mostly concerning diabetic foot injuries.
+
+However, those documents did not contain appropriate evidence about treating fractures.
+
+The final system response therefore stated that the available evidence did not contain information about treating a broken leg.
+
+This demonstrates an important property of the system:
+
+> Retrieval similarity alone does not guarantee that the retrieved evidence is appropriate for the question.
+
+---
+
+# 12. Evaluation
+
+The system was evaluated using multiple complementary evaluation methods.
+
+These include:
+
+- Self-retrieval accuracy
+- Recall@K
+- Precision@K
+- F1@K
+- MRR
+- NDCG@K
+- Category sanity check
+- Confidence-gate accuracy
+- Cross-validation threshold tuning
+- Latency evaluation
+
+---
+
+# 13. Self-Retrieval Evaluation
+
+A sample of 50 questions was tested.
+
+Result:
+
+```text
+Self-retrieval Top-1 Accuracy:
+50 / 50
+
+= 100%
+```
+
+However, this metric should NOT be interpreted as proof that the system has 100% real-world retrieval accuracy.
+
+It mainly verifies that the embedding/index pipeline can retrieve the original record when the exact same question is used.
+
+---
+
+# 14. Retrieval Evaluation
+
+The retrieval evaluation used questions that have duplicate Question entries in the dataset.
+
+This provides multiple possible relevant answers for the same question.
+
+```text
+Questions with duplicate entries usable for evaluation:
+67
+```
+
+## Retrieval Only
+
+```text
+Recall@1     = 0.690
+Precision@1  = 1.000
+F1@1         = 0.745
+
+Recall@3     = 0.851
+Precision@3  = 0.560
+F1@3         = 0.563
+
+Recall@5     = 0.899
+Precision@5  = 0.423
+F1@5         = 0.462
+
+Recall@10    = 0.959
+Precision@10 = 0.286
+F1@10        = 0.349
+
+MRR          = 1.000
+```
+
+Number of evaluated queries:
+
+```text
+150
+```
+
+---
+
+# 15. Retrieval + Cross-Encoder Evaluation
+
+The same evaluation was also performed after Cross-Encoder reranking.
+
+Results:
+
+```text
+Recall@1     = 0.083
+Precision@1  = 0.113
+F1@1         = 0.093
+
+Recall@3     = 0.157
+Precision@3  = 0.087
+F1@3         = 0.100
+
+Recall@5     = 0.222
+Precision@5  = 0.121
+F1@5         = 0.120
+
+Recall@10    = 0.388
+Precision@10 = 0.135
+F1@10        = 0.153
+
+MRR          = 0.237
+```
+
+Number of evaluated queries:
+
+```text
+150
+```
+
+---
+
+# 16. Important Evaluation Observation
+
+The evaluation shows that the Cross-Encoder performed significantly worse under this particular retrieval benchmark.
+
+This is an important finding rather than something to hide.
+
+The current reranker was trained for general passage relevance using:
+
+```text
+cross-encoder/ms-marco-MiniLM-L-6-v2
+```
+
+while this evaluation is based on medical Question-Answer records.
+
+Therefore, the Cross-Encoder may not rank the medical answers in the same way as the manually defined relevance criteria.
+
+This suggests that the reranking component requires further investigation before being considered production-ready.
+
+Possible future improvements include:
+
+- Medical-domain reranker
+- Fine-tuning a Cross-Encoder on medical relevance pairs
+- Using a manually reviewed gold set
+- Better relevance labeling
+- Evaluating reranking separately from retrieval
+- Testing alternative reranking models
+
+---
+
+# 17. Category Sanity Check
+
+A category-based sanity check was also performed.
+
+The goal is to determine whether retrieved documents contain at least one document belonging to the same category as the query's source document.
+
+## Retrieval + Reranker
+
+```text
+Category Recall = 0.98
+n = 100
+```
+
+## Retrieval Only
+
+```text
+Category Recall = 0.97
+n = 100
+```
+
+This indicates that the retrieval system generally retrieves documents from the expected medical category.
+
+However:
+
+> Category matching is only a supporting diagnostic metric and is NOT treated as the final ground truth for relevance.
+
+---
+
+# 18. Confidence Gate Evaluation
+
+A manually constructed evaluation set contained:
+
+```text
+25 in-domain questions
+25 out-of-domain questions
+```
+
+Total:
+
+```text
+50 questions
+```
+
+The system achieved:
+
+```text
+Gate Decision Accuracy = 88%
+```
+
+The system correctly accepted most in-domain questions and rejected most out-of-domain questions.
+
+However, several false positives were observed.
+
+Examples included questions about:
+
+```text
+broken legs
+kidney stones
+sprained ankles
+broken arms
+food poisoning
+```
+
+These questions sometimes retrieved medically related evidence even though the knowledge base did not contain appropriate evidence.
+
+This demonstrates why confidence gating and stronger relevance evaluation are necessary.
+
+---
+
+# 19. Confidence Score Distribution
+
+For the confidence-gate evaluation:
+
+### Expected answer = True
+
+```text
+Mean score = 5.539
+Minimum    = -8.938
+Maximum    = 9.321
+```
+
+### Expected answer = False
+
+```text
+Mean score = -6.167
+Minimum    = -11.062
+Maximum    = 1.532
+```
+
+There is overlap between the two distributions.
+
+Therefore, a single rerank-score threshold cannot perfectly separate valid and invalid questions.
+
+---
+
+# 20. Threshold Tuning
+
+The confidence threshold was evaluated using 5-fold cross-validation.
+
+Instead of selecting a threshold from one train/test split, the data is divided into five different folds.
+
+For every fold:
+
+1. A threshold is optimized on the training portion.
+2. The threshold is evaluated on the held-out fold.
+3. The threshold and test accuracy are recorded.
+4. Mean and standard deviation are calculated.
+
+This helps determine whether the selected threshold is stable or highly dependent on a small evaluation sample.
+
+A large threshold standard deviation indicates that the evaluation set is too small or unstable and should be expanded before selecting a production threshold.
+
+---
+
+# 21. Gold Set
+
+A semi-automatic gold-set construction process was implemented.
+
+The process:
+
+```text
+100 sampled questions
+        │
+        ▼
+Exact duplicate questions
+        │
+        ▼
+Semantic candidates with similarity >= 0.90
+        │
+        ▼
+Manual review
+        │
+        ▼
+Relevant = 1
+Not relevant = 0
+        │
+        ▼
+Final Gold Set
+```
+
+The initial run generated:
+
+```text
+100 questions
+16 candidate pairs
+3 questions with semantic candidates
+```
+
+The generated file:
+
+```text
+gold_set_for_manual_review.xlsx
+```
+
+contains candidate pairs that should be manually reviewed.
+
+For each candidate, the reviewer should enter:
+
+```text
+1 = relevant
+0 = not relevant
+```
+
+Only manually approved candidates should be added to the final relevant evidence set.
+
+---
+
+# 22. Gold Set Evaluation Metrics
+
+Once the manually reviewed gold set is completed, the system evaluates:
+
+```text
+Recall@5
+MRR
+NDCG@5
+```
+
+The evaluation compares:
+
+```text
+Retriever only
+```
+
+against:
+
+```text
+Retriever + Cross-Encoder Reranker
+```
+
+The manually reviewed gold set is preferred over category-based evaluation because it provides explicit relevance judgments.
+
+---
+
+# 23. Latency Evaluation
+
+The complete RAG pipeline was also measured for response latency.
+
+Example result:
+
+```text
+Retrieval       = 0.016 sec
+Reranking       = 0.094 sec
+Evidence build  = 0.025 sec
+LLM generation  = 0.858 sec
+
+Total           = 0.993 sec
+```
+
+Approximate breakdown:
+
+```text
+Retrieval       █
+Reranking       █████
+Evidence        ██
+LLM             █████████████████████████████████████
+```
+
+The LLM generation stage is the largest contributor to total latency.
+
+The measured end-to-end latency in this test was approximately:
+
+```text
+0.99 seconds
+```
+
+This should not be interpreted as a guaranteed production latency because API/network conditions, hardware, load, and model availability can change.
+
+---
+
+# 24. Saved Artifacts
+
+The system uses the following main artifacts:
+
+```text
+questions.index
+question_embeddings.npy
+README.md
+gold_set_for_manual_review.xlsx
+```
+
+## questions.index
+
+FAISS vector index containing the question embeddings.
+
+## question_embeddings.npy
+
+NumPy array containing the normalized question embeddings.
+
+## gold_set_for_manual_review.xlsx
+
+Human-review file used to create the manually verified relevance set.
+
+---
+
+# 25. Local Deployment
+
+The RAG system can be deployed locally as a Python service.
+
+Recommended architecture:
+
+```text
+Frontend
+   │
+   ▼
+Backend API
+   │
+   ▼
+Medical RAG Pipeline
+   │
+   ├── FAISS
+   ├── Sentence Transformer
+   ├── Cross Encoder
+   └── LLM API
+```
+
+The application should load the embedding model and FAISS index once when the service starts.
+
+They should NOT be reloaded for every user request.
+
+Example:
+
+```python
+from sentence_transformers import SentenceTransformer, CrossEncoder
+import faiss
+import numpy as np
+
+embedder = SentenceTransformer(
+    "sentence-transformers/all-MiniLM-L6-v2"
+)
+
+reranker = CrossEncoder(
+    "cross-encoder/ms-marco-MiniLM-L-6-v2"
+)
+
+faiss_index = faiss.read_index(
+    "questions.index"
+)
+
+question_embeddings = np.load(
+    "question_embeddings.npy"
+)
+```
+
+---
+
+# 26. Example API Architecture
+
+A backend can expose an endpoint such as:
+
+```http
+POST /api/rag/query
+```
+
+Request:
+
+```json
+{
+  "question": "What are the symptoms of diabetes?"
+}
+```
+
+Response:
+
+```json
+{
+  "answer": "...",
+  "evidence": [
+    {
+      "doc_id": 4793,
+      "similarity": 0.91,
+      "rerank_score": 8.21
+    }
+  ]
+}
+```
+
+This makes the RAG pipeline independent from the frontend.
+
+---
+
+# 27. Recommended Project Structure
+
+```text
+medical-rag/
+│
+├── README.md
+│
+├── artifacts/
+│   ├── questions.index
+│   └── question_embeddings.npy
+│
+├── src/
+│   ├── retrieval.py
+│   ├── reranker.py
+│   ├── query_expansion.py
+│   ├── evidence.py
+│   ├── confidence_gate.py
+│   ├── generation.py
+│   └── rag_pipeline.py
+│
+├── api/
+│   └── app.py
+│
+├── evaluation/
+│   ├── retrieval_evaluation.py
+│   ├── gate_evaluation.py
+│   └── gold_set_evaluation.py
+│
+├── data/
+│   └── gold_set_for_manual_review.xlsx
+│
+└── requirements.txt
+```
+
+---
+
+# 28. Installation
+
+Install the main dependencies:
+
+```bash
+pip install sentence-transformers faiss-cpu numpy pandas openpyxl
+```
+
+For Groq-based generation:
+
+```bash
+pip install groq
+```
+
+---
+
+# 29. Environment Variables
+
+Never hard-code API keys inside source code.
+
+Set:
+
+```bash
+GROQ_API_KEY=your_api_key_here
+```
+
+Python:
+
+```python
+import os
+
+api_key = os.environ["GROQ_API_KEY"]
+```
+
+---
+
+# 30. Security
+
+The API key used during development must not be committed to GitHub or Hugging Face.
+
+Recommended practice:
+
+```text
+.env
+```
+
+and:
+
+```text
+.gitignore
+```
+
+Example `.gitignore`:
+
+```text
+.env
+__pycache__/
+*.pyc
+.ipynb_checkpoints/
+```
+
+If an API key has already been exposed publicly, revoke it and generate a new one.
+
+---
+
+# 31. Limitations
+
+This system has several important limitations.
+
+### 1. Medical domain limitations
+
+The knowledge base does not necessarily cover every medical condition or clinical scenario.
+
+### 2. Retrieval limitations
+
+A semantically similar document is not necessarily an appropriate answer.
+
+### 3. Reranker limitations
+
+The current Cross-Encoder was not specifically trained on this medical dataset.
+
+### 4. Gold-set limitations
+
+The manually reviewed gold set is still small.
+
+### 5. Evaluation limitations
+
+Some evaluation metrics rely on duplicate Question entries rather than a fully manually annotated relevance dataset.
+
+### 6. LLM limitations
+
+The generated answer depends on the quality of the retrieved evidence.
+
+### 7. Clinical safety
+
+The system should not be used for diagnosis, emergency decisions, prescribing medication, or personalized treatment.
+
+---
+
+# 32. Future Improvements
+
+Recommended next steps:
+
+```text
+1. Complete the manual gold-set annotation.
+
+2. Increase the number of manually reviewed questions.
+
+3. Train or evaluate a medical-domain reranker.
+
+4. Re-evaluate the Cross-Encoder using the manual gold set.
+
+5. Tune the confidence gate using a larger validation set.
+
+6. Add citation/source metadata.
+
+7. Add automated regression tests.
+
+8. Add monitoring for retrieval failures.
+
+9. Add API-level authentication and rate limiting.
+
+10. Deploy the RAG pipeline behind a REST API.
+
+11. Add a frontend chat interface.
+
+12. Evaluate hallucination/faithfulness separately from retrieval quality.
+```
+
+---
+
+# 33. Evaluation Summary
+
+| Metric | Result |
+|---|---:|
+| Knowledge Base Size | 16,384 records |
+| Self-Retrieval Top-1 | 100% |
+| Retrieval Recall@1 | 69.0% |
+| Retrieval Recall@3 | 85.1% |
+| Retrieval Recall@5 | 89.9% |
+| Retrieval Recall@10 | 95.9% |
+| Retrieval Precision@1 | 100% |
+| Retrieval F1@1 | 74.5% |
+| Retrieval MRR | 1.000 |
+| Category Recall + Reranker | 98% |
+| Category Recall Retrieval Only | 97% |
+| Confidence Gate Accuracy | 88% |
+| Example End-to-End Latency | 0.993 sec |
+
+> The evaluation numbers above are specific to the current experimental setup and should not be interpreted as general medical accuracy.
+
+---
+
+# 34. Responsible Use
+
+This project provides general medical information retrieval and generation.
+
+It is not:
+
+- A diagnostic tool
+- A clinical decision-support system
+- A replacement for a physician
+- A prescription system
+- An emergency medical service
+
+Users should consult qualified healthcare professionals for medical decisions.
+
+---
+
+# 35. Citation and References
+
+### Sentence Transformers
+
+https://www.sbert.net/
+
+### FAISS
+
+https://github.com/facebookresearch/faiss
+
+### Cross-Encoder
+
+https://www.sbert.net/examples/applications/cross-encoder/README.html
+
+### Hugging Face Model Cards
+
+https://huggingface.co/docs/hub/en/model-cards
+
+### Hugging Face Model Release Checklist
+
+https://huggingface.co/docs/hub/en/model-release-checklist
+
+---
+
+# 36. Project Status
+
+```text
+Status: Research / Prototype
+
+Retrieval: Implemented
+FAISS: Implemented
+Query Expansion: Implemented
+Cross-Encoder Reranking: Implemented
+Evidence Deduplication: Implemented
+Confidence Gate: Implemented
+LLM Generation: Implemented
+Evaluation: Implemented
+Latency Measurement: Implemented
+Manual Gold Set: In Progress
+Production Deployment: Future Work
+```
+
+---
+
+# 37. Final Note
+
+The main objective of this project is not simply to generate medical answers.
+
+The goal is to build a RAG system that:
+
+```text
+Retrieves evidence
+       ↓
+Ranks evidence
+       ↓
+Checks confidence
+       ↓
+Generates from evidence
+       ↓
+Provides traceable document IDs
+```
+
+This design allows the system to be evaluated as a retrieval pipeline independently from the LLM generation layer.