Refactor: Move scraping logic to data_ingestion folder

README.md

@@ -46,10 +46,10 @@ graph TD
 
 ### Components
 
-1.  **Data Ingestion (`scraper.py`)**:
+1.  **Data Ingestion (`data_ingestion/scraper.py`)**:
     *   Fetches 8 specific road-test articles.
     *   Parses HTML to extract clean text (Title, Subtitle, Body).
-    *   Saves to `scraped_data.json`.
+    *   Saves to `data_ingestion/scraped_data.json`.
 
 2.  **RAG Engine (`rag_engine.py`)**:
     *   **Encoder**: `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` for creating vector embeddings of text chunks.

scraper.py → data_ingestion/scraper.py

@@ -105,7 +105,9 @@ def main():
 
 if __name__ == "__main__":
     import json
+    import os
     data = main()
-    with open("scraped_data.json", "w", encoding='utf-8') as f:
+    output_path = os.path.join(os.path.dirname(__file__), "scraped_data.json")
+    with open(output_path, "w", encoding='utf-8') as f:
         json.dump(data, f, ensure_ascii=False, indent=2)
-    print("Saved to scraped_data.json")
+    print(f"Saved to {output_path}")

rag_engine.py

@@ -5,7 +5,7 @@ import google.generativeai as genai
 import os
 
 class RAGEngine:
-    def __init__(self, data_path="scraped_data.json"):
+    def __init__(self, data_path="data_ingestion/scraped_data.json"):
         print("Initializing RAG Engine...")
         self.data_path = data_path
         self.encoder = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')