Feat: Introduce ReviewSense v2.0 with RAG Chatbot and Mistral LLM

README.md

@@ -1,188 +1,295 @@
 ![Banner](assets/banner.png)
-[![Python](https://img.shields.io/badge/Python-3.12.11-blue?logo=python)](https://www.python.org/)[![PyTorch](https://img.shields.io/badge/PyTorch-2.8-EE4C2C?logo=pytorch)](https://pytorch.org/)![Made with ML](https://img.shields.io/badge/Made%20with-ML-blueviolet?logo=openai)[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/Python-3.12+-blue?logo=python)](https://www.python.org/)[![PyTorch](https://img.shields.io/badge/LangChain-Integration-blueviolet)](https://python.langchain.com/)[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
-# 🛍️ ReviewSense: Product Review Analysis Engine
+# 🛍️ ReviewSense v2.0: Product Review Analysis & Chatbot Engine
 
-> *ReviewSense is a comprehensive, end-to-end Natural Language Processing application built to extract deep, actionable insights from unstructured product reviews.*  
-Where a simple star rating only tells part of the story, ReviewSense dives into the text to uncover what customers are saying, why they're saying it, and how they feel about specific product features.  
+> *ReviewSense v2.0 expands upon the initial analysis engine, adding a powerful, conversational RAG (Retrieval-Augmented Generation) chatbot. It leverages a single, efficient LLM (Mistral 7B GGUF) to provide both deep batch analysis and interactive Q&A grounded in user-provided reviews.*
 
-This project demonstrates a complete **MLOps workflow**, from initial data preparation and model fine-tuning to the development and deployment of a multi-model, interactive web application with Gradio.
+This project demonstrates an end-to-end workflow, integrating data processing, local LLM execution with `LlamaCpp`, vector storage with FAISS, conversational memory, intent classification, and an interactive Gradio web application.
 
-![Gradio app](assets/gradio.png)
-You can find the Gradio app [Here](https://www.kaggle.com/datasets/kritanjalijain/amazon-reviews)
+![demo_tab1](assets/tab1.png)
+![demo_tab2](assets/tab2.png)
+
+You can find the Web demo here ➡ [Web Demo](https://huggingface.co/spaces/Deathshot78/ReviewSense)
+**[Note]: running this model on the cpu takes a while to complete you can relax and get a cup of coffee while the model generates responses !☕**
 
 ---
 
 ## 📋 Table of Contents
 
-- [📖 Overview](#-overview)  
-- [✨ Key Features](#-key-features)
-- [📈 Model Development and Benchmarking](#-model-development-and-benchmarking)
-- [🧠 How It Works: The Analysis Pipeline](#-how-it-works-the-analysis-pipeline)
+- [📖 Overview](#-overview)
+- [🚀 What's New in v2.0](#-whats-new-in-v20)
+- [✨ Key Features (v2.0)](#-key-features-v20)
+- [🧠 How It Works: The v2.0 Pipeline](#-how-it-works-the-v20-pipeline)
+- [🔧 Challenges & Limitations](#-challenges--limitations)
+- [💡 Prompt Engineering Journey](#-prompt-engineering-journey)
 - [🔮 Future Improvements](#-future-improvements)
-- [⚙️ Setup and Installation](#️-setup-and-installation)  
-- [▶️ Usage](#️-usage)  
-- [📁 Project Structure](#-project-structure)  
-- [🛠️ Technologies and Models](#️-technologies-and-models)  
+- [⚙️ Setup and Installation](#️-setup-and-installation)
+- [▶️ Usage](#️-usage)
+- [📁 Project Structure (v2.0)](#-project-structure-v20)
+- [🛠️ Technologies and Models (v2.0)](#️-technologies-and-models-v20)
+- [📜 Version History](#-version-history)
 
 ---
 
 ## 📖 Overview
 
-In the world of e-commerce, customer reviews are a **goldmine of information**. However, manually reading through thousands of reviews is impossible.  
-
-This project solves that problem by creating an automated system that performs a **multi-layered analysis** on any given product review, providing a structured output that is far more valuable than a simple positive/negative label.
+Building upon the foundation of ReviewSense v1.0, which focused on extracting insights like sentiment, aspects, and summaries using multiple specialized models, **Version 2.0 introduces a significant upgrade: a conversational chatbot**.
 
-The application is built to be **modular** and showcases the power of combining a custom fine-tuned model with several pre-trained transformers, each specialized for a specific task.
+This chatbot allows users to ask specific questions about product reviews and receive answers synthesized directly from the provided text. To achieve this efficiently and enhance overall capabilities, v2.0 consolidates the architecture around a single, powerful yet locally runnable Large Language Model (Mistral 7B GGUF). This unified model now handles both the batch analysis tasks (with improved quality) and the interactive Q&A, demonstrating a modern approach to building multi-functional NLP applications.
 
 ---
 
-## ✨ Key Features
+## 🚀 What's New in v2.0
 
-- 📈 **Overall Sentiment Analysis**  
-  Classifies reviews as **POSITIVE** or **NEGATIVE**, powered by a fine-tuned DistilBERT model on the Amazon Reviews dataset.
+Version 2.0 represents a major leap in functionality and architecture:
 
-- 🔎 **Dynamic Aspect Extraction**  
-  Automatically identifies product features (e.g., *camera, battery life, plot*) using POS tagging combined with category-specific dictionaries.
+1.  **🤖 RAG Chatbot Implementation:** Added an interactive chatbot (Phase 2) that uses Retrieval-Augmented Generation (RAG) to answer user questions based on review context.
+2.  **🧠 Single LLM Architecture:** Replaced the multiple specialized models (DistilBERT, DistilBART, DeBERTa, POS Tagger) from v1.0 with a single, powerful Mistral 7B GGUF model, executed locally via `LlamaCpp`. This model now handles:
+    * Batch Analysis (Summary, Aspects, Sentiment - Phase 1) with higher quality.
+    * RAG-based Question Answering (Phase 2).
+    * Intent Classification (Guardrail for Phase 2).
+3.  **📄 Dynamic Context Management:** The chatbot can now operate on a default set of reviews or dynamically update its knowledge base using user-uploaded `.txt` or `.csv` files.
+4.  **💬 Conversational Memory:** Integrated LangChain's `ConversationBufferMemory` allowing the chatbot to understand follow-up questions.
+5.  **🛡️ Intent Classification Guardrail:** Implemented a robust intent classifier (using the same LLM) to prevent the chatbot from answering off-topic questions, ensuring responses stay grounded in product reviews.
+6.  **🖥️ Unified Gradio UI:** Developed a two-tab Gradio interface (`app.py`) providing access to both the Batch Analyzer and the RAG Chatbot in a single application.
+7.  **💻 Local Execution Script:** Added `main.py` for command-line execution of batch analysis or interactive chat without the Gradio UI.
+8.  **🧱 Modular Code Structure:** Refactored code into `src/pipeline.py` for core logic, improving organization and maintainability.
+
+---
 
-- 📊 **Aspect-Based Sentiment Analysis (ABSA)**  
-  Determines specific sentiment (*Positive, Negative, Neutral*) for each aspect, e.g., *“loved the camera, disappointed with the battery life.”*
+## ✨ Key Features (v2.0)
 
-- 📝 **Abstractive Summarization**  
-  Generates concise summaries of reviews using a pre-trained DistilBART model.
+Includes all features from v1.0 (now powered by Mistral 7B) **plus**:
 
-- 🚀 **Interactive UI**  
-  A clean, user-friendly **Gradio interface** for real-time review analysis.
+- **Interactive RAG Chatbot:**
+    * Ask specific questions about product reviews (e.g., "How is the battery life?", "Is the app reliable?").
+    * Answers synthesized directly from provided review context using RAG.
+    * **Conversational Memory:** Understands follow-up questions ("What about the screen?").
+    * **Grounded Responses:** Designed to answer only based on the reviews provided.
+    * **Intent Guardrail:** Filters out and responds appropriately to off-topic questions.
+- **Dynamic Context Loading:**
+    * Chatbot operates on default reviews or context loaded from user-uploaded files (`.txt`/`.csv`).
+    * Clear indication of the currently active context.
+- **Unified LLM Backend:** All NLP tasks (analysis, Q&A, classification) handled by a single Mistral 7B GGUF model running locally.
+- **Dual Interface:** Accessible via Gradio web UI (`app.py`) or command line (`main.py`).
 
 ---
 
-## 📈 Model Development and Benchmarking
+## 🧠 How It Works: The v2.0 Pipeline
+
+**Phase 1: Batch Analysis (via `analyze_reviews_only` or `analyze_reviews_logic`)**
+1.  User provides review text (paste or file).
+2.  The text is passed to the Mistral LLM using three distinct prompts (Summarization, Aspect Extraction, Sentiment Analysis).
+3.  The LLM generates the three analysis outputs.
+
+**Phase 2: RAG Chatbot (via `ask_question_with_guardrail` or `get_chatbot_response`)**
+1.  User asks a question.
+2.  **Intent Classification:** The query is first sent to the Mistral LLM with the `intent_prompt` (few-shot) to classify it as "Product" or "Off-Topic". Robust parsing checks the LLM output.
+3.  **Routing:**
+    * If "Off-Topic", a canned response is returned.
+    * If "Product", proceed to RAG.
+4.  **Context Retrieval:** The user's question is used to query the current FAISS vector store (containing embeddings of the active review context) to retrieve the top `k` relevant review snippets.
+5.  **Conversational Chain Execution (`ConversationalRetrievalChain`):**
+    * **Condense Question:** If there's chat history, the LLM uses `CONDENSE_QUESTION_PROMPT` to rephrase the current question into a standalone query.
+    * **RAG Generation:** The condensed question and retrieved context snippets are passed to the LLM with the strict `qa_prompt`. The LLM synthesizes an answer based *only* on the provided context.
+    * **Memory Update:** The question and final answer are added to the `ConversationBufferMemory`.
+6.  **Response:** The synthesized answer is returned to the user.
 
-A key part of this project was to not just build a model, but to prove its effectiveness. To do this, we followed a standard machine learning practice of establishing a strong baseline before moving to a more complex architecture.
+---
 
-**1. The Baseline Model**
-We first trained a classic `Multinomial Naive Bayes` model. The text was vectorized using `TF-IDF`, and we performed a hyperparameter grid search to find the optimal settings. This approach is fast, interpretable, and provides a strong benchmark for text classification.
+## 🔧 Challenges & Limitations
 
-![Results1](assets/confusion_bay.png)
+Developing v2.0 involved significant experimentation and revealed several challenges:
 
-- **Baseline Accuracy: 86.56%**
+1.  **Consistent Instruction Following:** While powerful, the Mistral 7B GGUF model sometimes struggled to consistently follow complex negative constraints or nuanced instructions in prompts, especially within the RAG chain. This led to:
+    * **Context Leakage:** Occasionally including irrelevant details from retrieved chunks (e.g., mentioning webcam when asked about battery).
+    * **Hallucination:** Making up information (e.g., mentioning "phone" for laptop battery, inventing prices or product names).
+    * **Over-Cautiousness:** Incorrectly stating "cannot find information" even when relevant details were present in the context, particularly for negative aspects (e.g., hardware issues).
+    * **Misinterpretation:** Failing to correctly understand the specific user question (e.g., "taste" vs. "type", comparison questions).
+2.  **Prompt Engineering Complexity:** Finding the right prompt structure required extensive iteration. Simple prompts lacked control, while overly complex prompts sometimes confused the model. Few-shot prompting proved essential for reliable intent classification. Balancing strictness (for grounding) with flexibility (to allow synthesis) in the RAG prompt was difficult.
+3.  **Intent Classification Brittleness:** Getting the LLM to output *only* the classification label required moving from zero-shot, to strict instructions, to few-shot examples, and finally adding robust parsing logic (`parse_intent`) to handle noisy LLM outputs reliably.
+4.  **Performance:** Running the 7B parameter GGUF model on a CPU is significantly slower than using smaller models or GPU acceleration. Batch analysis and RAG responses take noticeable time (though acceptable for demonstration).
+5.  **Evaluation Bottleneck:** Using external APIs (like OpenAI) for RAGAs evaluation can incur costs and hit rate limits. Using the local model for evaluation is free but slower and potentially less objective.
 
-**1. The Fine-Tuned Transformer**
-Next, we fine-tuned a `DistilBERT` model, a smaller and more efficient variant of BERT. By leveraging its pre-trained understanding of the English language and fine-tuning it on our specific Amazon reviews data, we aimed to capture more of the nuance and context that the baseline model might miss.
+---
 
-![Results2](assets/confusion_bert.png)
+## 💡 Prompt Engineering Journey
+
+Achieving the final, relatively stable performance required significant iteration on the prompts, particularly for the RAG chain (`qa_prompt`) and intent classification (`intent_prompt`).
+
+**Intent Classification (`intent_prompt`):**
+
+* Initial attempts with simple zero-shot prompts failed, with the model providing verbose, incorrect classifications.
+* Adding strict formatting rules (`MUST BE EXACTLY...`) helped but wasn't sufficient.
+* **Few-Shot Prompting** (providing explicit examples within the prompt) proved crucial for forcing the model to output the correct labels, although often with extra text.
+* **Robust Parsing (`parse_intent`)** was added to reliably extract the core "Product" or "Off-Topic" keyword from the model's potentially noisy output.
+
+**Final `intent_template`:**
+
+```python
+intent_template = """
+[INST]
+**CRITICAL INSTRUCTION:** Classify the user's query into ONLY ONE of two categories: "Product" or "Off-Topic".
+Your response MUST be EXACTLY "Product" or EXACTLY "Off-Topic".
+
+**EXAMPLES:**
+Query: How is the battery life?
+Classification: Product
+Query: What are the complaints about the screen?
+Classification: Product
+Query: Does it come in blue?
+Classification: Product
+Query: What is the capital of France?
+Classification: Off-Topic
+Query: Hello there
+Classification: Off-Topic
+Query: Who are you?
+Classification: Off-Topic
+
+**NOW CLASSIFY THIS QUERY:**
+Query: {query}
+[/INST]
+Classification:"""
+```
 
-- **Fine-Tuned Accuracy: ~95%**
+**RAG Generation (`qa_system_prompt`):**
 
-![Results3](assets/wordcloud.png)
+* Initial simple prompts led to significant hallucination and context leakage.
 
-**Conclusion**
+* Adding strict rules improved grounding but sometimes made the model overly cautious, failing to find information present in the context.
 
-The significant increase in accuracy from **86.6%** to **~95%** demonstrates the power of transfer learning and the superior contextual understanding of transformer models. This performance gain justified the selection of the more computationally intensive DistilBERT model for the final application.
+* Explicitly addressing failure modes (like comparisons) helped for those specific cases.
 
----
+* Experimenting with different chain types (`stuff`, `map_reduce`, `refine`) showed limitations related to context window size and model instruction following. `stuff` with `ConversationalRetrievalChain` proved most practical.
 
-## 🧠 How It Works: The Analysis Pipeline
+**Final qa_system_prompt (within qa_prompt):**
 
-When a user submits a review, the application processes it through these steps:
+```python
+# RAG System Prompt (qa_system_prompt)
+qa_system_prompt = """[INST]You are a factual assistant providing answers based **only** on the customer reviews provided.
+Your task is to answer the user's question concisely using information explicitly found in the 'CONTEXT' snippets below.
 
-1. **Category Selection** → User selects product type (*Phone, Book, etc.*).  
-2. **Aspect Extraction** → Extracts nouns via POS tagging and filters with category dictionary.  
-3. **Aspect Sentiment Analysis** → DeBERTa-based ABSA model assigns sentiment per aspect.  
-4. **Overall Sentiment Classification** → DistilBERT model predicts POSITIVE/NEGATIVE.  
-5. **Summarization** → DistilBART generates a short summary.  
-6. **Display** → Results presented in the Gradio UI.
+**CRITICAL RULES TO FOLLOW:**
+1.  **STRICTLY Contextual:** Base your answer ENTIRELY and ONLY on the information within the 'CONTEXT' section. Do NOT use any prior knowledge or external information.
+2.  **Direct & Relevant:** Answer ONLY the specific question asked. Do NOT include details from the context that are irrelevant to the question, even if they appear nearby.
+3.  **Synthesize Concisely:** Combine relevant facts from potentially multiple snippets into a brief answer (usually 1-3 sentences). Do NOT quote long passages unless absolutely necessary.
+4.  **No Comparisons Outside Context:** If the question asks to compare the product to something *not mentioned* in the reviews, state *only*: "Cannot compare based on the provided reviews." Do not add details about the product itself in this case.
+5.  **Handle Missing Info Carefully:** If, after carefully reading the context, you genuinely cannot find any information relevant to the question, state *only*: "Based on the provided reviews, I cannot find information about that." Check thoroughly before using this response.
+6.  **Factual Tone:** Do NOT apologize, express opinions, make recommendations, or use conversational filler. Just state the facts found in the reviews.
+
+CONTEXT:
+---
+{context}
+---
+
+QUESTION: {question} [/INST]
+Answer:"""
+```
+
+This iterative process demonstrates the practical challenges and refinement needed when working with local LLMs in complex pipelines.
 
 ---
 
 ## 🔮 Future Improvements
 
-- 📂 **Batch Processing Mode** → Upload CSV, get analyzed CSV output.
+* **RAG Evaluation**: Fully implement and integrate RAGAs (or TruLens) evaluation using the local LLM or a free tier API to get quantitative metrics on Faithfulness, Answer Relevancy, etc.
+
+* **LLM Upgrade**: Experiment with larger or more advanced instruction-tuned models (e.g., Mixtral GGUF, Llama 3 70/8B Instruct GGUF, or API-based models like GPT-4/Claude 3) to achieve higher consistency in instruction following and synthesis.
+
+* **Advanced Retrieval**: Explore more sophisticated retrieval techniques (e.g., HyDE, MultiQueryRetriever, Re-ranking) to improve the quality of context chunks passed to the LLM, potentially reducing generation errors.
 
-- 📚 **Expand Aspect Dictionaries for more product categories.**
+* **Batch Processing for Analysis**: Re-implement batch processing for Phase 1 using techniques like `map_reduce` to handle large numbers of reviews that exceed the LLM's context window.
 
-- 🤖 **Advanced Aspect Extraction** → Replace POS heuristics with ATE models for discovering new aspects automatically.
-- 💹 **Train the sentiment reviewer for longer and on more of the data for better performance.**
+* **Error Handling & UI**: Add more granular error handling and user feedback in the Gradio UI (e.g., clearer messages if context loading fails).
+
+* **Automated Testing**: Implement unit and integration tests using `pytest` for the core logic in `src/pipeline.py`.
 
 ---
 
 ## ⚙️ Setup and Installation
 
-### 1. Clone the Repository
+**1. Clone the Repository**
 
 ```bash
-git clone https://github.com/Deathshot78/ReviewSense.git
+git clone [https://github.com/Deathshot78/ReviewSense.git](https://github.com/Deathshot78/ReviewSense.git) # Replace with your repo URL if different
 cd ReviewSense
 ```
 
-### 2. Install Required Packages
+**2. Install Required Packages**
 
 ```bash
 pip install -r requirements.txt
 ```
 
-### 3. Download Dataset
+**3. Download LLM Model**
 
-Download Amazon Reviews for Sentiment Analysis from [Kaggle](https://www.kaggle.com/datasets/kritanjalijain/amazon-reviews).
+The scripts will attempt to download the Mistral-7B GGUF model (`mistral-7b-instruct-v0.1.Q4_K_M.gguf`, ~4.4GB) automatically via `wget` on the first run if it's not found in the root directory. You can also download it manually from [Hugging Face](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF) and place it in the project root.
 
-Place `train.csv` and `test.csv` into the `data/` directory.
+---
 
-### 4. Preprocess Data
+## ▶️ Usage
 
-```bash
-python data_prepare.py
-```
+**Web App (Gradio)**
 
-### 5. Fine-Tune Sentiment Model
+Run the Gradio app:
 
 ```bash
-python python train.py
+python app.py
 ```
 
----
-
-## ▶️ Usage
+Access the interface in your browser 
 
-Run the main script:
+* **Tab 1 ("Batch Analyzer"):** Paste reviews or upload a file to perform Summary, Aspect Extraction, and Sentiment Analysis. This does not affect the chatbot context.
 
-```bash
-python python main.py
-```
+* **Tab 2 ("Ask a Question"):** Chat with the RAG bot. Use the file upload and "Update Chatbot Context" button within this tab to change the reviews the chatbot uses. Use "Reset Chatbot Context to Default" to revert to the built-in laptop reviews. Use "Reset Chat Memory" to clear the conversation history.
 
 ---
 
-## 📁 Project Structure
+## 📁 Project Structure (v2.0)
 
 ```bash
-├── 📄 README.md               # Project documentation
-├── 🐍 app.py                  # Gradio app (UI + logic)
-├── 🐍 models.py               # Model definitions & inference classes
-├── 🐍 train.py                # Fine-tune sentiment classifier
-├── 🐍 data_prepare.py         # Data preprocessing script
-├── 🐍 app.py                  # Gradio app
-├── 📄 requirements.txt        # Dependencies
-├── 📁 data/                   # Datasets (local only)
-└── 📁 checkpoints/            # Saved model checkpoints
+ReviewSense/
+├── 📄 README.md               # Project documentation 
+├── 📁 src/                    # Source code for core 
+│   ├── 📄 app.py              # Gradio web application
+│   ├── 📄 pipeline.py         # Core functions for analysis, RAG, etc. 
+│   └── 📄 main.py             # Command-line execution
+├── 📄 requirements.txt        # Python dependencies
+├── 📄 .gitignore              # Files ignored by Git
+└── 🖼️ assets/                 # images 
 ```
 
 ---
 
-## 🛠️ Technologies and Models
+## 🛠️ Technologies and Models (v2.0)
 
 **Core Technologies**
 
-- Python, PyTorch, PyTorch Lightning
+* Python 3.10+
+
+* LangChain: Orchestration, Chains (ConversationalRetrievalChain), Memory, Prompts
+
+* llama-cpp-python: Local execution of GGUF models on CPU
 
-- Gradio (UI)
+* FAISS (faiss-cpu): Efficient vector similarity search
 
-- Pandas, Scikit-learn
+* Sentence-Transformers (all-MiniLM-L6-v2): Text embeddings
 
-- NLP Models (Hugging Face)
+* Gradio: Interactive web UI
 
-- Sentiment Classifier: distilbert-base-uncased (fine-tuned)
+* PyTorch (dependency via transformers/sentence-transformers)
+
+* Pandas, NumPy (standard data handling)
+
+**Core LLM**
+
+* Mistral 7B Instruct v0.1 (GGUF Q4_K_M): Used for all NLP tasks (Analysis, RAG Generation, Intent Classification). Downloaded from TheBloke on Hugging Face.
+  
+---
 
-- Review Summarizer: sshleifer/distilbart-cnn-6-6
+## 📜 Version History
 
-- Aspect Extractor: vblagoje/bert-english-uncased-finetuned-pos
+* v2.0 (Current): RAG Chatbot, Single Mistral 7B model, Dynamic Context, Memory, Guardrails, Gradio UI, Code Refactoring.
 
-- Aspect Analyzer: yangheng/deberta-v3-base-absa-v1.1
+* v1.0: [Link to v1.0 Release/Tag on GitHub, e.g., https://www.google.com/search?q=https://github.com/Deathshot78/ReviewSense/releases/tag/v1.0] - Initial Batch Analysis Engine using multiple specialized models (DistilBERT, DistilBART, etc.). Focused on Sentiment, Aspects, and Summarization. (See v1.0 README for full details).

checkpoints/sentiment-binary-best-checkpoint.ckpt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b80c0f5882524f5859ac6a92f3311a2d8d4638bd6ef1236232fbb32057b43f3d
-size 803593979

notebooks/reviewsense.ipynb

@@ -1,1001 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "1754f3bb",
-   "metadata": {},
-   "source": [
-    "# 🛍️ ReviewSense: Product Review Analysis Engine\n",
-    "\n",
-    "> *ReviewSense is a comprehensive, end-to-end Natural Language Processing application built to extract deep, actionable insights from unstructured product reviews.*  \n",
-    "Where a simple star rating only tells part of the story, ReviewSense dives into the text to uncover what customers are saying, why they're saying it, and how they feel about specific product features.  "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "00d383d6",
-   "metadata": {},
-   "source": [
-    "## Imports"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "4d48ba17",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pytorch_lightning as pl\n",
-    "from torch.utils.data import DataLoader, Dataset\n",
-    "from transformers import AutoTokenizer\n",
-    "import pandas as pd\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "import torch\n",
-    "import os\n",
-    "import numpy as np\n",
-    "from sklearn.model_selection import train_test_split, ParameterGrid, StratifiedKFold\n",
-    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
-    "from sklearn.naive_bayes import MultinomialNB\n",
-    "from sklearn.pipeline import Pipeline\n",
-    "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
-    "import seaborn as sns\n",
-    "import matplotlib.pyplot as plt\n",
-    "from tqdm.notebook import tqdm\n",
-    "from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping\n",
-    "from pytorch_lightning.loggers import TensorBoardLogger\n",
-    "from transformers import T5ForConditionalGeneration, T5Tokenizer\n",
-    "from transformers import AutoModelForSequenceClassification, get_linear_schedule_with_warmup, AutoConfig\n",
-    "from torch.optim import AdamW\n",
-    "import torch\n",
-    "from torchmetrics.functional import accuracy\n",
-    "from transformers import T5ForConditionalGeneration, T5Tokenizer, AutoTokenizer, pipeline\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8263bc02",
-   "metadata": {},
-   "source": [
-    "## Prepare the data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a5f8dcda",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def explore_and_preprocess_reviews(\n",
-    "    train_path='data/train.csv', \n",
-    "    test_path='data/test.csv',\n",
-    "    output_dir='data'\n",
-    "):\n",
-    "    \"\"\"\n",
-    "    Loads the Amazon Sentiment Analysis dataset (https://www.kaggle.com/datasets/kritanjalijain/amazon-reviews)\n",
-    "    (you need to extract the train/test splits from the zip file in the data folder),\n",
-    "    performs basic EDA, and preprocesses it for model training.\n",
-    "\n",
-    "    Args:\n",
-    "        train_path (str): Path to the training CSV file.\n",
-    "        test_path (str): Path to the testing CSV file.\n",
-    "        output_dir (str): Directory to save the processed file.\n",
-    "    \"\"\"\n",
-    "    # --- 1. Load Data ---\n",
-    "    # This dataset typically comes without headers. We'll assign them.\n",
-    "    # Column 1: Sentiment (1 = Negative, 2 = Positive)\n",
-    "    # Column 2: Title\n",
-    "    # Column 3: Review Text\n",
-    "    print(f\"Loading data from '{train_path}' and '{test_path}'...\")\n",
-    "    try:\n",
-    "        col_names = ['sentiment_orig', 'title', 'review']\n",
-    "        train_df = pd.read_csv(train_path, header=None, names=col_names)\n",
-    "        test_df = pd.read_csv(test_path, header=None, names=col_names)\n",
-    "        \n",
-    "        # Combine for unified EDA and preprocessing\n",
-    "        df = pd.concat([train_df, test_df], ignore_index=True)\n",
-    "\n",
-    "    except FileNotFoundError:\n",
-    "        print(f\"\\nERROR: Make sure '{train_path}' and '{test_path}' are in the specified directory.\")\n",
-    "        print(\"This script is designed for the 'Amazon Reviews for Sentiment Analysis' dataset from Kaggle.\")\n",
-    "        return\n",
-    "\n",
-    "    df.dropna(inplace=True)\n",
-    "\n",
-    "    # --- 2. Preprocessing ---\n",
-    "    print(\"\\n--- Preprocessing Data for Sentiment Analysis ---\")\n",
-    "\n",
-    "    # a) Create new sentiment labels (0 = Negative, 1 = Positive)\n",
-    "    # This dataset is binary, not three-class like the previous one.\n",
-    "    df['sentiment'] = df['sentiment_orig'].apply(lambda x: 0 if x == 1 else 1)\n",
-    "\n",
-    "    # b) Combine title and review body\n",
-    "    df['full_text'] = df['title'].astype(str) + \". \" + df['review'].astype(str)\n",
-    "\n",
-    "    # c) Select and rename columns\n",
-    "    processed_df = df[['full_text', 'sentiment']].copy()\n",
-    "\n",
-    "    # --- 4. Save Processed Data ---\n",
-    "    os.makedirs(output_dir, exist_ok=True)\n",
-    "    output_path = os.path.join(output_dir, 'reviews_processed.csv')\n",
-    "    processed_df.to_csv(output_path, index=False)\n",
-    "    print(f\"\\nSaved {len(processed_df)} processed reviews to '{output_path}'\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "60ab838c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#--- Preprocess the Reviews Dataset ---\n",
-    "print(\"\\n--- Preprocessing started ---\")\n",
-    "explore_and_preprocess_reviews()\n",
-    "print(\"\\n--- Preprocessing finished ---\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4c381d73",
-   "metadata": {},
-   "source": [
-    "## Define a base model (Multinomial Naive Bayes)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b3cd2b5b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def train_baseline_sentiment_model(data_path='data/reviews_processed.csv', grid_search=True, nb__alpha=0.1, tfidf__max_df=0.75, tfidf__ngram_range=(1, 2), sample_size: int = 50000):\n",
-    "    \"\"\"\n",
-    "    Trains and evaluates a Multinomial Naive Bayes model for sentiment analysis.\n",
-    "    Can optionally perform a grid search.\n",
-    "\n",
-    "    Args:\n",
-    "        data_path (str): Path to the processed reviews CSV file.\n",
-    "        grid_search (bool): If True, performs a grid search.\n",
-    "        nb__alpha (float): Alpha for MultinomialNB.\n",
-    "        tfidf__max_df (float): max_df for TfidfVectorizer.\n",
-    "        tfidf__ngram_range (tuple): ngram_range for TfidfVectorizer.\n",
-    "        sample_size (int, optional): Number of reviews to use. If None, uses all.\n",
-    "    \"\"\"\n",
-    "    # --- 1. Load Data ---\n",
-    "    print(f\"Loading data from '{data_path}'...\")\n",
-    "    if not os.path.exists(data_path):\n",
-    "        print(f\"\\nERROR: '{data_path}' not found. Please run the EDA script first!\")\n",
-    "        return\n",
-    "        \n",
-    "    df = pd.read_csv(data_path)\n",
-    "    df.dropna(inplace=True)\n",
-    "\n",
-    "    # --- 2. Sample Data ---\n",
-    "    if sample_size:\n",
-    "        print(f\"Using a sample of {sample_size} reviews for training the baseline model.\")\n",
-    "        df = df.sample(n=sample_size, random_state=42)\n",
-    "\n",
-    "    # --- 3. Train-Test Split ---\n",
-    "    print(\"Splitting data into training and testing sets...\")\n",
-    "    X_train, X_test, y_train, y_test = train_test_split(\n",
-    "        df['full_text'],\n",
-    "        df['sentiment'],\n",
-    "        test_size=0.2,\n",
-    "        random_state=42,\n",
-    "        stratify=df['sentiment']\n",
-    "    )\n",
-    "\n",
-    "    # --- 4. Create a Pipeline ---\n",
-    "    pipeline = Pipeline([\n",
-    "        ('tfidf', TfidfVectorizer(stop_words='english')),\n",
-    "        ('nb', MultinomialNB()),\n",
-    "    ])\n",
-    "\n",
-    "    best_params = None\n",
-    "\n",
-    "    if grid_search:\n",
-    "        # --- 5a. Perform Grid Search ---\n",
-    "        print(\"Performing Grid Search to find the best hyperparameters...\")\n",
-    "        parameters = {\n",
-    "            'tfidf__ngram_range': [(1, 1), (1, 2)],\n",
-    "            'tfidf__max_df': [0.5, 0.75, 1.0],\n",
-    "            'nb__alpha': [0.1, 0.5, 1.0],\n",
-    "        }\n",
-    "        param_grid = list(ParameterGrid(parameters))\n",
-    "        best_score = -1\n",
-    "\n",
-    "        for params in tqdm(param_grid, desc=\"Grid Search Progress\"):\n",
-    "            pipeline.set_params(**params)\n",
-    "            pipeline.fit(X_train, y_train)\n",
-    "            score = pipeline.score(X_test, y_test)\n",
-    "            if score > best_score:\n",
-    "                best_score = score\n",
-    "                best_params = params\n",
-    "        \n",
-    "        print(f\"\\nBest score on test set: {best_score:.4f}\")\n",
-    "        print(\"Best parameters found:\")\n",
-    "        print(best_params)\n",
-    "\n",
-    "    else:\n",
-    "        # --- 5b. Use provided hyperparameters ---\n",
-    "        print(\"Skipping grid search and using provided hyperparameters...\")\n",
-    "        best_params = {\n",
-    "            'nb__alpha': nb__alpha,\n",
-    "            'tfidf__max_df': tfidf__max_df,\n",
-    "            'tfidf__ngram_range': tfidf__ngram_range\n",
-    "        }\n",
-    "\n",
-    "    # --- 6. Train the Final Model ---\n",
-    "    print(\"\\nTraining final model...\")\n",
-    "    best_model = pipeline.set_params(**best_params)\n",
-    "    best_model.fit(X_train, y_train)\n",
-    "    print(\"Model training complete.\")\n",
-    "\n",
-    "    # --- 7. Evaluate the Best Model ---\n",
-    "    print(\"\\n--- Model Evaluation ---\")\n",
-    "    y_pred = best_model.predict(X_test)\n",
-    "    \n",
-    "    accuracy = accuracy_score(y_test, y_pred)\n",
-    "    target_names = ['Negative', 'Positive']\n",
-    "    \n",
-    "    print(f\"Accuracy: {accuracy:.4f}\")\n",
-    "    print(\"\\nClassification Report:\")\n",
-    "    print(classification_report(y_test, y_pred, target_names=target_names))\n",
-    "    \n",
-    "    print(\"Confusion Matrix:\")\n",
-    "    cm = confusion_matrix(y_test, y_pred)\n",
-    "    plt.figure(figsize=(8, 6))\n",
-    "    sns.heatmap(cm, annot=True, fmt='d', cmap='Greens', \n",
-    "                xticklabels=target_names, yticklabels=target_names)\n",
-    "    plt.title('Confusion Matrix for Naive Bayes on Amazon Reviews')\n",
-    "    plt.xlabel('Predicted Label')\n",
-    "    plt.ylabel('True Label')\n",
-    "    plt.show()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "093e6ae9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#--- Train the base model ---\n",
-    "train_baseline_sentiment_model(sample_size=150000, grid_search=False)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "71f5e4ba",
-   "metadata": {},
-   "source": [
-    "## Define the dataset and lightning DataModule"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c977e0f4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class ReviewDataset(Dataset):\n",
-    "    \"\"\"\n",
-    "    Custom PyTorch Dataset for Amazon Reviews.\n",
-    "\n",
-    "    This class takes a pandas DataFrame of review data, a tokenizer, and a max\n",
-    "    token length, and prepares it for use in a PyTorch model. It handles the\n",
-    "    tokenization of the text and the formatting of the labels for each item.\n",
-    "\n",
-    "    Attributes:\n",
-    "        tokenizer: The Hugging Face tokenizer to use for processing text.\n",
-    "        data (pd.DataFrame): The DataFrame containing the review data.\n",
-    "        max_token_len (int): The maximum sequence length for the tokenizer.\n",
-    "    \"\"\"\n",
-    "    def __init__(self, data: pd.DataFrame, tokenizer, max_token_len: int):\n",
-    "        \"\"\"\n",
-    "        Initializes the ReviewDataset.\n",
-    "\n",
-    "        Args:\n",
-    "            data (pd.DataFrame): The input DataFrame containing 'full_text' and\n",
-    "                                 'sentiment' columns.\n",
-    "            tokenizer: The pre-trained tokenizer instance.\n",
-    "            max_token_len (int): The maximum length for tokenized sequences.\n",
-    "        \"\"\"\n",
-    "        self.tokenizer = tokenizer\n",
-    "        self.data = data\n",
-    "        self.max_token_len = max_token_len\n",
-    "\n",
-    "    def __len__(self):\n",
-    "        \"\"\"\n",
-    "        Returns the total number of samples in the dataset.\n",
-    "        \"\"\"\n",
-    "        return len(self.data)\n",
-    "\n",
-    "    def __getitem__(self, index: int):\n",
-    "        \"\"\"\n",
-    "        Retrieves one sample from the dataset at the specified index.\n",
-    "\n",
-    "        This method handles the tokenization of a single review text, including\n",
-    "        padding and truncation, and formats the output into a dictionary of\n",
-    "        tensors ready for the model.\n",
-    "\n",
-    "        Args:\n",
-    "            index (int): The index of the data sample to retrieve.\n",
-    "\n",
-    "        Returns:\n",
-    "            dict: A dictionary containing the tokenized inputs and the label,\n",
-    "                  with the following keys:\n",
-    "                  - 'input_ids': The token IDs of the review text.\n",
-    "                  - 'attention_mask': The attention mask for the review text.\n",
-    "                  - 'labels': The sentiment label as a tensor.\n",
-    "        \"\"\"\n",
-    "        data_row = self.data.iloc[index]\n",
-    "        text = str(data_row.full_text)\n",
-    "        labels = data_row.sentiment\n",
-    "\n",
-    "        encoding = self.tokenizer.encode_plus(\n",
-    "            text,\n",
-    "            add_special_tokens=True,\n",
-    "            max_length=self.max_token_len,\n",
-    "            return_token_type_ids=False,\n",
-    "            padding=\"max_length\",\n",
-    "            truncation=True,\n",
-    "            return_attention_mask=True,\n",
-    "            return_tensors='pt',\n",
-    "        )\n",
-    "\n",
-    "        return dict(\n",
-    "            input_ids=encoding[\"input_ids\"].flatten(),\n",
-    "            attention_mask=encoding[\"attention_mask\"].flatten(),\n",
-    "            labels=torch.tensor(labels, dtype=torch.long)\n",
-    "        )\n",
-    "\n",
-    "class ReviewDataModule(pl.LightningDataModule):\n",
-    "    \"\"\"\n",
-    "    PyTorch Lightning DataModule to handle the Amazon Reviews dataset.\n",
-    "\n",
-    "    This class encapsulates all the steps needed to process the data:\n",
-    "    loading, splitting, and creating PyTorch DataLoaders for training,\n",
-    "    validation, and testing. It allows for using a smaller random sample of the\n",
-    "    full dataset for faster experimentation.\n",
-    "\n",
-    "    Attributes:\n",
-    "        data_path (str): Path to the processed CSV file.\n",
-    "        batch_size (int): The size of each data batch.\n",
-    "        max_token_len (int): The maximum sequence length for the tokenizer.\n",
-    "        tokenizer: The Hugging Face tokenizer instance.\n",
-    "        num_workers (int): The number of CPU cores to use for data loading.\n",
-    "        sample_size (int, optional): The number of samples to use. If None,\n",
-    "                                     the full dataset is used.\n",
-    "    \"\"\"\n",
-    "    def __init__(self, data_path: str, batch_size: int = 16, max_token_len: int = 256, model_name='distilbert-base-uncased', num_workers: int = 0, sample_size: int = None):\n",
-    "        \"\"\"\n",
-    "        Initializes the ReviewDataModule.\n",
-    "\n",
-    "        Args:\n",
-    "            data_path (str): The path to the processed CSV data file.\n",
-    "            batch_size (int): The number of samples per batch.\n",
-    "            max_token_len (int): Maximum length of tokenized sequences.\n",
-    "            model_name (str): The name of the pre-trained model to use for the tokenizer.\n",
-    "            num_workers (int): Number of subprocesses to use for data loading.\n",
-    "            sample_size (int, optional): If specified, a random sample of this\n",
-    "                                         size will be used from the dataset.\n",
-    "                                         Defaults to None, which uses the full dataset.\n",
-    "        \"\"\"\n",
-    "        super().__init__()\n",
-    "        self.data_path = data_path\n",
-    "        self.batch_size = batch_size\n",
-    "        self.max_token_len = max_token_len\n",
-    "        self.tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
-    "        self.num_workers = num_workers\n",
-    "        self.sample_size = sample_size\n",
-    "        self.train_df = None\n",
-    "        self.val_df = None\n",
-    "        self.test_df = None\n",
-    "\n",
-    "    def setup(self, stage=None):\n",
-    "        \"\"\"\n",
-    "        Loads and splits the data for training, validation, and testing.\n",
-    "\n",
-    "        This method is called by PyTorch Lightning. It reads the CSV, handles\n",
-    "        missing values, optionally takes a random sample, and performs a\n",
-    "        stratified train-validation-test split. The indices of the resulting\n",
-    "        DataFrames are reset to prevent potential KeyErrors during data loading.\n",
-    "        \"\"\"\n",
-    "        df = pd.read_csv(self.data_path)\n",
-    "        df.dropna(inplace=True)\n",
-    "\n",
-    "        # If a sample size is provided, sample the dataframe\n",
-    "        if self.sample_size:\n",
-    "            print(f\"Using a sample of {self.sample_size} reviews.\")\n",
-    "            df = df.sample(n=self.sample_size, random_state=42)\n",
-    "\n",
-    "        # Stratified split to maintain label distribution\n",
-    "        train_val_df, self.test_df = train_test_split(df, test_size=0.1, random_state=42, stratify=df.sentiment)\n",
-    "        self.train_df, self.val_df = train_test_split(train_val_df, test_size=0.1, random_state=42, stratify=train_val_df.sentiment)\n",
-    "\n",
-    "        # Reset indices to prevent KeyErrors\n",
-    "        self.train_df = self.train_df.reset_index(drop=True)\n",
-    "        self.val_df = self.val_df.reset_index(drop=True)\n",
-    "        self.test_df = self.test_df.reset_index(drop=True)\n",
-    "\n",
-    "        print(f\"Size of training set: {len(self.train_df)}\")\n",
-    "        print(f\"Size of validation set: {len(self.val_df)}\")\n",
-    "        print(f\"Size of test set: {len(self.test_df)}\")\n",
-    "\n",
-    "    def train_dataloader(self):\n",
-    "        \"\"\"Returns the DataLoader for the training set.\"\"\"\n",
-    "        return DataLoader(\n",
-    "            ReviewDataset(self.train_df, self.tokenizer, self.max_token_len),\n",
-    "            batch_size=self.batch_size,\n",
-    "            shuffle=True,\n",
-    "            num_workers=self.num_workers\n",
-    "        )\n",
-    "\n",
-    "    def val_dataloader(self):\n",
-    "        \"\"\"Returns the DataLoader for the validation set.\"\"\"\n",
-    "        return DataLoader(\n",
-    "            ReviewDataset(self.val_df, self.tokenizer, self.max_token__len),\n",
-    "            batch_size=self.batch_size,\n",
-    "            num_workers=self.num_workers\n",
-    "        )\n",
-    "\n",
-    "    def test_dataloader(self):\n",
-    "        \"\"\"Returns the DataLoader for the test set.\"\"\"\n",
-    "        return DataLoader(\n",
-    "            ReviewDataset(self.test_df, self.tokenizer, self.max_token_len),\n",
-    "            batch_size=self.batch_size,\n",
-    "            num_workers=self.num_workers\n",
-    "        )\n",
-    "        "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "985ac47b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# --- Configuration ---\n",
-    "data_path = \"data/reviews_processed.csv\"\n",
-    "BATCH_SIZE = 64\n",
-    "MAX_TOKEN_LEN = 256\n",
-    "\n",
-    "print(\"Initializing ReviewDataModule...\")\n",
-    "review_datamodule = ReviewDataModule(\n",
-    "    data_path=data_path,\n",
-    "    batch_size=BATCH_SIZE,\n",
-    "    max_token_len=MAX_TOKEN_LEN,\n",
-    "    model_name='distilbert-base-uncased',\n",
-    "    sample_size=100000 # Pass the sample size to the datamodule\n",
-    ")\n",
-    "review_datamodule.setup()\n",
-    "\n",
-    "# Fetch one batch from the training dataloader to inspect its contents\n",
-    "print(\"\\n--- Fetching one batch from the training dataloader ---\")\n",
-    "train_batch = next(iter(review_datamodule.train_dataloader()))\n",
-    "\n",
-    "print(\"\\n--- Example Batch ---\")\n",
-    "print(f\"Input IDs shape: {train_batch['input_ids'].shape}\")\n",
-    "print(f\"Attention Mask shape: {train_batch['attention_mask'].shape}\")\n",
-    "print(f\"Labels: {train_batch['labels']}\")\n",
-    "print(f\"Labels shape: {train_batch['labels'].shape}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2c7781f4",
-   "metadata": {},
-   "source": [
-    "## FineTune DistilBert"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d046b940",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class SentimentClassifier(pl.LightningModule):\n",
-    "    \"\"\"\n",
-    "    PyTorch Lightning module for the sentiment classification model.\n",
-    "    \"\"\"\n",
-    "    def __init__(self, model_name='distilbert-base-uncased', n_classes=2, learning_rate=2e-5, n_warmup_steps=0, n_training_steps=0, dropout_prob=0.2): # Added dropout\n",
-    "        super().__init__()\n",
-    "        self.save_hyperparameters()\n",
-    "\n",
-    "        # Configure dropout\n",
-    "        config = AutoConfig.from_pretrained(model_name)\n",
-    "        config.hidden_dropout_prob = dropout_prob\n",
-    "        config.attention_probs_dropout_prob = dropout_prob\n",
-    "        config.num_labels = n_classes\n",
-    "\n",
-    "        self.model = AutoModelForSequenceClassification.from_pretrained(model_name, config=config)\n",
-    "\n",
-    "    def forward(self, input_ids, attention_mask, labels=None):\n",
-    "        return self.model(\n",
-    "            input_ids=input_ids,\n",
-    "            attention_mask=attention_mask,\n",
-    "            labels=labels\n",
-    "        )\n",
-    "\n",
-    "    def training_step(self, batch, batch_idx):\n",
-    "        output = self.forward(**batch)\n",
-    "        self.log(\"train_loss\", output.loss, prog_bar=True, logger=True)\n",
-    "        return output.loss\n",
-    "\n",
-    "    def validation_step(self, batch, batch_idx):\n",
-    "        output = self.forward(**batch)\n",
-    "        preds = torch.argmax(output.logits, dim=1)\n",
-    "        val_acc = accuracy(preds, batch['labels'], task='binary')\n",
-    "        self.log(\"val_loss\", output.loss, prog_bar=True, logger=True)\n",
-    "        self.log(\"val_accuracy\", val_acc, prog_bar=True, logger=True)\n",
-    "        return output.loss\n",
-    "\n",
-    "    def test_step(self, batch, batch_idx):\n",
-    "        output = self.forward(**batch)\n",
-    "        preds = torch.argmax(output.logits, dim=1)\n",
-    "        test_acc = accuracy(preds, batch['labels'], task='binary')\n",
-    "        self.log(\"test_accuracy\", test_acc)\n",
-    "        return test_acc\n",
-    "\n",
-    "    def predict_step(self, batch, batch_idx, dataloader_idx=0):\n",
-    "        output = self.forward(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'])\n",
-    "        return torch.argmax(output.logits, dim=1)\n",
-    "\n",
-    "    def configure_optimizers(self):\n",
-    "        optimizer = AdamW(self.parameters(), lr=self.hparams.learning_rate, weight_decay=0.01)\n",
-    "        scheduler = get_linear_schedule_with_warmup(\n",
-    "            optimizer,\n",
-    "            num_warmup_steps=self.hparams.n_warmup_steps,\n",
-    "            num_training_steps=self.hparams.n_training_steps\n",
-    "        )\n",
-    "        return dict(optimizer=optimizer, lr_scheduler=dict(scheduler=scheduler, interval='step'))\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b3a3708d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def train_sentiment_model(data_path='data/reviews_processed.csv', model_name='distilbert-base-uncased', n_epochs=5, sample_size: int = None):\n",
-    "    \"\"\"\n",
-    "    Main function to train the sentiment analysis model on the Amazon Reviews dataset.\n",
-    "\n",
-    "    Args:\n",
-    "        data_path (str): Path to the processed data file.\n",
-    "        model_name (str): Name of the transformer model to use.\n",
-    "        n_epochs (int): Maximum number of epochs for training.\n",
-    "        sample_size (int, optional): The number of reviews to use for training.\n",
-    "                                     If None, the full dataset is used.\n",
-    "    \"\"\"\n",
-    "    # --- 1. Hyperparameters ---\n",
-    "    BATCH_SIZE = 64\n",
-    "    MAX_TOKEN_LEN = 256\n",
-    "    LEARNING_RATE = 2e-5\n",
-    "    N_CLASSES = 2  # Negative, Positive\n",
-    "\n",
-    "    # --- 2. Initialize DataModule ---\n",
-    "    print(\"Initializing ReviewDataModule...\")\n",
-    "    review_datamodule = ReviewDataModule(\n",
-    "        data_path=data_path,\n",
-    "        batch_size=BATCH_SIZE,\n",
-    "        max_token_len=MAX_TOKEN_LEN,\n",
-    "        model_name=model_name,\n",
-    "        sample_size=sample_size # Pass the sample size to the datamodule\n",
-    "    )\n",
-    "    review_datamodule.setup()\n",
-    "\n",
-    "    n_training_steps = len(review_datamodule.train_dataloader()) * n_epochs\n",
-    "    n_warmup_steps = int(n_training_steps * 0.1)\n",
-    "\n",
-    "    # --- 3. Initialize Model ---\n",
-    "    print(\"Initializing SentimentClassifier model...\")\n",
-    "    model = SentimentClassifier(\n",
-    "        model_name=model_name,\n",
-    "        n_classes=N_CLASSES,\n",
-    "        learning_rate=LEARNING_RATE,\n",
-    "        n_warmup_steps=n_warmup_steps,\n",
-    "        n_training_steps=n_training_steps\n",
-    "    )\n",
-    "\n",
-    "    # --- 4. Configure Training Callbacks ---\n",
-    "    checkpoint_callback = ModelCheckpoint(\n",
-    "        dirpath=\"checkpoints\",\n",
-    "        filename=\"sentiment-binary-best-checkpoint\",\n",
-    "        save_top_k=1,\n",
-    "        verbose=True,\n",
-    "        monitor=\"val_loss\",\n",
-    "        mode=\"min\"\n",
-    "    )\n",
-    "    logger = TensorBoardLogger(\"lightning_logs\", name=\"sentiment-classifier-binary\")\n",
-    "    early_stopping_callback = EarlyStopping(monitor='val_loss', patience=2)\n",
-    "\n",
-    "    # --- 5. Initialize Trainer ---\n",
-    "    print(\"Initializing PyTorch Lightning Trainer...\")\n",
-    "    trainer = pl.Trainer(\n",
-    "        logger=logger,\n",
-    "        callbacks=[checkpoint_callback, early_stopping_callback],\n",
-    "        max_epochs=n_epochs,\n",
-    "        accelerator='gpu' if torch.cuda.is_available() else 'cpu',\n",
-    "        devices=1,\n",
-    "    )\n",
-    "\n",
-    "    # --- 6. Start Training ---\n",
-    "    print(f\"Starting training with {model_name} for up to {n_epochs} epochs...\")\n",
-    "    trainer.fit(model, review_datamodule)\n",
-    "\n",
-    "    # --- 7. Evaluate on Test Set and Generate Confusion Matrix ---\n",
-    "    print(\"\\nTraining complete. Evaluating on the test set...\")\n",
-    "    trainer.test(model, datamodule=review_datamodule)\n",
-    "\n",
-    "    predictions = trainer.predict(model, datamodule=review_datamodule)\n",
-    "    if predictions:\n",
-    "        all_preds = torch.cat(predictions).cpu().numpy()\n",
-    "        true_labels = review_datamodule.test_df.sentiment.to_numpy()\n",
-    "        target_names = ['Negative', 'Positive'] # Updated labels\n",
-    "\n",
-    "        cm = confusion_matrix(true_labels, all_preds)\n",
-    "        plt.figure(figsize=(8, 6))\n",
-    "        sns.heatmap(cm, annot=True, fmt='d', cmap='YlGnBu',\n",
-    "                    xticklabels=target_names, yticklabels=target_names)\n",
-    "        plt.title('Confusion Matrix for Sentiment Analysis')\n",
-    "        plt.xlabel('Predicted Label')\n",
-    "        plt.ylabel('True Label')\n",
-    "        plt.show()\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "3dae58e3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#--- Train DistilBert ---\n",
-    "train_sentiment_model(data_path=data_path, sample_size=100000)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ddbc7315",
-   "metadata": {},
-   "source": [
-    "## Define the models"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "85bd352b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class ReviewSummarizer:\n",
-    "    \"\"\"\n",
-    "    A class to handle the summarization of product reviews using a pre-trained T5 model.\n",
-    "    \"\"\"\n",
-    "    def __init__(self, model_name='t5-small'):\n",
-    "        \"\"\"\n",
-    "        Initializes the summarizer with a pre-trained T5 model and tokenizer.\n",
-    "\n",
-    "        Args:\n",
-    "            model_name (str): The name of the pre-trained T5 model to use.\n",
-    "        \"\"\"\n",
-    "        print(f\"Loading summarization model: {model_name}...\")\n",
-    "        self.model_name = model_name\n",
-    "        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
-    "\n",
-    "        # Load the tokenizer and model from Hugging Face\n",
-    "        self.tokenizer = T5Tokenizer.from_pretrained(self.model_name)\n",
-    "        self.model = T5ForConditionalGeneration.from_pretrained(self.model_name).to(self.device)\n",
-    "        print(\"Summarization model loaded successfully.\")\n",
-    "\n",
-    "    def summarize(self, text: str, max_length: int = 50, min_length: int = 10) -> str:\n",
-    "        \"\"\"\n",
-    "        Generates a summary for a given text.\n",
-    "\n",
-    "        Args:\n",
-    "            text (str): The review text to summarize.\n",
-    "            max_length (int): The maximum length of the generated summary.\n",
-    "            min_length (int): The minimum length of the generated summary.\n",
-    "\n",
-    "        Returns:\n",
-    "            str: The generated summary.\n",
-    "        \"\"\"\n",
-    "        if not text or not isinstance(text, str):\n",
-    "            return \"\"\n",
-    "\n",
-    "        # T5 models require a prefix for the task. For summarization, it's \"summarize: \"\n",
-    "        preprocess_text = f\"summarize: {text.strip()}\"\n",
-    "\n",
-    "        # Tokenize the input text\n",
-    "        tokenized_text = self.tokenizer.encode(preprocess_text, return_tensors=\"pt\").to(self.device)\n",
-    "\n",
-    "        # Generate the summary\n",
-    "        summary_ids = self.model.generate(\n",
-    "            tokenized_text,\n",
-    "            max_length=max_length,\n",
-    "            min_length=min_length,\n",
-    "            length_penalty=2.0,\n",
-    "            num_beams=4,\n",
-    "            early_stopping=True\n",
-    "        )\n",
-    "\n",
-    "        # Decode the summary and return it\n",
-    "        summary = self.tokenizer.decode(summary_ids[0], skip_special_tokens=True)\n",
-    "        return summary\n",
-    "\n",
-    "class AspectAnalyzer:\n",
-    "    \"\"\"\n",
-    "    A class to handle Aspect-Based Sentiment Analysis (ABSA) using a pre-trained model.\n",
-    "    \"\"\"\n",
-    "    # Changed to a different, currently valid lightweight model for ABSA.\n",
-    "    def __init__(self, model_name='yangheng/deberta-v3-base-absa-v1.1', force_cpu=False):\n",
-    "        \"\"\"\n",
-    "        Initializes the ABSA pipeline with a pre-trained model.\n",
-    "\n",
-    "        Args:\n",
-    "            model_name (str): The name of the pre-trained ABSA model.\n",
-    "            force_cpu (bool): If True, forces the model to run on the CPU.\n",
-    "        \"\"\"\n",
-    "        print(f\"Loading Aspect-Based Sentiment Analysis model: {model_name}...\")\n",
-    "        self.model_name = model_name\n",
-    "\n",
-    "        if force_cpu:\n",
-    "            self.device = -1 # Use -1 for CPU in pipeline\n",
-    "            print(\"Forcing ABSA model to run on CPU.\")\n",
-    "        else:\n",
-    "            self.device = 0 if torch.cuda.is_available() else -1\n",
-    "\n",
-    "        print(f\"Using device: {self.device} (0 for GPU, -1 for CPU)\")\n",
-    "\n",
-    "        self.absa_pipeline = pipeline(\n",
-    "            \"text-classification\",\n",
-    "            model=self.model_name,\n",
-    "            tokenizer=self.model_name,\n",
-    "            device=self.device\n",
-    "        )\n",
-    "        print(\"ABSA model loaded successfully.\")\n",
-    "\n",
-    "    def analyze(self, text: str, aspects: list) -> dict:\n",
-    "        \"\"\"\n",
-    "        Analyzes the sentiment towards a list of aspects within a given text.\n",
-    "        \"\"\"\n",
-    "        if not text or not isinstance(text, str) or not aspects:\n",
-    "            return {}\n",
-    "\n",
-    "        # The model expects the review and aspect separated by a special token.\n",
-    "        # Note: Different ABSA models might expect different input formats.\n",
-    "        # This format is common but may need adjustment for other models.\n",
-    "        inputs = [f\"{text} [SEP] {aspect}\" for aspect in aspects]\n",
-    "        results = self.absa_pipeline(inputs)\n",
-    "\n",
-    "        # Process results into a user-friendly dictionary\n",
-    "        aspect_sentiments = {}\n",
-    "        for aspect, result in zip(aspects, results):\n",
-    "            aspect_sentiments[aspect] = {'sentiment': result['label'], 'score': result['score']}\n",
-    "\n",
-    "        return aspect_sentiments\n",
-    "\n",
-    "class FineTunedSentimentClassifier:\n",
-    "    \"\"\"\n",
-    "    This class handles loading the fine-tuned checkpoint and making predictions.\n",
-    "    \"\"\"\n",
-    "    def __init__(self, checkpoint_path, model_name='distilbert-base-uncased', force_cpu=False):\n",
-    "        self.device = 'cpu' if force_cpu else ('cuda' if torch.cuda.is_available() else 'cpu')\n",
-    "        print(f\"Loading fine-tuned sentiment model from checkpoint: {checkpoint_path}...\")\n",
-    "        print(f\"Using device: {self.device}\")\n",
-    "\n",
-    "        self.model = SentimentClassifier.load_from_checkpoint(checkpoint_path, map_location=self.device)\n",
-    "        self.model.to(self.device)\n",
-    "        self.model.eval() # Set model to evaluation mode\n",
-    "\n",
-    "        self.tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
-    "        self.labels = ['NEGATIVE', 'POSITIVE']\n",
-    "        print(\"Fine-tuned sentiment model loaded successfully.\")\n",
-    "\n",
-    "    def classify(self, text: str) -> dict:\n",
-    "        encoding = self.tokenizer.encode_plus(\n",
-    "            text, add_special_tokens=True, max_length=128,\n",
-    "            return_token_type_ids=False, padding=\"max_length\",\n",
-    "            truncation=True, return_attention_mask=True, return_tensors='pt',\n",
-    "        )\n",
-    "        input_ids = encoding[\"input_ids\"].to(self.device)\n",
-    "        attention_mask = encoding[\"attention_mask\"].to(self.device)\n",
-    "        with torch.no_grad():\n",
-    "            outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)\n",
-    "        logits = outputs.logits\n",
-    "        probabilities = torch.softmax(logits, dim=1)\n",
-    "        prediction_idx = torch.argmax(probabilities, dim=1).item()\n",
-    "        return {'label': self.labels[prediction_idx], 'score': probabilities[0][prediction_idx].item()}\n",
-    "\n",
-    "class AspectExtractor:\n",
-    "    \"\"\"\n",
-    "    This class uses a Part-of-Speech (POS) tagging model to first extract all\n",
-    "    potential aspect terms (nouns) from a review text. It then filters these\n",
-    "    nouns against a pre-defined dictionary of valid aspects for a given\n",
-    "    product category to return only the relevant features.\n",
-    "    \"\"\"\n",
-    "    def __init__(self, model_name=\"vblagoje/bert-english-uncased-finetuned-pos\", force_cpu=False):\n",
-    "        self.model_name = model_name\n",
-    "        self.device = 'cpu' if force_cpu else ('cuda' if torch.cuda.is_available() else 'cpu')\n",
-    "        print(f\"Loading Part-of-Speech (POS) tagging model: {self.model_name}...\")\n",
-    "        print(f\"Using device: {self.device}\")\n",
-    "\n",
-    "        self.pipeline = pipeline(\n",
-    "            \"token-classification\",\n",
-    "            model=self.model_name,\n",
-    "            device=-1 if self.device == 'cpu' else 0,\n",
-    "            aggregation_strategy=\"simple\"\n",
-    "        )\n",
-    "        print(\"POS tagging model loaded successfully.\")\n",
-    "\n",
-    "    def extract(self, text: str, aspect_dictionary: list) -> list:\n",
-    "        \"\"\"\n",
-    "        Extracts aspects from the given text that are present in the provided\n",
-    "        aspect dictionary.\n",
-    "\n",
-    "        Args:\n",
-    "            text (str): The review text to analyze.\n",
-    "            aspect_dictionary (list): A list of valid, known aspects for the\n",
-    "                                      product category.\n",
-    "\n",
-    "        Returns:\n",
-    "            list: A list of aspects that were both found in the text and are\n",
-    "                  present in the aspect dictionary.\n",
-    "        \"\"\"\n",
-    "        if not text or not aspect_dictionary:\n",
-    "            return []\n",
-    "\n",
-    "        # 1. Extract all nouns from the text using the POS model\n",
-    "        model_outputs = self.pipeline(text)\n",
-    "        noun_tags = {'NOUN', 'PROPN'}\n",
-    "        extracted_nouns = {\n",
-    "            output['word'].lower() for output in model_outputs\n",
-    "            if output['entity_group'] in noun_tags\n",
-    "        }\n",
-    "\n",
-    "        # 2. Filter the extracted nouns against the provided dictionary\n",
-    "        # We find the intersection between the two sets.\n",
-    "        valid_aspects = {aspect.lower() for aspect in aspect_dictionary}\n",
-    "\n",
-    "        final_aspects = list(extracted_nouns.intersection(valid_aspects))\n",
-    "\n",
-    "        return final_aspects\n",
-    "    "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6fc21c8b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# --- Configuration ---\n",
-    "# --- IMPORTANT: UPDATE THIS PATH ---\n",
-    "# You need to provide the path to the best checkpoint file that was saved\n",
-    "# during the training of your sentiment model.\n",
-    "SENTIMENT_CHECKPOINT_PATH = \"checkpoints/sentiment-binary-best-checkpoint.ckpt\"\n",
-    "\n",
-    "# --- Pre-defined Aspect Dictionaries for Different Product Categories ---\n",
-    "ASPECT_DICTIONARIES = {\n",
-    "    \"Phone\": ['camera', 'battery', 'battery life', 'screen', 'performance', 'price', 'design'],\n",
-    "    \"Coffee Maker\": ['ease of use', 'design', 'noise level', 'coffee quality', 'brew time', 'cleaning'],\n",
-    "    \"Book\": ['plot', 'characters', 'writing style', 'pacing', 'ending'],\n",
-    "    \"Default\": ['quality', 'price', 'service', 'design', 'features'] # A fallback list\n",
-    "}\n",
-    "\n",
-    "def main():\n",
-    "    \"\"\"\n",
-    "    Main function to run the command-line review analysis tool.\n",
-    "    \"\"\"\n",
-    "    # --- 1. Load All Models ---\n",
-    "    print(\"--- Initializing all models ---\")\n",
-    "    sentiment_classifier, summarizer, aspect_analyzer, aspect_extractor = None, None, None, None\n",
-    "    try:\n",
-    "        summarizer = ReviewSummarizer(force_cpu=True)\n",
-    "        aspect_analyzer = AspectAnalyzer(force_cpu=True)\n",
-    "        aspect_extractor = AspectExtractor(force_cpu=True)\n",
-    "\n",
-    "        if not os.path.exists(SENTIMENT_CHECKPOINT_PATH):\n",
-    "            print(\"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\")\n",
-    "            print(\"!!! WARNING: Sentiment checkpoint path not found or not set.         !!!\")\n",
-    "            print(f\"!!! Please update the 'SENTIMENT_CHECKPOINT_PATH' variable in main.py\")\n",
-    "            print(\"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\")\n",
-    "        else:\n",
-    "            sentiment_classifier = FineTunedSentimentClassifier(\n",
-    "                checkpoint_path=SENTIMENT_CHECKPOINT_PATH, force_cpu=True\n",
-    "            )\n",
-    "        print(\"\\n--- All models loaded successfully ---\\n\")\n",
-    "    except Exception as e:\n",
-    "        print(f\"An error occurred during model initialization: {e}\")\n",
-    "        return\n",
-    "\n",
-    "    # --- 2. Interactive Loop ---\n",
-    "    while True:\n",
-    "        print(\"\\n==================================================\")\n",
-    "        print(\"          Product Review Analysis Tool          \")\n",
-    "        print(\"==================================================\")\n",
-    "\n",
-    "        # Get user input\n",
-    "        review_text = input(\"Enter the product review text (or type 'quit' to exit):\\n> \")\n",
-    "        if review_text.lower() == 'quit':\n",
-    "            break\n",
-    "\n",
-    "        print(\"\\nAvailable Product Categories:\")\n",
-    "        for i, category in enumerate(ASPECT_DICTIONARIES.keys(), 1):\n",
-    "            print(f\"{i}. {category}\")\n",
-    "\n",
-    "        category_choice = input(f\"Select a product category (1-{len(ASPECT_DICTIONARIES)}):\\n> \")\n",
-    "        try:\n",
-    "            category_idx = int(category_choice) - 1\n",
-    "            product_category = list(ASPECT_DICTIONARIES.keys())[category_idx]\n",
-    "        except (ValueError, IndexError):\n",
-    "            print(\"Invalid choice. Using 'Default' category.\")\n",
-    "            product_category = \"Default\"\n",
-    "\n",
-    "        # --- 3. Run Analysis ---\n",
-    "        print(\"\\n--- Analyzing Review... ---\")\n",
-    "\n",
-    "        # a. Overall Sentiment\n",
-    "        sentiment_result = sentiment_classifier.classify(review_text)\n",
-    "\n",
-    "        # b. Summary\n",
-    "        summary_result = summarizer.summarize(review_text)\n",
-    "\n",
-    "        # c. Aspect Extraction and Analysis\n",
-    "        aspect_dictionary = ASPECT_DICTIONARIES.get(product_category)\n",
-    "        extracted_aspects = aspect_extractor.extract(review_text, aspect_dictionary)\n",
-    "        aspect_results = None\n",
-    "        if extracted_aspects:\n",
-    "            aspect_results = aspect_analyzer.analyze(review_text, extracted_aspects)\n",
-    "\n",
-    "        # --- 4. Display Results ---\n",
-    "        print(\"\\n-------------------- ANALYSIS RESULTS --------------------\")\n",
-    "        print(f\"\\n[ Overall Sentiment ]\")\n",
-    "        print(f\"  - Sentiment: {sentiment_result['label']} (Score: {sentiment_result['score']:.2f})\")\n",
-    "\n",
-    "        print(f\"\\n[ Generated Summary ]\")\n",
-    "        print(f\"  - {summary_result}\")\n",
-    "\n",
-    "        print(f\"\\n[ Detected Aspect Sentiments ]\")\n",
-    "        if aspect_results:\n",
-    "            for aspect, result in aspect_results.items():\n",
-    "                print(f\"  - {aspect.title()}: {result['sentiment']} (Score: {result['score']:.2f})\")\n",
-    "        else:\n",
-    "            print(\"  - No relevant aspects from the dictionary were detected in the review.\")\n",
-    "        print(\"----------------------------------------------------------\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "71257428",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# --- Run the workflow ---\n",
-    "main()"
-   ]
-  }
- ],
- "metadata": {
-  "language_info": {
-   "name": "python"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

requirements.txt

@@ -1,11 +1,16 @@
-torch==2.8.0
-transformers==4.56.1
-pytorch-lightning==2.5.5
-torchmetrics==1.8.2
-sentencepiece==0.2.1
+langchain==0.3.27
+langchain-community==0.3.31
+gradio==5.49.1
+llama_cpp_python==0.3.16
+sentence-transformers==5.1.1
+torch==2.8.0 
+transformers==4.57.1
+faiss-cpu==1.12.0
+ragas==0.3.7
+openai==1.109.1
 pandas==2.2.2
-scikit-learn==1.6.1
-gradio==5.44.1
-matplotlib==3.10.0
-seaborn==0.13.2
-wordcloud==1.9.4
+datasets==4.0.0
+numpy==2.0.2
+accelerate==1.11.0
+aiohttp==3.13.1
+huggingface-hub==0.35.3

scripts/app.py

@@ -1,163 +1,280 @@
+# app.py
+
 import gradio as gr
-import os
 import torch
-from transformers import AutoTokenizer
-import pandas as pd
-import re
-
-# --- IMPORTANT ---
-# This script assumes you have a 'models.py' file in the same directory
-# containing the definitions for all model and inference classes.
-try:
-    from models import (
-        ReviewSummarizer,
-        AspectAnalyzer,
-        AspectExtractor,
-        FineTunedSentimentClassifier
-    )
-except ImportError:
-    print("CRITICAL ERROR: Make sure 'models.py' exists and contains the required classes.")
-    # Define dummy classes if imports fail, so Gradio can at least launch with an error message.
-    class ReviewSummarizer: pass
-    class AspectAnalyzer: pass
-    class AspectExtractor: pass
-    class FineTunedSentimentClassifier: pass
-
-# --- Configuration ---
-# This should be the relative path to your checkpoint file within the repository.
-SENTIMENT_CHECKPOINT_PATH = "checkpoints/sentiment-binary-best-checkpoint.ckpt"
-
-
-# --- Pre-defined Aspect Dictionaries for Different Product Categories ---
-ASPECT_DICTIONARIES = {
-    "Phone": ['camera', 'battery', 'battery life', 'screen', 'performance', 'price', 'design'],
-    "Coffee Maker": ['ease of use', 'design', 'noise level', 'coffee quality', 'brew time', 'cleaning'],
-    "Book": ['plot', 'characters', 'writing style', 'pacing', 'ending'],
-    "Default": ['quality', 'price', 'service', 'design', 'features'] # A fallback list
-}
-
-
-# --- Load All Models (Global Objects) ---
-print("--- Initializing all models for the Gradio App ---")
-sentiment_classifier, summarizer, aspect_analyzer, aspect_extractor = None, None, None, None
-try:
-    summarizer = ReviewSummarizer(force_cpu=True)
-    aspect_analyzer = AspectAnalyzer(force_cpu=True)
-    aspect_extractor = AspectExtractor(force_cpu=True)
-
-    if os.path.exists(SENTIMENT_CHECKPOINT_PATH):
-        sentiment_classifier = FineTunedSentimentClassifier(
-            checkpoint_path=SENTIMENT_CHECKPOINT_PATH, force_cpu=True
-        )
-    else:
-        print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
-        print("!!! WARNING: Sentiment checkpoint path not found.                    !!!")
-        print(f"!!! Path checked: '{SENTIMENT_CHECKPOINT_PATH}'")
-        print("!!! The fine-tuned sentiment model will NOT be loaded.               !!!")
-        print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
-    print("\n--- All models loaded successfully ---\n")
-except Exception as e:
-    print(f"An error occurred during model initialization: {e}")
-
-
-# --- Define the Core Analysis Function ---
-def analyze_review(review_text, product_category):
-    if not review_text:
-        return {"ERROR": "Please enter a review."}, "", None
-
-    # --- a. Overall Sentiment Analysis ---
-    if sentiment_classifier:
-        sentiment_result = sentiment_classifier.classify(review_text)
-        sentiment_output = {
-            sentiment_result['label']: f"{sentiment_result['score']:.2f}"
-        }
-    else:
-        # **ROBUST ERROR HANDLING:** This prevents the app from crashing.
-        # It returns a dictionary that the Gradio Label component can display.
-        sentiment_output = {"Error: Model Not Loaded": 1.0}
+from langchain_community.embeddings import HuggingFaceEmbeddings
+from langchain_community.llms import LlamaCpp
+from langchain.memory import ConversationBufferMemory
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.prompts import PromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate, ChatPromptTemplate
+import os
+import io
 
-    # --- b. Review Summarization ---
-    if summarizer:
-        summary_output = summarizer.summarize(review_text)
-    else:
-        summary_output = "ERROR: Summarizer model not loaded."
+# Import the logic functions from src
+import src.pipeline as pipeline
 
-    # --- c. Dynamic Aspect Extraction & Analysis ---
-    aspect_df = None
-    if aspect_extractor and aspect_analyzer:
-        aspect_dictionary = ASPECT_DICTIONARIES.get(product_category, ASPECT_DICTIONARIES["Default"])
-        extracted_aspects = aspect_extractor.extract(review_text, aspect_dictionary=aspect_dictionary)
+# --- Global Objects & Setup ---
+# (Most setup code remains here as it's needed globally for the app)
 
-        if extracted_aspects:
-            aspect_results = aspect_analyzer.analyze(review_text, extracted_aspects)
-            aspect_df = pd.DataFrame([
-                {'Aspect': aspect, 'Sentiment': result['sentiment'], 'Score': f"{result['score']:.2f}"}
-                for aspect, result in aspect_results.items()
-            ])
+print("--- Starting App Setup ---")
+# 1. Download Model File
+model_name = "mistral-7b-instruct-v0.1.Q4_K_M.gguf"
+model_url = "https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q4_K_M.gguf"
+if not os.path.exists(model_name):
+    print("Downloading model...")
+    os.system(f"wget {model_url}")
+else:
+    print("Model already downloaded.")
 
-    return sentiment_output, summary_output, aspect_df
+# 2. Prepare Default Sample Data & Example Batch
+print("Loading default reviews...")
+default_reviews_text = """
+This laptop is a beast! The M3 chip is incredibly fast, and the battery lasts a solid 10 hours of heavy use... (rest of laptop reviews) ...dongle life is real.
+---
+I'm a student, and the battery life is a lifesaver... Highly recommend for college.
+---
+The keyboard is a dream to type on... Bluetooth connection dropping...
+---
+Video editing on this machine is flawless... price is very expensive...
+---
+I bought this for travel... battery easily gets me through a 6-hour flight...
+---
+Don't buy this if you need a lot of ports... only two USB-C ports...
+"""
+default_reviews_list = [r.strip() for r in default_reviews_text.strip().split('---') if r.strip()]
 
+example_batch = """
+I'm absolutely blown away by the "NovaBlend Pro" blender!... (rest of blender example)... save your money.
+"""
 
-# --- Build the Gradio Interface ---
-with gr.Blocks(theme=gr.themes.Soft()) as demo:
-    gr.Markdown("# 🛍️ ReviewSense: Product Review Analysis Engine")
-    gr.Markdown(
-        "Enter a product review and select the product category. The tool will automatically "
-        "detect relevant features and provide an overall sentiment score, a summary, and a "
-        "breakdown of sentiment towards each feature."
-    )
+# 3. Load Embedding Model, Text Splitter
+print("Loading embedding model and text splitter...")
+model_kwargs = {'device': 'cuda' if torch.cuda.is_available() else 'cpu'}
+embeddings = HuggingFaceEmbeddings(
+    model_name="sentence-transformers/all-MiniLM-L6-v2",
+    model_kwargs=model_kwargs
+)
+text_splitter = RecursiveCharacterTextSplitter(chunk_size=250, chunk_overlap=40)
 
-    with gr.Row():
-        with gr.Column(scale=2):
-            review_input = gr.Textbox(
-                lines=10,
-                label="Enter Product Review Here",
-                placeholder="e.g., The camera is amazing, but the battery life is terrible..."
-            )
-            category_input = gr.Dropdown(
-                choices=list(ASPECT_DICTIONARIES.keys()),
-                label="Select Product Category",
-                value="Phone"
-            )
-            analyze_button = gr.Button("Analyze Review", variant="primary")
+# 4. Create Default Vector Store
+print("Creating default FAISS vector store...")
+default_vector_store = pipeline.create_vector_store_from_content(
+    "\n---\n".join(default_reviews_list), text_splitter, embeddings
+)
+if default_vector_store is None:
+    raise ValueError("Failed to create default vector store!")
+print("Default vector store created successfully.")
+
+# Global variable to hold the CURRENT vector store for the chatbot
+# NOTE: Using a global like this works for simple Gradio apps but isn't
+# robust for multiple users. Gradio state or session management is better
+# for multi-user scenarios, but this keeps it simpler for now.
+current_chatbot_vector_store = default_vector_store
+current_context_source = "Default Laptop Reviews"
+
+# 5. Load the LLM
+print("Loading LLM (Mistral-7B GGUF)...")
+llm = LlamaCpp(
+    model_path=model_name, n_gpu_layers=0, n_batch=512, n_ctx=4096,
+    f16_kv=True, temperature=0.0, max_tokens=512, verbose=False,
+    stop=["[/INST]", "User:", "Assistant:"]
+)
+
+# 6. Define All Prompts
+print("Defining all prompts...")
+# -- Phase 1 --
+summary_template = """[INST] You are a helpful assistant... Reviews:\n{reviews} [/INST]\nConcise Summary:"""
+summary_prompt = PromptTemplate(template=summary_template, input_variables=["reviews"])
+aspect_template = """[INST] You are a helpful product analyst... Reviews:\n{reviews} [/INST]\nKey Pros and Cons:"""
+aspect_prompt = PromptTemplate(template=aspect_template, input_variables=["reviews"])
+sentiment_template = """[INST] You are a helpful sentiment analyst... Reviews:\n{reviews} [/INST]\nOverall Sentiment (Score 1-10):"""
+sentiment_prompt = PromptTemplate(template=sentiment_template, input_variables=["reviews"])
+# -- Phase 2 --
+condense_question_template = """[INST] Given the following conversation... Follow Up Input: {question} [/INST]\nStandalone question:"""
+CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(condense_question_template)
+qa_system_prompt = """[INST]
+You are a factual assistant that answers only using the provided product reviews.
+If the reviews include partial or uncertain information, summarize what they say.
+If there is no information at all about the user’s question, respond with:
+"I'm sorry, there isn't enough information in the reviews to answer that."
+
+Do not use or infer information about price, comparisons to other brands, or availability unless they are directly mentioned in the reviews.
+Always include a short "Evidence:" sentence if you found relevant mentions.
+
+Context:
+{context}
+
+User question:
+{question}
+[/INST]
+"""
+qa_prompt = ChatPromptTemplate.from_messages([SystemMessagePromptTemplate.from_template(qa_system_prompt), HumanMessagePromptTemplate.from_template("Context:\n{context}\n\nQuestion:\n{question}\n\nHelpful Answer:")])
+intent_template = """
+[INST]
+**CRITICAL INSTRUCTION:** Classify the user's query into ONLY ONE of two categories: "Product" or "Off-Topic".
+Your response MUST be EXACTLY "Product" or EXACTLY "Off-Topic".
+
+**EXAMPLES:**
+Query: How is the battery life?
+Classification: Product
+Query: What are the complaints about the screen?
+Classification: Product
+Query: Does it come in blue?
+Classification: Product
+Query: What is the capital of France?
+Classification: Off-Topic
+Query: Hello there
+Classification: Off-Topic
+Query: Who are you?
+Classification: Off-Topic
 
-        with gr.Column(scale=1):
-            gr.Markdown("### Overall Sentiment")
-            sentiment_output = gr.Label()
+**NOW CLASSIFY THIS QUERY:**
+Query: {query}
+[/INST]
+Classification:"""
+intent_prompt = PromptTemplate(template=intent_template, input_variables=["query"])
 
-            gr.Markdown("### Generated Summary")
-            summary_output = gr.Textbox(lines=5, label="Summary", interactive=False)
+# 7. Global Memory Object
+chat_memory = ConversationBufferMemory(memory_key='chat_history', return_messages=True, output_key='answer')
 
-            gr.Markdown("### Detected Aspect Sentiments")
-            aspect_output = gr.DataFrame(headers=["Aspect", "Sentiment", "Score"], label="Aspects", interactive=False)
+print("--- App Setup Complete ---")
 
-    # Connect the button to the function
-    analyze_button.click(
-        fn=analyze_review,
-        inputs=[review_input, category_input],
-        outputs=[sentiment_output, summary_output, aspect_output]
+
+# --- Gradio Helper Functions (Wrappers around pipeline logic) ---
+
+def analyze_reviews_gradio_wrapper(review_text, review_file):
+    """Gradio wrapper for Phase 1 analysis."""
+    content = ""
+    if review_file is not None:
+        try:
+            if hasattr(review_file, 'name'): file_path = review_file.name; f=open(file_path, 'rb'); byte_content = f.read(); f.close()
+            else: byte_content = review_file
+            try: content = byte_content.decode('utf-8')
+            except UnicodeDecodeError: content = byte_content.decode('latin-1')
+        except Exception as e: return f"Error reading file: {e}", "", ""
+        if not content: return "Error: File empty", "", ""
+    elif review_text:
+        content = review_text
+    else:
+        return "Please paste reviews or upload a file.", "", ""
+
+    # Call the core logic function
+    return pipeline.analyze_reviews_logic(
+        content, llm, summary_prompt, aspect_prompt, sentiment_prompt
     )
 
-    gr.Examples(
-        examples=[
-            [
-                "The camera on this phone is incredible, the pictures are professional quality. However, the battery life is a total disaster, it barely lasts half a day with light use. The screen is bright and responsive, which I love.",
-                "Phone"
-            ],
-            [
-                "I am absolutely in love with this coffee maker! It's incredibly easy to use, brews a perfect cup every single time, and the design looks fantastic on my countertop. It's also surprisingly quiet.",
-                "Coffee Maker"
-            ],
-            [
-                "An amazing story with characters that felt so real. The plot had me hooked from the first page, though I felt the ending was a bit rushed.",
-                "Book"
-            ]
-        ],
-        inputs=[review_input, category_input]
+def update_chatbot_context_gradio_wrapper(chatbot_file_upload):
+    """Gradio wrapper to update chatbot context."""
+    global current_chatbot_vector_store, current_context_source # Modify globals
+
+    if chatbot_file_upload is None:
+        return f"No file uploaded. Chatbot context remains: **{current_context_source}**."
+
+    print("Processing chatbot context file via Gradio...")
+    content = ""
+    file_name = "Uploaded File"
+    try:
+        if hasattr(chatbot_file_upload, 'name'):
+            file_path = chatbot_file_upload.name
+            file_name = os.path.basename(file_path)
+            with open(file_path, 'rb') as f: byte_content = f.read()
+        else: byte_content = chatbot_file_upload
+        try: content = byte_content.decode('utf-8')
+        except UnicodeDecodeError: content = byte_content.decode('latin-1')
+    except Exception as e: return f"Error reading file: {e}. Context not updated."
+    if not content: return "File empty. Context not updated."
+
+    # Call the core logic function to create the store
+    new_vector_store = pipeline.create_vector_store_from_content(content, text_splitter, embeddings)
+
+    if new_vector_store:
+        current_chatbot_vector_store = new_vector_store # Update global store
+        current_context_source = f"File: {file_name}"
+        status_message = f"Chatbot context updated using **{file_name}**."
+        print(status_message)
+        return status_message
+    else:
+        # If store creation failed, keep the old one
+        status_message = f"Error creating context from {file_name}. Chatbot context remains: **{current_context_source}**."
+        print(status_message)
+        return status_message
+
+
+def chat_responder_gradio_wrapper(message, chat_history):
+    """Gradio wrapper for the chatbot response logic."""
+    # Pass necessary global objects to the core logic function
+    response = pipeline.get_chatbot_response(
+        message=message,
+        chat_memory=chat_memory,
+        vector_store=current_chatbot_vector_store, # Use the current global store
+        llm=llm,
+        intent_prompt=intent_prompt,
+        condense_prompt=CONDENSE_QUESTION_PROMPT,
+        qa_prompt=qa_prompt
     )
+    return response
+
+def clear_chat_memory_gradio_wrapper():
+    """Gradio wrapper to clear memory."""
+    print("Clearing chat memory via Gradio button...")
+    chat_memory.clear()
+    print("Chat memory cleared.")
+    return [] # Return empty list to clear ChatInterface display
+
+def reset_context_to_default_gradio_wrapper():
+     """Gradio wrapper to reset context to default."""
+     global current_chatbot_vector_store, current_context_source
+     print("Resetting context via Gradio button...")
+     current_chatbot_vector_store = default_vector_store
+     current_context_source = "Default Laptop Reviews"
+     status_msg = f"Chatbot context reset to **{current_context_source}**."
+     print(status_msg)
+     return status_msg
+
 
+# --- Gradio UI Definition ---
+with gr.Blocks(theme=gr.themes.Soft()) as demo:
+    gr.Markdown("# 🤖 Product Review Intelligence Center")
+    gr.Markdown("Analyze product reviews using Mistral-7B (Tab 1) or chat about reviews with customizable context (Tab 2).")
+
+    with gr.Tabs():
+        # --- TAB 1: BATCH ANALYZER ---
+        with gr.TabItem("Batch Analyzer"):
+            gr.Markdown("Paste reviews OR upload a file (.txt, .csv) to analyze them.")
+            gr.Markdown("**Note:** This analysis does *not* affect the chatbot's context in Tab 2.")
+            with gr.Row():
+                with gr.Column(scale=2):
+                     review_input_text_tab1 = gr.Textbox(lines=15, placeholder="Paste reviews here...", label="Reviews Text Input")
+                     review_input_file_tab1 = gr.File(label="Upload Reviews File (.txt, .csv)", file_types=[".txt", ".csv"])
+                with gr.Column(scale=1):
+                    summary_output_tab1 = gr.Textbox(label="Overall Summary", lines=5, interactive=False)
+                    aspect_output_tab1 = gr.Textbox(label="Key Aspects (Pros/Cons)", lines=5, interactive=False)
+                    sentiment_output_tab1 = gr.Textbox(label="Sentiment Analysis", lines=5, interactive=False)
+            analyze_button_tab1 = gr.Button("Analyze Reviews")
+            gr.Examples(examples=[[example_batch, None]], inputs=[review_input_text_tab1, review_input_file_tab1], outputs=[summary_output_tab1, aspect_output_tab1, sentiment_output_tab1], fn=analyze_reviews_gradio_wrapper, cache_examples=False) # Use wrapper
+            analyze_button_tab1.click(fn=analyze_reviews_gradio_wrapper, inputs=[review_input_text_tab1, review_input_file_tab1], outputs=[summary_output_tab1, aspect_output_tab1, sentiment_output_tab1]) # Use wrapper
+
+        # --- TAB 2: CHAT ABOUT REVIEWS ---
+        with gr.TabItem("Ask a Question (Chatbot)"):
+            gr.Markdown("Ask specific questions about product reviews. Upload a file below to change the chatbot's knowledge base.")
+            chatbot_status_display = gr.Markdown(f"Chatbot is currently using: **{current_context_source}**")
+            with gr.Row():
+                chatbot_context_file = gr.File(label="Upload Chatbot Context File (.txt, .csv)", file_types=[".txt", ".csv"], scale=3)
+                update_context_button = gr.Button("Update Chatbot Context", scale=1)
+            chatbot_interface = gr.ChatInterface(
+                fn=chat_responder_gradio_wrapper, # Use wrapper
+                examples=["How is the battery life?", "What about the screen?", "What are the complaints about connectivity?", "What is the capital of France?"],
+                title="Review Chatbot"
+            )
+            with gr.Row():
+                reset_memory_button = gr.Button("🔄 Reset Chat Memory")
+                reset_context_button = gr.Button("🔄 Reset Chatbot Context to Default")
+            # Link actions to wrapper functions
+            update_context_button.click(fn=update_chatbot_context_gradio_wrapper, inputs=[chatbot_context_file], outputs=[chatbot_status_display])
+            reset_memory_button.click(fn=clear_chat_memory_gradio_wrapper, inputs=None, outputs=[chatbot_interface])
+            reset_context_button.click(fn=reset_context_to_default_gradio_wrapper, inputs=None, outputs=[chatbot_status_display])
 
-# --- Launch the App ---
+# --- Launch Command ---
 if __name__ == "__main__":
-    print("Launching Gradio App...")
-    demo.launch()
+    chat_memory.clear() # Clear memory each time app starts
+    demo.launch(debug=True)

scripts/data_prepare.py

@@ -1,263 +0,0 @@
-import pytorch_lightning as pl
-from torch.utils.data import DataLoader, Dataset
-from transformers import AutoTokenizer
-import pandas as pd
-from sklearn.model_selection import train_test_split
-import torch
-import os
-
-def explore_and_preprocess_reviews(
-    train_path='data/train.csv', 
-    test_path='data/test.csv',
-    output_dir='data'
-):
-    """
-    Loads the Amazon Sentiment Analysis dataset (https://www.kaggle.com/datasets/kritanjalijain/amazon-reviews)
-    (you need to extract the train/test splits from the zip file in the data folder),
-    performs basic EDA, and preprocesses it for model training.
-
-    Args:
-        train_path (str): Path to the training CSV file.
-        test_path (str): Path to the testing CSV file.
-        output_dir (str): Directory to save the processed file.
-    """
-    # --- 1. Load Data ---
-    # This dataset typically comes without headers. We'll assign them.
-    # Column 1: Sentiment (1 = Negative, 2 = Positive)
-    # Column 2: Title
-    # Column 3: Review Text
-    print(f"Loading data from '{train_path}' and '{test_path}'...")
-    try:
-        col_names = ['sentiment_orig', 'title', 'review']
-        train_df = pd.read_csv(train_path, header=None, names=col_names)
-        test_df = pd.read_csv(test_path, header=None, names=col_names)
-        
-        # Combine for unified EDA and preprocessing
-        df = pd.concat([train_df, test_df], ignore_index=True)
-
-    except FileNotFoundError:
-        print(f"\nERROR: Make sure '{train_path}' and '{test_path}' are in the specified directory.")
-        print("This script is designed for the 'Amazon Reviews for Sentiment Analysis' dataset from Kaggle.")
-        return
-
-    df.dropna(inplace=True)
-
-    # --- 2. Preprocessing ---
-    print("\n--- Preprocessing Data for Sentiment Analysis ---")
-
-    # a) Create new sentiment labels (0 = Negative, 1 = Positive)
-    # This dataset is binary, not three-class like the previous one.
-    df['sentiment'] = df['sentiment_orig'].apply(lambda x: 0 if x == 1 else 1)
-
-    # b) Combine title and review body
-    df['full_text'] = df['title'].astype(str) + ". " + df['review'].astype(str)
-
-    # c) Select and rename columns
-    processed_df = df[['full_text', 'sentiment']].copy()
-
-    # --- 4. Save Processed Data ---
-    os.makedirs(output_dir, exist_ok=True)
-    output_path = os.path.join(output_dir, 'reviews_processed.csv')
-    processed_df.to_csv(output_path, index=False)
-    print(f"\nSaved {len(processed_df)} processed reviews to '{output_path}'")
-
-class ReviewDataset(Dataset):
-    """
-    Custom PyTorch Dataset for Amazon Reviews.
-
-    This class takes a pandas DataFrame of review data, a tokenizer, and a max
-    token length, and prepares it for use in a PyTorch model. It handles the
-    tokenization of the text and the formatting of the labels for each item.
-
-    Attributes:
-        tokenizer: The Hugging Face tokenizer to use for processing text.
-        data (pd.DataFrame): The DataFrame containing the review data.
-        max_token_len (int): The maximum sequence length for the tokenizer.
-    """
-    def __init__(self, data: pd.DataFrame, tokenizer, max_token_len: int):
-        """
-        Initializes the ReviewDataset.
-
-        Args:
-            data (pd.DataFrame): The input DataFrame containing 'full_text' and
-                                 'sentiment' columns.
-            tokenizer: The pre-trained tokenizer instance.
-            max_token_len (int): The maximum length for tokenized sequences.
-        """
-        self.tokenizer = tokenizer
-        self.data = data
-        self.max_token_len = max_token_len
-
-    def __len__(self):
-        """
-        Returns the total number of samples in the dataset.
-        """
-        return len(self.data)
-
-    def __getitem__(self, index: int):
-        """
-        Retrieves one sample from the dataset at the specified index.
-
-        This method handles the tokenization of a single review text, including
-        padding and truncation, and formats the output into a dictionary of
-        tensors ready for the model.
-
-        Args:
-            index (int): The index of the data sample to retrieve.
-
-        Returns:
-            dict: A dictionary containing the tokenized inputs and the label,
-                  with the following keys:
-                  - 'input_ids': The token IDs of the review text.
-                  - 'attention_mask': The attention mask for the review text.
-                  - 'labels': The sentiment label as a tensor.
-        """
-        data_row = self.data.iloc[index]
-        text = str(data_row.full_text)
-        labels = data_row.sentiment
-
-        encoding = self.tokenizer.encode_plus(
-            text,
-            add_special_tokens=True,
-            max_length=self.max_token_len,
-            return_token_type_ids=False,
-            padding="max_length",
-            truncation=True,
-            return_attention_mask=True,
-            return_tensors='pt',
-        )
-
-        return dict(
-            input_ids=encoding["input_ids"].flatten(),
-            attention_mask=encoding["attention_mask"].flatten(),
-            labels=torch.tensor(labels, dtype=torch.long)
-        )
-
-class ReviewDataModule(pl.LightningDataModule):
-    """
-    PyTorch Lightning DataModule to handle the Amazon Reviews dataset.
-
-    This class encapsulates all the steps needed to process the data:
-    loading, splitting, and creating PyTorch DataLoaders for training,
-    validation, and testing. It allows for using a smaller random sample of the
-    full dataset for faster experimentation.
-
-    Attributes:
-        data_path (str): Path to the processed CSV file.
-        batch_size (int): The size of each data batch.
-        max_token_len (int): The maximum sequence length for the tokenizer.
-        tokenizer: The Hugging Face tokenizer instance.
-        num_workers (int): The number of CPU cores to use for data loading.
-        sample_size (int, optional): The number of samples to use. If None,
-                                     the full dataset is used.
-    """
-    def __init__(self, data_path: str, batch_size: int = 16, max_token_len: int = 256, model_name='distilbert-base-uncased', num_workers: int = 0, sample_size: int = None):
-        """
-        Initializes the ReviewDataModule.
-
-        Args:
-            data_path (str): The path to the processed CSV data file.
-            batch_size (int): The number of samples per batch.
-            max_token_len (int): Maximum length of tokenized sequences.
-            model_name (str): The name of the pre-trained model to use for the tokenizer.
-            num_workers (int): Number of subprocesses to use for data loading.
-            sample_size (int, optional): If specified, a random sample of this
-                                         size will be used from the dataset.
-                                         Defaults to None, which uses the full dataset.
-        """
-        super().__init__()
-        self.data_path = data_path
-        self.batch_size = batch_size
-        self.max_token_len = max_token_len
-        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
-        self.num_workers = num_workers
-        self.sample_size = sample_size
-        self.train_df = None
-        self.val_df = None
-        self.test_df = None
-
-    def setup(self, stage=None):
-        """
-        Loads and splits the data for training, validation, and testing.
-
-        This method is called by PyTorch Lightning. It reads the CSV, handles
-        missing values, optionally takes a random sample, and performs a
-        stratified train-validation-test split. The indices of the resulting
-        DataFrames are reset to prevent potential KeyErrors during data loading.
-        """
-        df = pd.read_csv(self.data_path)
-        df.dropna(inplace=True)
-
-        # If a sample size is provided, sample the dataframe
-        if self.sample_size:
-            print(f"Using a sample of {self.sample_size} reviews.")
-            df = df.sample(n=self.sample_size, random_state=42)
-
-        # Stratified split to maintain label distribution
-        train_val_df, self.test_df = train_test_split(df, test_size=0.1, random_state=42, stratify=df.sentiment)
-        self.train_df, self.val_df = train_test_split(train_val_df, test_size=0.1, random_state=42, stratify=train_val_df.sentiment)
-
-        # Reset indices to prevent KeyErrors
-        self.train_df = self.train_df.reset_index(drop=True)
-        self.val_df = self.val_df.reset_index(drop=True)
-        self.test_df = self.test_df.reset_index(drop=True)
-
-        print(f"Size of training set: {len(self.train_df)}")
-        print(f"Size of validation set: {len(self.val_df)}")
-        print(f"Size of test set: {len(self.test_df)}")
-
-    def train_dataloader(self):
-        """Returns the DataLoader for the training set."""
-        return DataLoader(
-            ReviewDataset(self.train_df, self.tokenizer, self.max_token_len),
-            batch_size=self.batch_size,
-            shuffle=True,
-            num_workers=self.num_workers
-        )
-
-    def val_dataloader(self):
-        """Returns the DataLoader for the validation set."""
-        return DataLoader(
-            ReviewDataset(self.val_df, self.tokenizer, self.max_token__len),
-            batch_size=self.batch_size,
-            num_workers=self.num_workers
-        )
-
-    def test_dataloader(self):
-        """Returns the DataLoader for the test set."""
-        return DataLoader(
-            ReviewDataset(self.test_df, self.tokenizer, self.max_token_len),
-            batch_size=self.batch_size,
-            num_workers=self.num_workers
-        )
-        
-if __name__ == "__main__":
-    
-    #--- Step 1: Preprocess the Reviews Dataset ---
-    print("\n--- Preprocessing started ---")
-    explore_and_preprocess_reviews()
-    print("\n--- Preprocessing finished ---")
-    # --- Configuration ---
-    data_path = "data/reviews_processed.csv"
-    BATCH_SIZE = 64
-    MAX_TOKEN_LEN = 256
-    
-    print("Initializing ReviewDataModule...")
-    review_datamodule = ReviewDataModule(
-        data_path=data_path,
-        batch_size=BATCH_SIZE,
-        max_token_len=MAX_TOKEN_LEN,
-        model_name='distilbert-base-uncased',
-        sample_size=100000 # Pass the sample size to the datamodule
-    )
-    review_datamodule.setup()
-
-    # Fetch one batch from the training dataloader to inspect its contents
-    print("\n--- Fetching one batch from the training dataloader ---")
-    train_batch = next(iter(review_datamodule.train_dataloader()))
-    
-    print("\n--- Example Batch ---")
-    print(f"Input IDs shape: {train_batch['input_ids'].shape}")
-    print(f"Attention Mask shape: {train_batch['attention_mask'].shape}")
-    print(f"Labels: {train_batch['labels']}")
-    print(f"Labels shape: {train_batch['labels'].shape}")

scripts/main.py

@@ -1,109 +1,212 @@
-import os
-import torch
-import pandas as pd
+# main.py
 
-try:
-    from data_prepare import ReviewDataset, ReviewDataModule
-    from models import SentimentClassifier, ReviewSummarizer, AspectAnalyzer, FineTunedSentimentClassifier, AspectExtractor
-except ImportError:
-    print("CRITICAL ERROR: Make sure 'review_summarizer.py', 'aspect_extractor.py', and 'sentiment_classifier_model.py' are in the same directory.")
+import torch
+from langchain_community.embeddings import HuggingFaceEmbeddings
+from langchain_community.llms import LlamaCpp
+from langchain.memory import ConversationBufferMemory
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.prompts import PromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate, ChatPromptTemplate
+import os
+import argparse # For command-line arguments
+
+# Import the logic functions from src
+import src.pipeline as pipeline
+
+# --- Global Objects & Setup ---
+# (Similar setup as app.py, load models, prompts etc.)
+print("--- Starting Local Execution Setup ---")
+# 1. Check/Define Model Path
+model_name = "mistral-7b-instruct-v0.1.Q4_K_M.gguf"
+if not os.path.exists(model_name):
+    print(f"ERROR: Model file '{model_name}' not found. Please download it first.")
     exit()
 
-# --- Configuration ---
-# --- IMPORTANT: UPDATE THIS PATH ---
-# You need to provide the path to the best checkpoint file that was saved
-# during the training of your sentiment model.
-SENTIMENT_CHECKPOINT_PATH = "checkpoints/sentiment-binary-best-checkpoint.ckpt"
-
-# --- Pre-defined Aspect Dictionaries for Different Product Categories ---
-ASPECT_DICTIONARIES = {
-    "Phone": ['camera', 'battery', 'battery life', 'screen', 'performance', 'price', 'design'],
-    "Coffee Maker": ['ease of use', 'design', 'noise level', 'coffee quality', 'brew time', 'cleaning'],
-    "Book": ['plot', 'characters', 'writing style', 'pacing', 'ending'],
-    "Default": ['quality', 'price', 'service', 'design', 'features'] # A fallback list
-}
-
-def main():
-    """
-    Main function to run the command-line review analysis tool.
-    """
-    # --- 1. Load All Models ---
-    print("--- Initializing all models ---")
-    sentiment_classifier, summarizer, aspect_analyzer, aspect_extractor = None, None, None, None
-    try:
-        summarizer = ReviewSummarizer(force_cpu=True)
-        aspect_analyzer = AspectAnalyzer(force_cpu=True)
-        aspect_extractor = AspectExtractor(force_cpu=True)
-
-        if not os.path.exists(SENTIMENT_CHECKPOINT_PATH):
-            print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
-            print("!!! WARNING: Sentiment checkpoint path not found or not set.         !!!")
-            print(f"!!! Please update the 'SENTIMENT_CHECKPOINT_PATH' variable in main.py")
-            print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
-        else:
-            sentiment_classifier = FineTunedSentimentClassifier(
-                checkpoint_path=SENTIMENT_CHECKPOINT_PATH, force_cpu=True
-            )
-        print("\n--- All models loaded successfully ---\n")
-    except Exception as e:
-        print(f"An error occurred during model initialization: {e}")
-        return
-
-    # --- 2. Interactive Loop ---
-    while True:
-        print("\n==================================================")
-        print("          Product Review Analysis Tool          ")
-        print("==================================================")
-
-        # Get user input
-        review_text = input("Enter the product review text (or type 'quit' to exit):\n> ")
-        if review_text.lower() == 'quit':
-            break
-
-        print("\nAvailable Product Categories:")
-        for i, category in enumerate(ASPECT_DICTIONARIES.keys(), 1):
-            print(f"{i}. {category}")
-
-        category_choice = input(f"Select a product category (1-{len(ASPECT_DICTIONARIES)}):\n> ")
-        try:
-            category_idx = int(category_choice) - 1
-            product_category = list(ASPECT_DICTIONARIES.keys())[category_idx]
-        except (ValueError, IndexError):
-            print("Invalid choice. Using 'Default' category.")
-            product_category = "Default"
-
-        # --- 3. Run Analysis ---
-        print("\n--- Analyzing Review... ---")
-
-        # a. Overall Sentiment
-        sentiment_result = sentiment_classifier.classify(review_text)
-
-        # b. Summary
-        summary_result = summarizer.summarize(review_text)
-
-        # c. Aspect Extraction and Analysis
-        aspect_dictionary = ASPECT_DICTIONARIES.get(product_category)
-        extracted_aspects = aspect_extractor.extract(review_text, aspect_dictionary)
-        aspect_results = None
-        if extracted_aspects:
-            aspect_results = aspect_analyzer.analyze(review_text, extracted_aspects)
-
-        # --- 4. Display Results ---
-        print("\n-------------------- ANALYSIS RESULTS --------------------")
-        print(f"\n[ Overall Sentiment ]")
-        print(f"  - Sentiment: {sentiment_result['label']} (Score: {sentiment_result['score']:.2f})")
-
-        print(f"\n[ Generated Summary ]")
-        print(f"  - {summary_result}")
-
-        print(f"\n[ Detected Aspect Sentiments ]")
-        if aspect_results:
-            for aspect, result in aspect_results.items():
-                print(f"  - {aspect.title()}: {result['sentiment']} (Score: {result['score']:.2f})")
-        else:
-            print("  - No relevant aspects from the dictionary were detected in the review.")
-        print("----------------------------------------------------------")
-
-
+# 2. Prepare Default Sample Data (Optional, for context testing)
+default_reviews_text = """...""" # Paste default laptop reviews
+default_reviews_list = [r.strip() for r in default_reviews_text.strip().split('---') if r.strip()]
+
+# 3. Load Embedding Model, Text Splitter
+print("Loading embedding model and text splitter...")
+model_kwargs = {'device': 'cuda' if torch.cuda.is_available() else 'cpu'}
+embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2", model_kwargs=model_kwargs)
+text_splitter = RecursiveCharacterTextSplitter(chunk_size=250, chunk_overlap=40)
+
+# 4. Create Default Vector Store
+print("Creating default FAISS vector store...")
+default_vector_store = pipeline.create_vector_store_from_content(
+    "\n---\n".join(default_reviews_list), text_splitter, embeddings
+)
+if default_vector_store is None: raise ValueError("Failed to create default vector store!")
+
+# 5. Load the LLM
+print("Loading LLM (Mistral-7B GGUF)...")
+llm = LlamaCpp(
+    model_path=model_name, n_gpu_layers=0, n_batch=512, n_ctx=4096,
+    f16_kv=True, temperature=0.0, max_tokens=512, verbose=False,
+    stop=["[/INST]", "User:", "Assistant:"]
+)
+
+# 6. Define All Prompts
+print("Defining all prompts...")
+# -- Phase 1 --
+summary_template = """[INST] ... Reviews:\n{reviews} [/INST]\nConcise Summary:"""
+summary_prompt = PromptTemplate(template=summary_template, input_variables=["reviews"])
+aspect_template = """[INST] ... Reviews:\n{reviews} [/INST]\nKey Pros and Cons:"""
+aspect_prompt = PromptTemplate(template=aspect_template, input_variables=["reviews"])
+sentiment_template = """[INST] ... Reviews:\n{reviews} [/INST]\nOverall Sentiment (Score 1-10):"""
+sentiment_prompt = PromptTemplate(template=sentiment_template, input_variables=["reviews"])
+# -- Phase 2 --
+condense_question_template = """[INST] Given the following conversation... Follow Up Input: {question} [/INST]\nStandalone question:"""
+CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(condense_question_template)
+qa_system_prompt = """[INST]
+You are a factual assistant that answers only using the provided product reviews.
+If the reviews include partial or uncertain information, summarize what they say.
+If there is no information at all about the user’s question, respond with:
+"I'm sorry, there isn't enough information in the reviews to answer that."
+
+Do not use or infer information about price, comparisons to other brands, or availability unless they are directly mentioned in the reviews.
+Always include a short "Evidence:" sentence if you found relevant mentions.
+
+Context:
+{context}
+
+User question:
+{question}
+[/INST]
+"""
+qa_prompt = ChatPromptTemplate.from_messages([SystemMessagePromptTemplate.from_template(qa_system_prompt), HumanMessagePromptTemplate.from_template("Context:\n{context}\n\nQuestion:\n{question}\n\nHelpful Answer:")])
+intent_template = """
+[INST]
+**CRITICAL INSTRUCTION:** Classify the user's query into ONLY ONE of two categories: "Product" or "Off-Topic".
+Your response MUST be EXACTLY "Product" or EXACTLY "Off-Topic".
+
+**EXAMPLES:**
+Query: How is the battery life?
+Classification: Product
+Query: What are the complaints about the screen?
+Classification: Product
+Query: Does it come in blue?
+Classification: Product
+Query: What is the capital of France?
+Classification: Off-Topic
+Query: Hello there
+Classification: Off-Topic
+Query: Who are you?
+Classification: Off-Topic
+
+**NOW CLASSIFY THIS QUERY:**
+Query: {query}
+[/INST]
+Classification:"""
+intent_prompt = PromptTemplate(template=intent_template, input_variables=["query"])
+
+# 7. Memory Object (Needed for chatbot logic)
+chat_memory = ConversationBufferMemory(memory_key='chat_history', return_messages=True, output_key='answer')
+
+print("--- Local Setup Complete ---")
+
+
+# --- Main Execution Logic ---
 if __name__ == "__main__":
-    main()
+    parser = argparse.ArgumentParser(description="Run ReviewSense Analysis or Chat locally.")
+    parser.add_argument("--mode", choices=['analyze', 'chat'], required=True, help="Mode to run: 'analyze' reviews from a file, or 'chat' interactively.")
+    parser.add_argument("--input", type=str, help="Path to input .txt file for 'analyze' mode, or initial query for 'chat' mode.")
+    parser.add_argument("--context", type=str, help="Optional: Path to a .txt file to use as context for 'chat' mode (defaults to built-in laptop reviews).")
+
+    args = parser.parse_args()
+
+    # --- ANALYZE MODE ---
+    if args.mode == 'analyze':
+        if not args.input or not os.path.exists(args.input):
+            print(f"Error: Input file '{args.input}' not found for analyze mode.")
+            exit()
+        print(f"\n--- Running Analysis on: {args.input} ---")
+        try:
+            with open(args.input, 'r', encoding='utf-8') as f:
+                review_content = f.read()
+        except Exception as e:
+            print(f"Error reading input file: {e}")
+            exit()
+
+        summary, aspects, sentiment = pipeline.analyze_reviews_logic(
+            review_content, llm, summary_prompt, aspect_prompt, sentiment_prompt
+        )
+        print("\n--- Analysis Results ---")
+        print("\n[Summary]")
+        print(summary)
+        print("\n[Aspects]")
+        print(aspects)
+        print("\n[Sentiment]")
+        print(sentiment)
+
+    # --- CHAT MODE ---
+    elif args.mode == 'chat':
+        print("\n--- Starting Interactive Chat ---")
+        # Determine context
+        chat_vector_store = default_vector_store
+        context_name = "Default Laptop Reviews"
+        if args.context:
+            if not os.path.exists(args.context):
+                print(f"Warning: Context file '{args.context}' not found. Using default context.")
+            else:
+                print(f"Loading context from: {args.context}")
+                try:
+                    with open(args.context, 'r', encoding='utf-8') as f:
+                        context_content = f.read()
+                    chat_vector_store = pipeline.create_vector_store_from_content(
+                        context_content, text_splitter, embeddings
+                    )
+                    if chat_vector_store:
+                        context_name = f"File: {os.path.basename(args.context)}"
+                    else:
+                        print("Failed to load context file. Using default context.")
+                        chat_vector_store = default_vector_store
+                except Exception as e:
+                    print(f"Error reading context file '{args.context}': {e}. Using default context.")
+                    chat_vector_store = default_vector_store
+
+        print(f"Using context: {context_name}")
+        chat_memory.clear() # Start fresh chat session
+
+        # Handle initial query if provided
+        if args.input:
+            print("\nUser:", args.input)
+            response = pipeline.get_chatbot_response(
+                message=args.input,
+                chat_memory=chat_memory,
+                vector_store=chat_vector_store,
+                llm=llm,
+                intent_prompt=intent_prompt,
+                condense_prompt=CONDENSE_QUESTION_PROMPT,
+                qa_prompt=qa_prompt
+            )
+            print("\nAssistant:", response)
+
+        # Interactive loop
+        print("\nEnter your questions (type 'quit' or 'exit' to stop):")
+        while True:
+            try:
+                user_message = input("\nUser: ")
+                if user_message.lower() in ['quit', 'exit']:
+                    break
+                if not user_message:
+                    continue
+
+                response = pipeline.get_chatbot_response(
+                    message=user_message,
+                    chat_memory=chat_memory,
+                    vector_store=chat_vector_store,
+                    llm=llm,
+                    intent_prompt=intent_prompt,
+                    condense_prompt=CONDENSE_QUESTION_PROMPT,
+                    qa_prompt=qa_prompt
+                )
+                print("\nAssistant:", response)
+
+            except EOFError: # Handle Ctrl+D
+                break
+            except KeyboardInterrupt: # Handle Ctrl+C
+                break
+        print("\n--- Chat session ended. ---")
+
+    print("\n--- Local Execution Finished ---")

scripts/models.py

@@ -1,256 +0,0 @@
-import pytorch_lightning as pl
-from transformers import AutoModelForSequenceClassification, get_linear_schedule_with_warmup, AutoConfig
-from torch.optim import AdamW
-import torch
-from torchmetrics.functional import accuracy
-from transformers import T5ForConditionalGeneration, T5Tokenizer, AutoTokenizer, pipeline
-
-class SentimentClassifier(pl.LightningModule):
-    """
-    PyTorch Lightning module for the sentiment classification model.
-    """
-    def __init__(self, model_name='distilbert-base-uncased', n_classes=2, learning_rate=2e-5, n_warmup_steps=0, n_training_steps=0, dropout_prob=0.2): # Added dropout
-        super().__init__()
-        self.save_hyperparameters()
-
-        # Configure dropout
-        config = AutoConfig.from_pretrained(model_name)
-        config.hidden_dropout_prob = dropout_prob
-        config.attention_probs_dropout_prob = dropout_prob
-        config.num_labels = n_classes
-
-        self.model = AutoModelForSequenceClassification.from_pretrained(model_name, config=config)
-
-    def forward(self, input_ids, attention_mask, labels=None):
-        return self.model(
-            input_ids=input_ids,
-            attention_mask=attention_mask,
-            labels=labels
-        )
-
-    def training_step(self, batch, batch_idx):
-        output = self.forward(**batch)
-        self.log("train_loss", output.loss, prog_bar=True, logger=True)
-        return output.loss
-
-    def validation_step(self, batch, batch_idx):
-        output = self.forward(**batch)
-        preds = torch.argmax(output.logits, dim=1)
-        val_acc = accuracy(preds, batch['labels'], task='binary')
-        self.log("val_loss", output.loss, prog_bar=True, logger=True)
-        self.log("val_accuracy", val_acc, prog_bar=True, logger=True)
-        return output.loss
-
-    def test_step(self, batch, batch_idx):
-        output = self.forward(**batch)
-        preds = torch.argmax(output.logits, dim=1)
-        test_acc = accuracy(preds, batch['labels'], task='binary')
-        self.log("test_accuracy", test_acc)
-        return test_acc
-
-    def predict_step(self, batch, batch_idx, dataloader_idx=0):
-        output = self.forward(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'])
-        return torch.argmax(output.logits, dim=1)
-
-    def configure_optimizers(self):
-        optimizer = AdamW(self.parameters(), lr=self.hparams.learning_rate, weight_decay=0.01)
-        scheduler = get_linear_schedule_with_warmup(
-            optimizer,
-            num_warmup_steps=self.hparams.n_warmup_steps,
-            num_training_steps=self.hparams.n_training_steps
-        )
-        return dict(optimizer=optimizer, lr_scheduler=dict(scheduler=scheduler, interval='step'))
-
-class ReviewSummarizer:
-    """
-    A class to handle the summarization of product reviews using a pre-trained T5 model.
-    """
-    def __init__(self, model_name='t5-small'):
-        """
-        Initializes the summarizer with a pre-trained T5 model and tokenizer.
-
-        Args:
-            model_name (str): The name of the pre-trained T5 model to use.
-        """
-        print(f"Loading summarization model: {model_name}...")
-        self.model_name = model_name
-        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
-
-        # Load the tokenizer and model from Hugging Face
-        self.tokenizer = T5Tokenizer.from_pretrained(self.model_name)
-        self.model = T5ForConditionalGeneration.from_pretrained(self.model_name).to(self.device)
-        print("Summarization model loaded successfully.")
-
-    def summarize(self, text: str, max_length: int = 50, min_length: int = 10) -> str:
-        """
-        Generates a summary for a given text.
-
-        Args:
-            text (str): The review text to summarize.
-            max_length (int): The maximum length of the generated summary.
-            min_length (int): The minimum length of the generated summary.
-
-        Returns:
-            str: The generated summary.
-        """
-        if not text or not isinstance(text, str):
-            return ""
-
-        # T5 models require a prefix for the task. For summarization, it's "summarize: "
-        preprocess_text = f"summarize: {text.strip()}"
-
-        # Tokenize the input text
-        tokenized_text = self.tokenizer.encode(preprocess_text, return_tensors="pt").to(self.device)
-
-        # Generate the summary
-        summary_ids = self.model.generate(
-            tokenized_text,
-            max_length=max_length,
-            min_length=min_length,
-            length_penalty=2.0,
-            num_beams=4,
-            early_stopping=True
-        )
-
-        # Decode the summary and return it
-        summary = self.tokenizer.decode(summary_ids[0], skip_special_tokens=True)
-        return summary
-
-class AspectAnalyzer:
-    """
-    A class to handle Aspect-Based Sentiment Analysis (ABSA) using a pre-trained model.
-    """
-    # Changed to a different, currently valid lightweight model for ABSA.
-    def __init__(self, model_name='yangheng/deberta-v3-base-absa-v1.1', force_cpu=False):
-        """
-        Initializes the ABSA pipeline with a pre-trained model.
-
-        Args:
-            model_name (str): The name of the pre-trained ABSA model.
-            force_cpu (bool): If True, forces the model to run on the CPU.
-        """
-        print(f"Loading Aspect-Based Sentiment Analysis model: {model_name}...")
-        self.model_name = model_name
-
-        if force_cpu:
-            self.device = -1 # Use -1 for CPU in pipeline
-            print("Forcing ABSA model to run on CPU.")
-        else:
-            self.device = 0 if torch.cuda.is_available() else -1
-
-        print(f"Using device: {self.device} (0 for GPU, -1 for CPU)")
-
-        self.absa_pipeline = pipeline(
-            "text-classification",
-            model=self.model_name,
-            tokenizer=self.model_name,
-            device=self.device
-        )
-        print("ABSA model loaded successfully.")
-
-    def analyze(self, text: str, aspects: list) -> dict:
-        """
-        Analyzes the sentiment towards a list of aspects within a given text.
-        """
-        if not text or not isinstance(text, str) or not aspects:
-            return {}
-
-        # The model expects the review and aspect separated by a special token.
-        # Note: Different ABSA models might expect different input formats.
-        # This format is common but may need adjustment for other models.
-        inputs = [f"{text} [SEP] {aspect}" for aspect in aspects]
-        results = self.absa_pipeline(inputs)
-
-        # Process results into a user-friendly dictionary
-        aspect_sentiments = {}
-        for aspect, result in zip(aspects, results):
-            aspect_sentiments[aspect] = {'sentiment': result['label'], 'score': result['score']}
-
-        return aspect_sentiments
-
-class FineTunedSentimentClassifier:
-    """
-    This class handles loading the fine-tuned checkpoint and making predictions.
-    """
-    def __init__(self, checkpoint_path, model_name='distilbert-base-uncased', force_cpu=False):
-        self.device = 'cpu' if force_cpu else ('cuda' if torch.cuda.is_available() else 'cpu')
-        print(f"Loading fine-tuned sentiment model from checkpoint: {checkpoint_path}...")
-        print(f"Using device: {self.device}")
-
-        self.model = SentimentClassifier.load_from_checkpoint(checkpoint_path, map_location=self.device)
-        self.model.to(self.device)
-        self.model.eval() # Set model to evaluation mode
-
-        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
-        self.labels = ['NEGATIVE', 'POSITIVE']
-        print("Fine-tuned sentiment model loaded successfully.")
-
-    def classify(self, text: str) -> dict:
-        encoding = self.tokenizer.encode_plus(
-            text, add_special_tokens=True, max_length=128,
-            return_token_type_ids=False, padding="max_length",
-            truncation=True, return_attention_mask=True, return_tensors='pt',
-        )
-        input_ids = encoding["input_ids"].to(self.device)
-        attention_mask = encoding["attention_mask"].to(self.device)
-        with torch.no_grad():
-            outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)
-        logits = outputs.logits
-        probabilities = torch.softmax(logits, dim=1)
-        prediction_idx = torch.argmax(probabilities, dim=1).item()
-        return {'label': self.labels[prediction_idx], 'score': probabilities[0][prediction_idx].item()}
-
-class AspectExtractor:
-    """
-    This class uses a Part-of-Speech (POS) tagging model to first extract all
-    potential aspect terms (nouns) from a review text. It then filters these
-    nouns against a pre-defined dictionary of valid aspects for a given
-    product category to return only the relevant features.
-    """
-    def __init__(self, model_name="vblagoje/bert-english-uncased-finetuned-pos", force_cpu=False):
-        self.model_name = model_name
-        self.device = 'cpu' if force_cpu else ('cuda' if torch.cuda.is_available() else 'cpu')
-        print(f"Loading Part-of-Speech (POS) tagging model: {self.model_name}...")
-        print(f"Using device: {self.device}")
-
-        self.pipeline = pipeline(
-            "token-classification",
-            model=self.model_name,
-            device=-1 if self.device == 'cpu' else 0,
-            aggregation_strategy="simple"
-        )
-        print("POS tagging model loaded successfully.")
-
-    def extract(self, text: str, aspect_dictionary: list) -> list:
-        """
-        Extracts aspects from the given text that are present in the provided
-        aspect dictionary.
-
-        Args:
-            text (str): The review text to analyze.
-            aspect_dictionary (list): A list of valid, known aspects for the
-                                      product category.
-
-        Returns:
-            list: A list of aspects that were both found in the text and are
-                  present in the aspect dictionary.
-        """
-        if not text or not aspect_dictionary:
-            return []
-
-        # 1. Extract all nouns from the text using the POS model
-        model_outputs = self.pipeline(text)
-        noun_tags = {'NOUN', 'PROPN'}
-        extracted_nouns = {
-            output['word'].lower() for output in model_outputs
-            if output['entity_group'] in noun_tags
-        }
-
-        # 2. Filter the extracted nouns against the provided dictionary
-        # We find the intersection between the two sets.
-        valid_aspects = {aspect.lower() for aspect in aspect_dictionary}
-
-        final_aspects = list(extracted_nouns.intersection(valid_aspects))
-
-        return final_aspects
-    

scripts/pipeline.py

@@ -0,0 +1,127 @@
+# src/pipeline.py
+
+import os
+import io
+from langchain_community.vectorstores import FAISS
+from langchain.chains import ConversationalRetrievalChain
+from langchain.prompts import PromptTemplate # Ensure necessary Langchain imports are here if needed directly
+
+# --- Core Logic Functions ---
+
+def analyze_reviews_logic(review_text: str, llm, summary_prompt, aspect_prompt, sentiment_prompt):
+    """
+    Performs Phase 1 analysis (summary, aspects, sentiment) on the provided text.
+    """
+    print(f"Running batch analysis logic on {len(review_text)} chars...")
+    try:
+        summary_result = llm.invoke(summary_prompt.format(reviews=review_text)).strip()
+        print("   -> Summary generated.")
+        aspect_result = llm.invoke(aspect_prompt.format(reviews=review_text)).strip()
+        print("   -> Aspects extracted.")
+        sentiment_result = llm.invoke(sentiment_prompt.format(reviews=review_text)).strip()
+        print("   -> Sentiment analyzed.")
+        return summary_result, aspect_result, sentiment_result
+    except Exception as e:
+        print(f"ERROR during batch analysis logic: {e}")
+        error_msg = f"Error during analysis: {e}"
+        return error_msg, error_msg, error_msg
+
+def create_vector_store_from_content(content: str, text_splitter, embeddings):
+    """
+    Splits content and creates a new FAISS vector store.
+    Returns the vector store or None if an error occurs.
+    """
+    print("Creating new vector store from content...")
+    if not content:
+        print("Error: No content provided to create vector store.")
+        return None
+
+    # Split content
+    if "\n---\n" in content:
+        reviews_list = [r.strip() for r in content.strip().split('\n---\n') if r.strip()]
+    else:
+        reviews_list = [r.strip() for r in content.strip().split('\n\n') if r.strip()]
+        if len(reviews_list) <= 1: reviews_list = [content.strip()] # Single block case
+
+    if not reviews_list:
+        print("Error: Could not extract reviews from content.")
+        return None
+
+    review_chunks = text_splitter.create_documents(reviews_list)
+    if not review_chunks:
+       print("Error: Failed to create document chunks.")
+       return None
+
+    try:
+        vector_store = FAISS.from_documents(review_chunks, embeddings)
+        print("Vector store created successfully.")
+        return vector_store
+    except Exception as e:
+        print(f"Error creating FAISS index: {e}")
+        return None
+
+def parse_intent(llm_output: str) -> str:
+    """
+    Parses the LLM output to find 'Product' or 'Off-Topic'.
+    Defaults to 'Off-Topic' if neither is found or output is unexpected.
+    Uses case-insensitive 'in' check for robustness.
+    """
+    output_lower = llm_output.strip().lower()
+    if "product" in output_lower:
+        return "Product"
+    elif "off-topic" in output_lower:
+        return "Off-Topic"
+    else:
+        print(f"   -> Unexpected classification: '{llm_output.strip()}'. Defaulting to Off-Topic.")
+        return "Off-Topic"
+
+def get_chatbot_response(message: str, chat_memory, vector_store, llm, intent_prompt, condense_prompt, qa_prompt):
+    """
+    Handles Phase 2: Classifies intent and runs RAG if appropriate.
+    Returns the chatbot's response string.
+    """
+    print(f"\nProcessing chatbot query: {message}")
+
+    # --- 1. Classify Intent ---
+    formatted_intent_prompt = intent_prompt.format(query=message)
+    intent_result_raw = llm.invoke(formatted_intent_prompt)
+    print(f"   DEBUG: Raw Intent Output: '{intent_result_raw.strip()}'")
+    intent = parse_intent(intent_result_raw)
+    print(f"   -> Detected Intent: {intent}")
+
+    # --- 2. Route ---
+    if intent == "Product":
+        print("   -> Routing to RAG chain...")
+        if vector_store is None:
+             print("   ERROR: No vector store available for RAG.")
+             return "Sorry, I don't have any review context loaded to answer product questions."
+
+        retriever = vector_store.as_retriever(search_kwargs={"k": 4})
+
+        # Create chain dynamically for each call
+        conv_qa_chain = ConversationalRetrievalChain.from_llm(
+            llm=llm,
+            retriever=retriever,
+            memory=chat_memory,
+            condense_question_prompt=condense_prompt,
+            combine_docs_chain_kwargs={"prompt": qa_prompt},
+            return_source_documents=True, # Required for context list in result
+            verbose=False
+        )
+        try:
+            # Pass only question - memory handles history internally
+            result = conv_qa_chain.invoke({"question": message})
+            answer = result['answer'].strip()
+            print(f"   -> RAG Answer: {answer}")
+            return answer
+        except Exception as e:
+             print(f"ERROR during RAG chain execution: {e}")
+             # Optionally log traceback: import traceback; traceback.print_exc()
+             return "Sorry, I encountered an error trying to find an answer in the reviews."
+
+    else: # Off-Topic
+        print("   -> Routing to canned response...")
+        answer = "I'm sorry, I can only answer questions about the product reviews for this item."
+        # Optional: Save off-topic turn to memory if desired
+        # chat_memory.save_context({"question": message}, {"answer": answer})
+        return answer

scripts/train_distilbet.py

@@ -1,101 +0,0 @@
-import pytorch_lightning as pl
-from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
-from pytorch_lightning.loggers import TensorBoardLogger
-import torch
-import seaborn as sns
-import matplotlib.pyplot as plt
-from sklearn.metrics import confusion_matrix
-from data_prepare import ReviewDataModule, ReviewDataset
-from models import SentimentClassifier
-
-def train_sentiment_model(data_path='data/reviews_processed.csv', model_name='distilbert-base-uncased', n_epochs=5, sample_size: int = None):
-    """
-    Main function to train the sentiment analysis model on the Amazon Reviews dataset.
-
-    Args:
-        data_path (str): Path to the processed data file.
-        model_name (str): Name of the transformer model to use.
-        n_epochs (int): Maximum number of epochs for training.
-        sample_size (int, optional): The number of reviews to use for training.
-                                     If None, the full dataset is used.
-    """
-    # --- 1. Hyperparameters ---
-    BATCH_SIZE = 64
-    MAX_TOKEN_LEN = 256
-    LEARNING_RATE = 2e-5
-    N_CLASSES = 2  # Negative, Positive
-
-    # --- 2. Initialize DataModule ---
-    print("Initializing ReviewDataModule...")
-    review_datamodule = ReviewDataModule(
-        data_path=data_path,
-        batch_size=BATCH_SIZE,
-        max_token_len=MAX_TOKEN_LEN,
-        model_name=model_name,
-        sample_size=sample_size # Pass the sample size to the datamodule
-    )
-    review_datamodule.setup()
-
-    n_training_steps = len(review_datamodule.train_dataloader()) * n_epochs
-    n_warmup_steps = int(n_training_steps * 0.1)
-
-    # --- 3. Initialize Model ---
-    print("Initializing SentimentClassifier model...")
-    model = SentimentClassifier(
-        model_name=model_name,
-        n_classes=N_CLASSES,
-        learning_rate=LEARNING_RATE,
-        n_warmup_steps=n_warmup_steps,
-        n_training_steps=n_training_steps
-    )
-
-    # --- 4. Configure Training Callbacks ---
-    checkpoint_callback = ModelCheckpoint(
-        dirpath="checkpoints",
-        filename="sentiment-binary-best-checkpoint",
-        save_top_k=1,
-        verbose=True,
-        monitor="val_loss",
-        mode="min"
-    )
-    logger = TensorBoardLogger("lightning_logs", name="sentiment-classifier-binary")
-    early_stopping_callback = EarlyStopping(monitor='val_loss', patience=2)
-
-    # --- 5. Initialize Trainer ---
-    print("Initializing PyTorch Lightning Trainer...")
-    trainer = pl.Trainer(
-        logger=logger,
-        callbacks=[checkpoint_callback, early_stopping_callback],
-        max_epochs=n_epochs,
-        accelerator='gpu' if torch.cuda.is_available() else 'cpu',
-        devices=1,
-    )
-
-    # --- 6. Start Training ---
-    print(f"Starting training with {model_name} for up to {n_epochs} epochs...")
-    trainer.fit(model, review_datamodule)
-
-    # --- 7. Evaluate on Test Set and Generate Confusion Matrix ---
-    print("\nTraining complete. Evaluating on the test set...")
-    trainer.test(model, datamodule=review_datamodule)
-
-    predictions = trainer.predict(model, datamodule=review_datamodule)
-    if predictions:
-        all_preds = torch.cat(predictions).cpu().numpy()
-        true_labels = review_datamodule.test_df.sentiment.to_numpy()
-        target_names = ['Negative', 'Positive'] # Updated labels
-
-        cm = confusion_matrix(true_labels, all_preds)
-        plt.figure(figsize=(8, 6))
-        sns.heatmap(cm, annot=True, fmt='d', cmap='YlGnBu',
-                    xticklabels=target_names, yticklabels=target_names)
-        plt.title('Confusion Matrix for Sentiment Analysis')
-        plt.xlabel('Predicted Label')
-        plt.ylabel('True Label')
-        plt.show()
-
-
-        
-if __name__ == "__main__":
-    data_path = "data/reviews_processed.csv"
-    train_sentiment_model(data_path=data_path, sample_size=100000)

scripts/train_naive_bayes.py

@@ -1,118 +0,0 @@
-import pandas as pd
-import numpy as np
-from sklearn.model_selection import train_test_split, ParameterGrid, StratifiedKFold
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.naive_bayes import MultinomialNB
-from sklearn.pipeline import Pipeline
-from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
-import seaborn as sns
-import matplotlib.pyplot as plt
-from tqdm.notebook import tqdm
-import os
-
-def train_baseline_sentiment_model(data_path='data/reviews_processed.csv', grid_search=True, nb__alpha=0.1, tfidf__max_df=0.75, tfidf__ngram_range=(1, 2), sample_size: int = 50000):
-    """
-    Trains and evaluates a Multinomial Naive Bayes model for sentiment analysis.
-    Can optionally perform a grid search.
-
-    Args:
-        data_path (str): Path to the processed reviews CSV file.
-        grid_search (bool): If True, performs a grid search.
-        nb__alpha (float): Alpha for MultinomialNB.
-        tfidf__max_df (float): max_df for TfidfVectorizer.
-        tfidf__ngram_range (tuple): ngram_range for TfidfVectorizer.
-        sample_size (int, optional): Number of reviews to use. If None, uses all.
-    """
-    # --- 1. Load Data ---
-    print(f"Loading data from '{data_path}'...")
-    if not os.path.exists(data_path):
-        print(f"\nERROR: '{data_path}' not found. Please run the EDA script first!")
-        return
-        
-    df = pd.read_csv(data_path)
-    df.dropna(inplace=True)
-
-    # --- 2. Sample Data ---
-    if sample_size:
-        print(f"Using a sample of {sample_size} reviews for training the baseline model.")
-        df = df.sample(n=sample_size, random_state=42)
-
-    # --- 3. Train-Test Split ---
-    print("Splitting data into training and testing sets...")
-    X_train, X_test, y_train, y_test = train_test_split(
-        df['full_text'],
-        df['sentiment'],
-        test_size=0.2,
-        random_state=42,
-        stratify=df['sentiment']
-    )
-
-    # --- 4. Create a Pipeline ---
-    pipeline = Pipeline([
-        ('tfidf', TfidfVectorizer(stop_words='english')),
-        ('nb', MultinomialNB()),
-    ])
-
-    best_params = None
-
-    if grid_search:
-        # --- 5a. Perform Grid Search ---
-        print("Performing Grid Search to find the best hyperparameters...")
-        parameters = {
-            'tfidf__ngram_range': [(1, 1), (1, 2)],
-            'tfidf__max_df': [0.5, 0.75, 1.0],
-            'nb__alpha': [0.1, 0.5, 1.0],
-        }
-        param_grid = list(ParameterGrid(parameters))
-        best_score = -1
-
-        for params in tqdm(param_grid, desc="Grid Search Progress"):
-            pipeline.set_params(**params)
-            pipeline.fit(X_train, y_train)
-            score = pipeline.score(X_test, y_test)
-            if score > best_score:
-                best_score = score
-                best_params = params
-        
-        print(f"\nBest score on test set: {best_score:.4f}")
-        print("Best parameters found:")
-        print(best_params)
-
-    else:
-        # --- 5b. Use provided hyperparameters ---
-        print("Skipping grid search and using provided hyperparameters...")
-        best_params = {
-            'nb__alpha': nb__alpha,
-            'tfidf__max_df': tfidf__max_df,
-            'tfidf__ngram_range': tfidf__ngram_range
-        }
-
-    # --- 6. Train the Final Model ---
-    print("\nTraining final model...")
-    best_model = pipeline.set_params(**best_params)
-    best_model.fit(X_train, y_train)
-    print("Model training complete.")
-
-    # --- 7. Evaluate the Best Model ---
-    print("\n--- Model Evaluation ---")
-    y_pred = best_model.predict(X_test)
-    
-    accuracy = accuracy_score(y_test, y_pred)
-    target_names = ['Negative', 'Positive']
-    
-    print(f"Accuracy: {accuracy:.4f}")
-    print("\nClassification Report:")
-    print(classification_report(y_test, y_pred, target_names=target_names))
-    
-    print("Confusion Matrix:")
-    cm = confusion_matrix(y_test, y_pred)
-    plt.figure(figsize=(8, 6))
-    sns.heatmap(cm, annot=True, fmt='d', cmap='Greens', 
-                xticklabels=target_names, yticklabels=target_names)
-    plt.title('Confusion Matrix for Naive Bayes on Amazon Reviews')
-    plt.xlabel('Predicted Label')
-    plt.ylabel('True Label')
-    plt.show()
-    
-if __name__ == "__main__":
-    train_baseline_sentiment_model(sample_size=150000, grid_search=False)