DEPRESSION-DETECTION

.gitattributes

@@ -1,2 +1,4 @@
 Source[[:space:]]Code/assets/models/model_svm.pkl filter=lfs diff=lfs merge=lfs -text
 Source[[:space:]]Code/assets/models/model_svm1.pkl filter=lfs diff=lfs merge=lfs -text
+source_code/assets/models/model_svm.pkl filter=lfs diff=lfs merge=lfs -text
+source_code/assets/models/model_svm1.pkl filter=lfs diff=lfs merge=lfs -text

Dockerfile

@@ -19,7 +19,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 
 # Install dependencies
 # Note: Path is relative to the repository root where Dockerfile resides
-COPY "Source Code/requirements.txt" ./
+COPY source_code/requirements.txt ./
 RUN pip install --upgrade pip
 RUN pip install --no-cache-dir -r requirements.txt
 
@@ -27,7 +27,7 @@ RUN pip install --no-cache-dir -r requirements.txt
 RUN python -m spacy download en_core_web_lg
 
 # Copy project source code
-COPY "Source Code/" ./
+COPY source_code/ ./
 
 # Hugging Face Spaces requires port 7860
 EXPOSE 7860

README.md

@@ -1,312 +1,312 @@
----
-title: Depression Detection Using Tweets
-emoji: 🧠
-colorFrom: blue
-colorTo: indigo
-sdk: docker
-app_file: app.py
-pinned: false
-license: mit
-short_description: Depression Detection in Tweets ML Web App
----
-
-<div align="center">
-
-  <a name="readme-top"></a>
-  # Depression Detection Using Tweets
-
-  [![License: MIT](https://img.shields.io/badge/License-MIT-lightgrey)](LICENSE)
-  ![Status](https://img.shields.io/badge/Status-Completed-success)
-  [![Technology](https://img.shields.io/badge/Technology-Python%20%7C%20Machine%20Learning-blueviolet)](https://github.com/Amey-Thakur/DEPRESSION_DETECTION_USING_TWEETS)
-  [![Developed by Amey Thakur and Mega Satish](https://img.shields.io/badge/Developed%20by-Amey%20Thakur%20%26%20Mega%20Satish-blue.svg)](https://github.com/Amey-Thakur/DEPRESSION_DETECTION_USING_TWEETS)
-
-  A modern **Python** + **Flask** application designed to analyze tweet sentiment and predict depressive characteristics using a finalized **SVM** model and **spaCy** NLP pipeline.
-
-  **[Source Code](Source%20Code/)** &nbsp;·&nbsp; **[Technical Specification](docs/SPECIFICATION.md)** &nbsp;·&nbsp; **[Live Demo](https://huggingface.co/spaces/ameythakur/Depression-Detection-Using-Tweets)**
-
-</div>
-
----
-
-<div align="center">
-
-  [Authors](#authors) &nbsp;·&nbsp; [Overview](#overview) &nbsp;·&nbsp; [Features](#features) &nbsp;·&nbsp; [Structure](#project-structure) &nbsp;·&nbsp; [Results](#results) &nbsp;·&nbsp; [Quick Start](#quick-start) &nbsp;·&nbsp; [Usage Guidelines](#usage-guidelines) &nbsp;·&nbsp; [License](#license) &nbsp;·&nbsp; [About](#about-this-repository) &nbsp;·&nbsp; [Acknowledgments](#acknowledgments)
-
-</div>
-
----
-
-<!-- AUTHORS -->
-<div align="center">
-
-  <a name="authors"></a>
-  ## Authors
-
-| <a href="https://github.com/Amey-Thakur"><img src="https://github.com/Amey-Thakur.png" width="150" height="150" alt="Amey Thakur"></a><br>[**Amey Thakur**](https://github.com/Amey-Thakur)<br><br>[![ORCID](https://img.shields.io/badge/ORCID-0000--0001--5644--1575-green.svg)](https://orcid.org/0000-0001-5644-1575) | <a href="https://github.com/msatmod"><img src="Mega/Mega.png" width="150" height="150" alt="Mega Satish"></a><br>[**Mega Satish**](https://github.com/msatmod)<br><br>[![ORCID](https://img.shields.io/badge/ORCID-0000--0002--1844--9557-green.svg)](https://orcid.org/0000-0002-1844-9557) |
-| :---: | :---: |
-
-</div>
-
-> [!IMPORTANT]
-> ### 🤝🏻 Special Acknowledgement
-> *Special thanks to **[Mega Satish](https://github.com/msatmod)** for her meaningful contributions, guidance, and support that helped shape this work.*
-
----
-
-<!-- OVERVIEW -->
-<a name="overview"></a>
-## Overview
-
-**Depression Detection Using Tweets** is a specialized Machine Learning framework designed to translate complex linguistic patterns into empirical psychological insights. This repository prioritizes **high-dimensional feature extraction** and **probabilistic classification** to provide a robust baseline for sentiment analysis within the context of mental health monitoring.
-
-*   **Linguistic Determinism**: The system utilizes deep NLP preprocessing, including lemmatization and entity normalization, to ensure that the semantic core of a tweet is preserved regardless of slang or stylistic variation.
-*   **Vector-Space Inference**: By leveraging **Support Vector Machines (SVM)** and **TF-IDF vectorization**, the model maps textual input into a multi-dimensional hyperplane, enabling precise binary classification of depressive sentiment.
-*   **Architectural Efficiency**: The backend is architected for low-latency serving via Flask, ensuring that model inference and result rendering occur in sub-second cycles, critical for interactive user feedback.
-
-> [!TIP]
-> **NLP Pipeline Optimization**
->
-> To maximize classification reliability, the engine employs a **multi-stage linguistic filter**. **Stop-word suppression** and **morphological analysis** strip away structural noise, while the **en_core_web_lg** transformer model contextualizes surviving tokens. This ensures the classifier’s weights are strictly coupled with affective indicators, minimizing the false-positive skew common in generalized sentiment analysis models.
-
----
-
-<!-- FEATURES -->
-<a name="features"></a>
-## Features
-
-| Feature | Description |
-|---------|-------------|
-| **Core SVM Model** | **High-Dimensional Classification** engine optimized for binary depressive sentiment prediction. |
-| **NLP Pipeline** | Deep linguistic feature extraction powered by the **spaCy transformer model** (`en_core_web_lg`). |
-| **Prediction Hub** | **Real-Time Inference Interface** built with Flask for sub-second classification feedback. |
-| **Security Suite** | Integrated **Browser-Side Integrity** protocols including anti-right-click and anti-select systems. |
-| **Cinematic Surprise** | **Immersive Branding Overlay** featuring animated Twitter iconography and synchronized audio. |
-
-> [!NOTE]
-> ### Technical Polish: The Linguistic Singularity
-> We have engineered a **Probabilistic Sentiment Manager** that calibrates model weights across thousands of TF-IDF vectors to simulate human-like linguistic intuition. The visual language focuses on a "Neural Slate" aesthetic, ensuring maximum cognitive focus on the diagnostic outputs without procedural distraction.
-
-### Tech Stack
-- **Languages**: Python 3.9+
-- **Logic**: **SVM Classifier** (Scikit-Learn Inference Engine)
-- **Linguistic Data**: **spaCy NLP** (Transformer-based word embeddings)
-- **Web App**: **Flask Framework** (Micro-service architecture for model serving)
-- **UI System**: Premium Modern Aesthetics (Custom CSS / Play Typography)
-- **Deployment**: Standard Python Environment (PIP-managed dependencies)
-
----
-
-<!-- PROJECT STRUCTURE -->
-<a name="project-structure"></a>
-## Project Structure
-
-```python
-DEPRESSION-DETECTION-USING-TWEETS/
-│
-├── docs/                            # Technical Documentation
-│   └── SPECIFICATION.md             # Architecture & Design Specification
-│
-├── Mega/                            # Archival Attribution Assets
-│   ├── Filly.jpg                    # Companion (Filly)
-│   └── Mega.png                     # Author Profile Image (Mega Satish)
-│
-├── screenshots/                     # Project Visualization Gallery
-│   ├── 01_landing_page.png          # System Hub Initial State
-│   ├── 02_footer_details.png        # Brand and Metadata Footer
-│   ├── 03_surprise_cinematic.png    # Interactive Animated Sequence
-│   ├── 04_predict_interface.png     # Sentiment Analysis Entry Point
-│   ├── 05_analysis_output.png       # Model Inference result
-│   └── 06_result_prediction.png     # Final Sentiment Output
-│
-├── Source Code/                     # Primary Application Layer
-│   ├── assets/                      # Serialized Models & Linguistic Data
-│   ├── core/                        # ML Pipeline (Clean, Train, Predict)
-│   ├── static/                      # Styling, Audio, & Security Scripts
-│   ├── templates/                   # HTML Templates (Index, Result, 404)
-│   └── app.py                       # Flask Application (Entry Point)
-│
-├── .gitattributes                   # Git configuration
-├── .gitignore                       # Repository Filters
-├── CITATION.cff                     # Scholarly Citation Metadata
-├── codemeta.json                    # Machine-Readable Project Metadata
-├── LICENSE                          # MIT License Terms
-├── README.md                        # Comprehensive Scholarly Entrance
-└── SECURITY.md                      # Security Policy & Protocol
-```
-
----
-
-<!-- RESULTS -->
-<a name="results"></a>
-## Results
-
-<div align="center">
-  <b>Main Landing: System Hub Initialization</b>
-  <br>
-  <i>Minimalist interface for rapid tweet sentiment analysis.</i>
-  <br><br>
-  <img src="screenshots/01_landing_page.png" alt="Landing Page" width="90%">
-  <br><br><br>
-
-  <b>Metadata Synthesis: Branding and Footer Detail</b>
-  <br>
-  <i>Scholarly attribution and project status integration.</i>
-  <br><br>
-  <img src="screenshots/02_footer_details.png" alt="Footer Details" width="90%">
-  <br><br><br>
-
-  <b>Interactivity: Animated Twitter Sequence</b>
-  <br>
-  <i>Immersive audiovisual overlay triggered by core branding elements.</i>
-  <br><br>
-  <img src="screenshots/03_surprise_cinematic.png" alt="Cinematic Surprise" width="90%">
-  <br><br><br>
-
-  <b>Sentiment Entry: Real-time Analysis Interface</b>
-  <br>
-  <i>Direct manipulation environment for high-latency textual input.</i>
-  <br><br>
-  <img src="screenshots/04_predict_interface.png" alt="Predict Interface" width="90%">
-  <br><br><br>
-
-  <b>Model Inference: Feature Extraction Output</b>
-  <br>
-  <i>Deep linguistic analysis and probabilistic score generation.</i>
-  <br><br>
-  <img src="screenshots/05_analysis_output.png" alt="Analysis Output" width="90%">
-  <br><br><br>
-
-  <b>Statistical Output: Final Sentiment Classification</b>
-  <br>
-  <i>Categorized classification results with immediate visual feedback.</i>
-  <br><br>
-  <img src="screenshots/06_result_prediction.png" alt="Result Prediction" width="90%">
-</div>
-
----
-
-<!-- QUICK START -->
-<a name="quick-start"></a>
-## Quick Start
-
-### 1. Prerequisites
-- **Python 3.11+**: Required for runtime execution. [Download Python](https://www.python.org/downloads/)
-- **Git**: For version control and cloning. [Download Git](https://git-scm.com/downloads)
-
-> [!WARNING]
-> **Data Acquisition & Memory Constraints**
->
-> The linguistic pipeline relies on the **en_core_web_lg** transformer model, which requires an initial download of approximately **800MB**. Ensure a stable network connection during setup. Additionally, loading this model into memory requires at least **2GB of available RAM** to prevent swapping and ensure low-latency inference.
-
-### 2. Installation & Setup
-
-#### Step 1: Clone the Repository
-Open your terminal and clone the repository:
-```bash
-git clone https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS.git
-cd DEPRESSION-DETECTION-USING-TWEETS
-```
-
-#### Step 2: Configure Virtual Environment
-Prepare an isolated environment to manage dependencies:
-
-**Windows (Command Prompt / PowerShell):**
-```bash
-python -m venv venv
-venv\Scripts\activate
-```
-
-**macOS / Linux (Terminal):**
-```bash
-python3 -m venv venv
-source venv/bin/activate
-```
-
-#### Step 3: Install Core Dependencies
-Ensure your environment is active, then install the required libraries:
-```bash
-pip install -r "Source Code/requirements.txt"
-```
-
-#### Step 4: Linguistic Model Acquisition
-Download the large-scale linguistic model required for analysis (approx. 800MB):
-```bash
-python -m spacy download en_core_web_lg
-```
-
-### 3. Execution
-Launch the sentiment analysis dashboard:
-
-```bash
-python "Source Code/app.py"
-```
-
----
-
-<!-- USAGE GUIDELINES -->
-<a name="usage-guidelines"></a>
-## Usage Guidelines
-
-This repository is openly shared to support learning and knowledge exchange across the academic community.
-
-**For Students**  
-Use this project as reference material for understanding **Support Vector Machines (SVM)**, **spaCy NLP pipelines**, and **sentiment analysis within the context of mental health monitoring**. The source code is available for study to facilitate self-paced learning and exploration of **high-dimensional feature extraction and model serving via Flask**.
-
-**For Educators**  
-This project may serve as a practical lab example or supplementary teaching resource for **Data Science**, **Natural Language Processing**, and **Machine Learning** courses. Attribution is appreciated when utilizing content.
-
-**For Researchers**  
-The documentation and architectural approach may provide insights into **academic project structuring**, **psychological linguistic modeling**, and **algorithmic deployment**.
-
----
-
-<!-- LICENSE -->
-<a name="license"></a>
-## License
-
-This repository and all its creative and technical assets are made available under the **MIT License**. See the [LICENSE](LICENSE) file for complete terms.
-
-> [!NOTE]
-> **Summary**: You are free to share and adapt this content for any purpose, even commercially, as long as you provide appropriate attribution to the original authors.
-
-Copyright © 2022 Amey Thakur & Mega Satish
-
----
-
-<!-- ABOUT -->
-<a name="about-this-repository"></a>
-## About This Repository
-
-**Created & Maintained by**: [Amey Thakur](https://github.com/Amey-Thakur) & [Mega Satish](https://github.com/msatmod)
-
-This project features **Depression Detection**, a high-performance sentiment analysis system. It represents a personal exploration into **Python**-based machine learning and interactive web-service architecture.
-
-**Connect:** [GitHub](https://github.com/Amey-Thakur) &nbsp;·&nbsp; [LinkedIn](https://www.linkedin.com/in/amey-thakur) &nbsp;·&nbsp; [ORCID](https://orcid.org/0000-0001-5644-1575)
-
-### Acknowledgments
-
-Grateful acknowledgment to [**Mega Satish**](https://github.com/msatmod) for her exceptional collaboration and scholarly partnership during the development of this machine learning project. Her constant support, technical clarity, and dedication to software quality were instrumental in achieving the system's functional objectives. Learning alongside her was a transformative experience; her thoughtful approach to problem-solving and steady encouragement turned complex requirements into meaningful learning moments. This work reflects the growth and insights gained from our side-by-side academic journey. Thank you, Mega, for everything you shared and taught along the way.
-
-Special thanks to the **mentors and peers** whose encouragement, discussions, and support contributed meaningfully to this learning experience.
-
----
-
-<div align="center">
-
-  [↑ Back to Top](#readme-top)
-
-  [Authors](#authors) &nbsp;·&nbsp; [Overview](#overview) &nbsp;·&nbsp; [Features](#features) &nbsp;·&nbsp; [Structure](#project-structure) &nbsp;·&nbsp; [Results](#results) &nbsp;·&nbsp; [Quick Start](#quick-start) &nbsp;·&nbsp; [Usage Guidelines](#usage-guidelines) &nbsp;·&nbsp; [License](#license) &nbsp;·&nbsp; [About](#about-this-repository) &nbsp;·&nbsp; [Acknowledgments](#acknowledgments)
-
-  <br>
-
-  🧠 **[DEPRESSION-DETECTION](https://huggingface.co/spaces/ameythakur/Depression-Detection-Using-Tweets)**
-
-  ---
-
-  ### 🎓 [Computer Engineering Repository](https://github.com/Amey-Thakur/COMPUTER-ENGINEERING)
-
-  **Computer Engineering (B.E.)**
-
-  *Semester-wise curriculum, laboratories, projects, and academic notes.*
-
+---
+title: Depression Detection Using Tweets
+emoji: 🧠
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+app_file: app.py
+pinned: false
+license: mit
+short_description: Depression Detection in Tweets ML Web App
+---
+
+<div align="center">
+
+  <a name="readme-top"></a>
+  # Depression Detection Using Tweets
+
+  [![License: MIT](https://img.shields.io/badge/License-MIT-lightgrey)](LICENSE)
+  ![Status](https://img.shields.io/badge/Status-Completed-success)
+  [![Technology](https://img.shields.io/badge/Technology-Python%20%7C%20Machine%20Learning-blueviolet)](https://github.com/Amey-Thakur/DEPRESSION_DETECTION_USING_TWEETS)
+  [![Developed by Amey Thakur and Mega Satish](https://img.shields.io/badge/Developed%20by-Amey%20Thakur%20%26%20Mega%20Satish-blue.svg)](https://github.com/Amey-Thakur/DEPRESSION_DETECTION_USING_TWEETS)
+
+  A modern **Python** + **Flask** application designed to analyze tweet sentiment and predict depressive characteristics using a finalized **SVM** model and **spaCy** NLP pipeline.
+
+  **[Source Code](source_code/)** &nbsp;·&nbsp; **[Technical Specification](docs/SPECIFICATION.md)** &nbsp;·&nbsp; **[Live Demo](https://huggingface.co/spaces/ameythakur/Depression-Detection-Using-Tweets)**
+
+</div>
+
+---
+
+<div align="center">
+
+  [Authors](#authors) &nbsp;·&nbsp; [Overview](#overview) &nbsp;·&nbsp; [Features](#features) &nbsp;·&nbsp; [Structure](#project-structure) &nbsp;·&nbsp; [Results](#results) &nbsp;·&nbsp; [Quick Start](#quick-start) &nbsp;·&nbsp; [Usage Guidelines](#usage-guidelines) &nbsp;·&nbsp; [License](#license) &nbsp;·&nbsp; [About](#about-this-repository) &nbsp;·&nbsp; [Acknowledgments](#acknowledgments)
+
+</div>
+
+---
+
+<!-- AUTHORS -->
+<div align="center">
+
+  <a name="authors"></a>
+  ## Authors
+
+| <a href="https://github.com/Amey-Thakur"><img src="https://github.com/Amey-Thakur.png" width="150" height="150" alt="Amey Thakur"></a><br>[**Amey Thakur**](https://github.com/Amey-Thakur)<br><br>[![ORCID](https://img.shields.io/badge/ORCID-0000--0001--5644--1575-green.svg)](https://orcid.org/0000-0001-5644-1575) | <a href="https://github.com/msatmod"><img src="Mega/Mega.png" width="150" height="150" alt="Mega Satish"></a><br>[**Mega Satish**](https://github.com/msatmod)<br><br>[![ORCID](https://img.shields.io/badge/ORCID-0000--0002--1844--9557-green.svg)](https://orcid.org/0000-0002-1844-9557) |
+| :---: | :---: |
+
+</div>
+
+> [!IMPORTANT]
+> ### 🤝🏻 Special Acknowledgement
+> *Special thanks to **[Mega Satish](https://github.com/msatmod)** for her meaningful contributions, guidance, and support that helped shape this work.*
+
+---
+
+<!-- OVERVIEW -->
+<a name="overview"></a>
+## Overview
+
+**Depression Detection Using Tweets** is a specialized Machine Learning framework designed to translate complex linguistic patterns into empirical psychological insights. This repository prioritizes **high-dimensional feature extraction** and **probabilistic classification** to provide a robust baseline for sentiment analysis within the context of mental health monitoring.
+
+*   **Linguistic Determinism**: The system utilizes deep NLP preprocessing, including lemmatization and entity normalization, to ensure that the semantic core of a tweet is preserved regardless of slang or stylistic variation.
+*   **Vector-Space Inference**: By leveraging **Support Vector Machines (SVM)** and **TF-IDF vectorization**, the model maps textual input into a multi-dimensional hyperplane, enabling precise binary classification of depressive sentiment.
+*   **Architectural Efficiency**: The backend is architected for low-latency serving via Flask, ensuring that model inference and result rendering occur in sub-second cycles, critical for interactive user feedback.
+
+> [!TIP]
+> **NLP Pipeline Optimization**
+>
+> To maximize classification reliability, the engine employs a **multi-stage linguistic filter**. **Stop-word suppression** and **morphological analysis** strip away structural noise, while the **en_core_web_lg** transformer model contextualizes surviving tokens. This ensures the classifier’s weights are strictly coupled with affective indicators, minimizing the false-positive skew common in generalized sentiment analysis models.
+
+---
+
+<!-- FEATURES -->
+<a name="features"></a>
+## Features
+
+| Feature | Description |
+|---------|-------------|
+| **Core SVM Model** | **High-Dimensional Classification** engine optimized for binary depressive sentiment prediction. |
+| **NLP Pipeline** | Deep linguistic feature extraction powered by the **spaCy transformer model** (`en_core_web_lg`). |
+| **Prediction Hub** | **Real-Time Inference Interface** built with Flask for sub-second classification feedback. |
+| **Security Suite** | Integrated **Browser-Side Integrity** protocols including anti-right-click and anti-select systems. |
+| **Cinematic Surprise** | **Immersive Branding Overlay** featuring animated Twitter iconography and synchronized audio. |
+
+> [!NOTE]
+> ### Technical Polish: The Linguistic Singularity
+> We have engineered a **Probabilistic Sentiment Manager** that calibrates model weights across thousands of TF-IDF vectors to simulate human-like linguistic intuition. The visual language focuses on a "Neural Slate" aesthetic, ensuring maximum cognitive focus on the diagnostic outputs without procedural distraction.
+
+### Tech Stack
+- **Languages**: Python 3.9+
+- **Logic**: **SVM Classifier** (Scikit-Learn Inference Engine)
+- **Linguistic Data**: **spaCy NLP** (Transformer-based word embeddings)
+- **Web App**: **Flask Framework** (Micro-service architecture for model serving)
+- **UI System**: Premium Modern Aesthetics (Custom CSS / Play Typography)
+- **Deployment**: Standard Python Environment (PIP-managed dependencies)
+
+---
+
+<!-- PROJECT STRUCTURE -->
+<a name="project-structure"></a>
+## Project Structure
+
+```python
+DEPRESSION-DETECTION-USING-TWEETS/
+│
+├── docs/                            # Technical Documentation
+│   └── SPECIFICATION.md             # Architecture & Design Specification
+│
+├── Mega/                            # Archival Attribution Assets
+│   ├── Filly.jpg                    # Companion (Filly)
+│   └── Mega.png                     # Author Profile Image (Mega Satish)
+│
+├── screenshots/                     # Project Visualization Gallery
+│   ├── 01_landing_page.png          # System Hub Initial State
+│   ├── 02_footer_details.png        # Brand and Metadata Footer
+│   ├── 03_surprise_cinematic.png    # Interactive Animated Sequence
+│   ├── 04_predict_interface.png     # Sentiment Analysis Entry Point
+│   ├── 05_analysis_output.png       # Model Inference result
+│   └── 06_result_prediction.png     # Final Sentiment Output
+│
+├── source_code/                     # Primary Application Layer
+│   ├── assets/                      # Serialized Models & Linguistic Data
+│   ├── core/                        # ML Pipeline (Clean, Train, Predict)
+│   ├── static/                      # Styling, Audio, & Security Scripts
+│   ├── templates/                   # HTML Templates (Index, Result, 404)
+│   └── app.py                       # Flask Application (Entry Point)
+│
+├── .gitattributes                   # Git configuration
+├── .gitignore                       # Repository Filters
+├── CITATION.cff                     # Scholarly Citation Metadata
+├── codemeta.json                    # Machine-Readable Project Metadata
+├── LICENSE                          # MIT License Terms
+├── README.md                        # Comprehensive Scholarly Entrance
+└── SECURITY.md                      # Security Policy & Protocol
+```
+
+---
+
+<!-- RESULTS -->
+<a name="results"></a>
+## Results
+
+<div align="center">
+  <b>Main Landing: System Hub Initialization</b>
+  <br>
+  <i>Minimalist interface for rapid tweet sentiment analysis.</i>
+  <br><br>
+  <img src="screenshots/01_landing_page.png" alt="Landing Page" width="90%">
+  <br><br><br>
+
+  <b>Metadata Synthesis: Branding and Footer Detail</b>
+  <br>
+  <i>Scholarly attribution and project status integration.</i>
+  <br><br>
+  <img src="screenshots/02_footer_details.png" alt="Footer Details" width="90%">
+  <br><br><br>
+
+  <b>Interactivity: Animated Twitter Sequence</b>
+  <br>
+  <i>Immersive audiovisual overlay triggered by core branding elements.</i>
+  <br><br>
+  <img src="screenshots/03_surprise_cinematic.png" alt="Cinematic Surprise" width="90%">
+  <br><br><br>
+
+  <b>Sentiment Entry: Real-time Analysis Interface</b>
+  <br>
+  <i>Direct manipulation environment for high-latency textual input.</i>
+  <br><br>
+  <img src="screenshots/04_predict_interface.png" alt="Predict Interface" width="90%">
+  <br><br><br>
+
+  <b>Model Inference: Feature Extraction Output</b>
+  <br>
+  <i>Deep linguistic analysis and probabilistic score generation.</i>
+  <br><br>
+  <img src="screenshots/05_analysis_output.png" alt="Analysis Output" width="90%">
+  <br><br><br>
+
+  <b>Statistical Output: Final Sentiment Classification</b>
+  <br>
+  <i>Categorized classification results with immediate visual feedback.</i>
+  <br><br>
+  <img src="screenshots/06_result_prediction.png" alt="Result Prediction" width="90%">
+</div>
+
+---
+
+<!-- QUICK START -->
+<a name="quick-start"></a>
+## Quick Start
+
+### 1. Prerequisites
+- **Python 3.11+**: Required for runtime execution. [Download Python](https://www.python.org/downloads/)
+- **Git**: For version control and cloning. [Download Git](https://git-scm.com/downloads)
+
+> [!WARNING]
+> **Data Acquisition & Memory Constraints**
+>
+> The linguistic pipeline relies on the **en_core_web_lg** transformer model, which requires an initial download of approximately **800MB**. Ensure a stable network connection during setup. Additionally, loading this model into memory requires at least **2GB of available RAM** to prevent swapping and ensure low-latency inference.
+
+### 2. Installation & Setup
+
+#### Step 1: Clone the Repository
+Open your terminal and clone the repository:
+```bash
+git clone https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS.git
+cd DEPRESSION-DETECTION-USING-TWEETS
+```
+
+#### Step 2: Configure Virtual Environment
+Prepare an isolated environment to manage dependencies:
+
+**Windows (Command Prompt / PowerShell):**
+```bash
+python -m venv venv
+venv\Scripts\activate
+```
+
+**macOS / Linux (Terminal):**
+```bash
+python3 -m venv venv
+source venv/bin/activate
+```
+
+#### Step 3: Install Core Dependencies
+Ensure your environment is active, then install the required libraries:
+```bash
+pip install -r "Source Code/requirements.txt"
+```
+
+#### Step 4: Linguistic Model Acquisition
+Download the large-scale linguistic model required for analysis (approx. 800MB):
+```bash
+python -m spacy download en_core_web_lg
+```
+
+### 3. Execution
+Launch the sentiment analysis dashboard:
+
+```bash
+python "Source Code/app.py"
+```
+
+---
+
+<!-- USAGE GUIDELINES -->
+<a name="usage-guidelines"></a>
+## Usage Guidelines
+
+This repository is openly shared to support learning and knowledge exchange across the academic community.
+
+**For Students**  
+Use this project as reference material for understanding **Support Vector Machines (SVM)**, **spaCy NLP pipelines**, and **sentiment analysis within the context of mental health monitoring**. The source code is available for study to facilitate self-paced learning and exploration of **high-dimensional feature extraction and model serving via Flask**.
+
+**For Educators**  
+This project may serve as a practical lab example or supplementary teaching resource for **Data Science**, **Natural Language Processing**, and **Machine Learning** courses. Attribution is appreciated when utilizing content.
+
+**For Researchers**  
+The documentation and architectural approach may provide insights into **academic project structuring**, **psychological linguistic modeling**, and **algorithmic deployment**.
+
+---
+
+<!-- LICENSE -->
+<a name="license"></a>
+## License
+
+This repository and all its creative and technical assets are made available under the **MIT License**. See the [LICENSE](LICENSE) file for complete terms.
+
+> [!NOTE]
+> **Summary**: You are free to share and adapt this content for any purpose, even commercially, as long as you provide appropriate attribution to the original authors.
+
+Copyright © 2022 Amey Thakur & Mega Satish
+
+---
+
+<!-- ABOUT -->
+<a name="about-this-repository"></a>
+## About This Repository
+
+**Created & Maintained by**: [Amey Thakur](https://github.com/Amey-Thakur) & [Mega Satish](https://github.com/msatmod)
+
+This project features **Depression Detection**, a high-performance sentiment analysis system. It represents a personal exploration into **Python**-based machine learning and interactive web-service architecture.
+
+**Connect:** [GitHub](https://github.com/Amey-Thakur) &nbsp;·&nbsp; [LinkedIn](https://www.linkedin.com/in/amey-thakur) &nbsp;·&nbsp; [ORCID](https://orcid.org/0000-0001-5644-1575)
+
+### Acknowledgments
+
+Grateful acknowledgment to [**Mega Satish**](https://github.com/msatmod) for her exceptional collaboration and scholarly partnership during the development of this machine learning project. Her constant support, technical clarity, and dedication to software quality were instrumental in achieving the system's functional objectives. Learning alongside her was a transformative experience; her thoughtful approach to problem-solving and steady encouragement turned complex requirements into meaningful learning moments. This work reflects the growth and insights gained from our side-by-side academic journey. Thank you, Mega, for everything you shared and taught along the way.
+
+Special thanks to the **mentors and peers** whose encouragement, discussions, and support contributed meaningfully to this learning experience.
+
+---
+
+<div align="center">
+
+  [↑ Back to Top](#readme-top)
+
+  [Authors](#authors) &nbsp;·&nbsp; [Overview](#overview) &nbsp;·&nbsp; [Features](#features) &nbsp;·&nbsp; [Structure](#project-structure) &nbsp;·&nbsp; [Results](#results) &nbsp;·&nbsp; [Quick Start](#quick-start) &nbsp;·&nbsp; [Usage Guidelines](#usage-guidelines) &nbsp;·&nbsp; [License](#license) &nbsp;·&nbsp; [About](#about-this-repository) &nbsp;·&nbsp; [Acknowledgments](#acknowledgments)
+
+  <br>
+
+  🧠 **[DEPRESSION-DETECTION](https://huggingface.co/spaces/ameythakur/Depression-Detection-Using-Tweets)**
+
+  ---
+
+  ### 🎓 [Computer Engineering Repository](https://github.com/Amey-Thakur/COMPUTER-ENGINEERING)
+
+  **Computer Engineering (B.E.)**
+
+  *Semester-wise curriculum, laboratories, projects, and academic notes.*
+
 </div>

source_code/app.py

@@ -0,0 +1,59 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Flask application entry point for the tweet analysis project.
+# ==============================================================================
+
+#!/usr/bin/env python3
+
+import pickle
+from flask import Flask, request, render_template
+from flask_bootstrap import Bootstrap
+import app_utilities
+
+# Initialize the Flask application
+# Flask-Bootstrap is utilized for enhanced UI styling consistency
+app = Flask(__name__)
+Bootstrap(app)
+
+@app.route('/')
+def index():
+    """Renders the landing page for tweet input."""
+    return render_template('index.html')
+
+@app.route('/predict', methods=['POST'])
+def predict():
+    """
+    Handles the form submission and displays the prediction result.
+        
+    Returns:
+        Rendered result HTML with the model's prediction outcome.
+    """
+    if request.method == 'POST':
+        # Retrieve the tweet content submitted via the web interface
+        tweet = request.form["tweet"]
+        input_data = [tweet]
+        
+        # Invoke the backend prediction utility to classify the tweet's sentiment
+        # The engine utilizes an SVM classifier with spaCy word embeddings
+        my_prediction = app_utilities.tweet_prediction(str(input_data))
+        
+        return render_template("result.html", prediction=my_prediction, name=tweet)
+
+@app.errorhandler(404)
+def page_not_found(e):
+    """
+    Custom 404 error handler.
+    Renders the personalized 404 page when a resource is not found.
+    """
+    return render_template('404.html'), 404
+
+# Entry point for the Flask development server
+if __name__ == '__main__':
+    # Execution on port 7860 as required for Hugging Face Spaces
+    app.run(host='0.0.0.0', port=7860)

source_code/app_utilities.py

@@ -0,0 +1,65 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Utility module for tweet analysis predictions.
+# ==============================================================================
+
+import sys
+import pickle
+import warnings
+import numpy as np
+import pandas as pd
+import spacy
+import en_core_web_lg
+# Configure sys.path to permit localized module discovery within the core directory
+sys.path.append('./core')
+
+import clean_utilities as CU
+
+# Suppression of non-critical runtime warnings to maintain a clean console log
+warnings.filterwarnings("ignore")
+
+def tweet_prediction(tweet: str) -> int:
+    """
+    Takes a tweet and returns whether it's classified as depressive (1) or not (0).
+    
+    The process:
+        1. Clean the text using our utility module.
+        2. Convert text to numbers using spaCy.
+        3. Use the trained SVM model to make a prediction.
+    Args:
+        tweet (str): The tweet text from the user.
+
+    Returns:
+        int: 1 for Depressive, 0 for Non-depressive.
+    """
+    # Step 1: Clean the text
+    processed_tweet = tweet
+    cleaned_input = []
+    cleaned_input.append(CU.tweets_cleaner(processed_tweet))
+    
+    # Step 2: Convert text to numbers using spaCy
+    nlp_engine = en_core_web_lg.load()
+    
+    # Step 3: Compute centroid word embeddings
+    # We calculate the mean vector of all tokens to represent the tweet's semantic context
+    semantic_vectors = np.array([
+        np.array([token.vector for token in nlp_engine(s)]).mean(axis=0) * np.ones((300))
+        for s in cleaned_input
+    ])
+    
+    # Step 4: Load the pre-trained Support Vector Machine (SVM) model artifact
+    # The SVM was selected for its robust performance in high-dimensional text classification
+    model_path = "./assets/models/model_svm1.pkl"
+    with open(model_path, 'rb') as model_file:
+        classifier = pickle.load(model_file)
+    
+    # Step 5: Perform binary classification
+    prediction_result = classifier.predict(semantic_vectors)
+    
+    return int(prediction_result[0])

source_code/assets/data/external/README.md

@@ -0,0 +1,3 @@
+## External dataset:
+
+We need two types of datasets one with tweets containing depressive characteristic which is obtained from twitter API and the other one with  random tweets which is available in one of the [Kaggle datasets](https://www.kaggle.com/ywang311/twitter-sentiment/data). 

source_code/assets/models/model_LSTM.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b53fb9941244f6f5ad410246b5606c222763d55462b66b0be08d1845c3dd8574
+size 81

source_code/assets/models/model_LogReg.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc7de925c22a478f8168b698570b9775b28613847b0dfa998fe972a7c9273a0e
+size 197693

source_code/assets/models/model_svm.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db18cfbfd954728d2091154eda313a9c1ffb0f4a1778e878985c055181b17170
+size 24450690

source_code/assets/models/model_svm1.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84c846534b54571c35b0d34bc73d948a0c230e9047ce94df3796750c14e18351
+size 24450707

source_code/assets/notebooks/data_gathering_twint.ipynb

@@ -0,0 +1,483 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "name": "Data_Gathering_Twint.ipynb",
+      "provenance": [],
+      "collapsed_sections": []
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "QgxmLN_lyiCS",
+        "outputId": "58d94201-559a-45fc-8e85-3dc7b73968fd"
+      },
+      "source": [
+        "from google.colab import drive\n",
+        "drive.mount('/content/drive')"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Mounted at /content/drive\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "-nI0XtuNwnmQ",
+        "outputId": "66c8ac16-109f-4ba6-e556-574091755cfc"
+      },
+      "source": [
+        "!git clone https://github.com/twintproject/twint.git"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Cloning into 'twint'...\n",
+            "remote: Enumerating objects: 4457, done.\u001b[K\n",
+            "remote: Counting objects: 100% (4/4), done.\u001b[K\n",
+            "remote: Compressing objects: 100% (4/4), done.\u001b[K\n",
+            "remote: Total 4457 (delta 0), reused 2 (delta 0), pack-reused 4453\u001b[K\n",
+            "Receiving objects: 100% (4457/4457), 4.47 MiB | 13.40 MiB/s, done.\n",
+            "Resolving deltas: 100% (2634/2634), done.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "DcLRNsvGya2i"
+      },
+      "source": [
+        "import os\n",
+        "os.chdir(\"/content/twint\")"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "XV0Tp_SQydvh"
+      },
+      "source": [
+        "!pip freeze > requirements.txt"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "84dyXWmLyrsn",
+        "outputId": "57188228-60e3-4a80-b3b9-737362a81227"
+      },
+      "source": [
+        "!pip install ."
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Processing /content/twint\n",
+            "\u001b[33m  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
+            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\u001b[0m\n",
+            "Collecting aiohttp\n",
+            "  Downloading aiohttp-3.7.4.post0-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)\n",
+            "\u001b[K     |████████████████████████████████| 1.3 MB 7.3 MB/s \n",
+            "\u001b[?25hCollecting aiodns\n",
+            "  Downloading aiodns-3.0.0-py3-none-any.whl (5.0 kB)\n",
+            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (4.6.3)\n",
+            "Collecting cchardet\n",
+            "  Downloading cchardet-2.1.7-cp37-cp37m-manylinux2010_x86_64.whl (263 kB)\n",
+            "\u001b[K     |████████████████████████████████| 263 kB 47.9 MB/s \n",
+            "\u001b[?25hCollecting dataclasses\n",
+            "  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)\n",
+            "Collecting elasticsearch\n",
+            "  Downloading elasticsearch-7.15.1-py2.py3-none-any.whl (378 kB)\n",
+            "\u001b[K     |████████████████████████████████| 378 kB 70.9 MB/s \n",
+            "\u001b[?25hRequirement already satisfied: pysocks in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.7.1)\n",
+            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.1.5)\n",
+            "Collecting aiohttp_socks\n",
+            "  Downloading aiohttp_socks-0.6.0-py3-none-any.whl (9.2 kB)\n",
+            "Collecting schedule\n",
+            "  Downloading schedule-1.1.0-py2.py3-none-any.whl (10 kB)\n",
+            "Requirement already satisfied: geopy in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.17.0)\n",
+            "Collecting fake-useragent\n",
+            "  Downloading fake-useragent-0.1.11.tar.gz (13 kB)\n",
+            "Collecting googletransx\n",
+            "  Downloading googletransx-2.4.2.tar.gz (13 kB)\n",
+            "Collecting pycares>=4.0.0\n",
+            "  Downloading pycares-4.0.0-cp37-cp37m-manylinux2010_x86_64.whl (291 kB)\n",
+            "\u001b[K     |████████████████████████████████| 291 kB 59.7 MB/s \n",
+            "\u001b[?25hRequirement already satisfied: cffi>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pycares>=4.0.0->aiodns->twint==2.1.21) (1.14.6)\n",
+            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns->twint==2.1.21) (2.20)\n",
+            "Collecting multidict<7.0,>=4.5\n",
+            "  Downloading multidict-5.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (160 kB)\n",
+            "\u001b[K     |████████████████████████████████| 160 kB 67.8 MB/s \n",
+            "\u001b[?25hRequirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (3.7.4.3)\n",
+            "Requirement already satisfied: chardet<5.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (3.0.4)\n",
+            "Collecting yarl<2.0,>=1.0\n",
+            "  Downloading yarl-1.7.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)\n",
+            "\u001b[K     |████████████████████████████████| 271 kB 65.8 MB/s \n",
+            "\u001b[?25hCollecting async-timeout<4.0,>=3.0\n",
+            "  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)\n",
+            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (21.2.0)\n",
+            "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl<2.0,>=1.0->aiohttp->twint==2.1.21) (2.10)\n",
+            "Collecting python-socks[asyncio]>=1.2.2\n",
+            "  Downloading python_socks-1.2.4-py3-none-any.whl (35 kB)\n",
+            "Requirement already satisfied: urllib3<2,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint==2.1.21) (1.24.3)\n",
+            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint==2.1.21) (2021.5.30)\n",
+            "Requirement already satisfied: geographiclib<2,>=1.49 in /usr/local/lib/python3.7/dist-packages (from geopy->twint==2.1.21) (1.52)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from googletransx->twint==2.1.21) (2.23.0)\n",
+            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->twint==2.1.21) (2018.9)\n",
+            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint==2.1.21) (2.8.2)\n",
+            "Requirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.7/dist-packages (from pandas->twint==2.1.21) (1.19.5)\n",
+            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->twint==2.1.21) (1.15.0)\n",
+            "Building wheels for collected packages: twint, fake-useragent, googletransx\n",
+            "  Building wheel for twint (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+            "  Created wheel for twint: filename=twint-2.1.21-py3-none-any.whl size=38870 sha256=a648841e8abdbeafa3718d69334377eef20411c64b11553f70be9937a84be56a\n",
+            "  Stored in directory: /tmp/pip-ephem-wheel-cache-oe_ws1ie/wheels/f7/3e/11/2803f3c6890e87a9bec35bb8e37ef1ad0777a00f43e2441fb1\n",
+            "  Building wheel for fake-useragent (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+            "  Created wheel for fake-useragent: filename=fake_useragent-0.1.11-py3-none-any.whl size=13502 sha256=db8ea0f861a9b913fb4822f3cbfb76d4ff27a371f7c125657f7cbb17766fc316\n",
+            "  Stored in directory: /root/.cache/pip/wheels/ed/f7/62/50ab6c9a0b5567267ab76a9daa9d06315704209b2c5d032031\n",
+            "  Building wheel for googletransx (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+            "  Created wheel for googletransx: filename=googletransx-2.4.2-py3-none-any.whl size=15968 sha256=513c5ee44dad1d794a351939b43e11cd2a5d2fbe541ad2a9271d661c96e29221\n",
+            "  Stored in directory: /root/.cache/pip/wheels/66/d5/b1/31104b338f7fd45aa8f7d22587765db06773b13df48a89735f\n",
+            "Successfully built twint fake-useragent googletransx\n",
+            "Installing collected packages: multidict, yarl, python-socks, async-timeout, pycares, aiohttp, schedule, googletransx, fake-useragent, elasticsearch, dataclasses, cchardet, aiohttp-socks, aiodns, twint\n",
+            "Successfully installed aiodns-3.0.0 aiohttp-3.7.4.post0 aiohttp-socks-0.6.0 async-timeout-3.0.1 cchardet-2.1.7 dataclasses-0.6 elasticsearch-7.15.1 fake-useragent-0.1.11 googletransx-2.4.2 multidict-5.2.0 pycares-4.0.0 python-socks-1.2.4 schedule-1.1.0 twint-2.1.21 yarl-1.7.0\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "b8rOKGVQ8OBq",
+        "outputId": "499b3126-f2b4-4799-f265-5065322a0146"
+      },
+      "source": [
+        "!pip install -U git+https://github.com/cyxv/twint.git@master"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Collecting git+https://github.com/cyxv/twint.git@master\n",
+            "  Cloning https://github.com/cyxv/twint.git (to revision master) to /tmp/pip-req-build-bjyd0ng2\n",
+            "  Running command git clone -q https://github.com/cyxv/twint.git /tmp/pip-req-build-bjyd0ng2\n",
+            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (3.7.4.post0)\n",
+            "Requirement already satisfied: aiodns in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (3.0.0)\n",
+            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (4.6.3)\n",
+            "Requirement already satisfied: cchardet in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (2.1.7)\n",
+            "Requirement already satisfied: dataclasses in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (0.6)\n",
+            "Requirement already satisfied: elasticsearch in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (7.15.1)\n",
+            "Requirement already satisfied: pysocks in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.7.1)\n",
+            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.1.5)\n",
+            "Requirement already satisfied: aiohttp_socks in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (0.6.0)\n",
+            "Requirement already satisfied: schedule in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.1.0)\n",
+            "Requirement already satisfied: geopy in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (1.17.0)\n",
+            "Requirement already satisfied: fake-useragent in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (0.1.11)\n",
+            "Requirement already satisfied: googletransx in /usr/local/lib/python3.7/dist-packages (from twint==2.1.21) (2.4.2)\n",
+            "Requirement already satisfied: pycares>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from aiodns->twint==2.1.21) (4.0.0)\n",
+            "Requirement already satisfied: cffi>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pycares>=4.0.0->aiodns->twint==2.1.21) (1.14.6)\n",
+            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns->twint==2.1.21) (2.20)\n",
+            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (1.7.0)\n",
+            "Requirement already satisfied: chardet<5.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (3.0.4)\n",
+            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (5.2.0)\n",
+            "Requirement already satisfied: async-timeout<4.0,>=3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (3.0.1)\n",
+            "Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (3.7.4.3)\n",
+            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint==2.1.21) (21.2.0)\n",
+            "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl<2.0,>=1.0->aiohttp->twint==2.1.21) (2.10)\n",
+            "Requirement already satisfied: python-socks[asyncio]>=1.2.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp_socks->twint==2.1.21) (1.2.4)\n",
+            "Requirement already satisfied: urllib3<2,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint==2.1.21) (1.24.3)\n",
+            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint==2.1.21) (2021.5.30)\n",
+            "Requirement already satisfied: geographiclib<2,>=1.49 in /usr/local/lib/python3.7/dist-packages (from geopy->twint==2.1.21) (1.52)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from googletransx->twint==2.1.21) (2.23.0)\n",
+            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->twint==2.1.21) (2018.9)\n",
+            "Requirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.7/dist-packages (from pandas->twint==2.1.21) (1.19.5)\n",
+            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint==2.1.21) (2.8.2)\n",
+            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->twint==2.1.21) (1.15.0)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "PKbYSKBJy2Ow",
+        "outputId": "8f24189f-56ec-41d5-f8c0-475ffb3b1bc2"
+      },
+      "source": [
+        "!pip install nest_asyncio"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.7/dist-packages (1.5.1)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "Fks7CZYMy5cR",
+        "outputId": "180f0f46-4fca-4e7f-a134-2cf99283a6a4"
+      },
+      "source": [
+        "!pip3 install twint"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Requirement already satisfied: twint in /usr/local/lib/python3.7/dist-packages (2.1.21)\n",
+            "Requirement already satisfied: schedule in /usr/local/lib/python3.7/dist-packages (from twint) (1.1.0)\n",
+            "Requirement already satisfied: pysocks in /usr/local/lib/python3.7/dist-packages (from twint) (1.7.1)\n",
+            "Requirement already satisfied: cchardet in /usr/local/lib/python3.7/dist-packages (from twint) (2.1.7)\n",
+            "Requirement already satisfied: fake-useragent in /usr/local/lib/python3.7/dist-packages (from twint) (0.1.11)\n",
+            "Requirement already satisfied: elasticsearch in /usr/local/lib/python3.7/dist-packages (from twint) (7.15.1)\n",
+            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from twint) (3.7.4.post0)\n",
+            "Requirement already satisfied: googletransx in /usr/local/lib/python3.7/dist-packages (from twint) (2.4.2)\n",
+            "Requirement already satisfied: geopy in /usr/local/lib/python3.7/dist-packages (from twint) (1.17.0)\n",
+            "Requirement already satisfied: aiohttp-socks in /usr/local/lib/python3.7/dist-packages (from twint) (0.6.0)\n",
+            "Requirement already satisfied: aiodns in /usr/local/lib/python3.7/dist-packages (from twint) (3.0.0)\n",
+            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from twint) (1.1.5)\n",
+            "Requirement already satisfied: dataclasses in /usr/local/lib/python3.7/dist-packages (from twint) (0.6)\n",
+            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from twint) (4.6.3)\n",
+            "Requirement already satisfied: pycares>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from aiodns->twint) (4.0.0)\n",
+            "Requirement already satisfied: cffi>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pycares>=4.0.0->aiodns->twint) (1.14.6)\n",
+            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns->twint) (2.20)\n",
+            "Requirement already satisfied: chardet<5.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (3.0.4)\n",
+            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (1.7.0)\n",
+            "Requirement already satisfied: async-timeout<4.0,>=3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (3.0.1)\n",
+            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (21.2.0)\n",
+            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (5.2.0)\n",
+            "Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (3.7.4.3)\n",
+            "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl<2.0,>=1.0->aiohttp->twint) (2.10)\n",
+            "Requirement already satisfied: python-socks[asyncio]>=1.2.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp-socks->twint) (1.2.4)\n",
+            "Requirement already satisfied: urllib3<2,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint) (1.24.3)\n",
+            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint) (2021.5.30)\n",
+            "Requirement already satisfied: geographiclib<2,>=1.49 in /usr/local/lib/python3.7/dist-packages (from geopy->twint) (1.52)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from googletransx->twint) (2.23.0)\n",
+            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (2.8.2)\n",
+            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (2018.9)\n",
+            "Requirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (1.19.5)\n",
+            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->twint) (1.15.0)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "qZyxEstcy8R9"
+      },
+      "source": [
+        "# Import required libraries\n",
+        "import nest_asyncio\n",
+        "nest_asyncio.apply()\n",
+        "import pandas as pd\n",
+        "import twint\n",
+        "import pandas as pd\n",
+        "import re"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 371
+        },
+        "id": "oAkguWB20dB4",
+        "outputId": "e595e29d-f470-4f3c-9846-32a60f7ed2ef"
+      },
+      "source": [
+        "# add some tweets with depressed and depression tags, for a particular year\n",
+        "\n",
+        "depress_tags = [\"#depressed\", \"#anxiety\", \"#depression\", \"#suicide\", \"#mentalhealth\"\n",
+        "                \"#loneliness\", \"#hopelessness\", \"#itsokaynottobeokay\"]\n",
+        "\n",
+        "content = {}\n",
+        "for i in range(len(depress_tags)):\n",
+        "    print(depress_tags[i])\n",
+        "    c = twint.Config()\n",
+        "    \n",
+        "    c.Format = \"Tweet id: {id} | Tweet: {tweet}\"\n",
+        "    c.Search = depress_tags[i]\n",
+        "    c.Limit = 1000\n",
+        "    c.Year = 2019\n",
+        "    c.Lang = \"en\"\n",
+        "    c.Store_csv = True\n",
+        "    c.Store_Object = True\n",
+        "    c.Output = \"/content/drive/MyDrive/NLP/Depression_Detection/depressive_en_2019.csv\"\n",
+        "    c.Hide_output = True\n",
+        "    c.Stats = True\n",
+        "    c.Lowercase  = True\n",
+        "    c.Filter_retweets = True\n",
+        "    twint.run.Search(c)"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "#depressed\n"
+          ]
+        },
+        {
+          "output_type": "error",
+          "ename": "TypeError",
+          "evalue": "ignored",
+          "traceback": [
+            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
+            "\u001b[0;32m<ipython-input-3-092f46e39459>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     21\u001b[0m     \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLowercase\u001b[0m  \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m     \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFilter_retweets\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 23\u001b[0;31m     \u001b[0mtwint\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mSearch\u001b[0;34m(config, callback)\u001b[0m\n\u001b[1;32m    408\u001b[0m     \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFollowers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    409\u001b[0m     \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mProfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 410\u001b[0;31m     \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    411\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPandas_au\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    412\u001b[0m         \u001b[0mstorage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpanda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_autoget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tweet\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(config, callback)\u001b[0m\n\u001b[1;32m    327\u001b[0m         \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 329\u001b[0;31m     \u001b[0mget_event_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTwint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    331\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36mrun_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m     68\u001b[0m                 raise RuntimeError(\n\u001b[1;32m     69\u001b[0m                     'Event loop stopped before Future completed.')\n\u001b[0;32m---> 70\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/futures.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    179\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__log_traceback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/tasks.py\u001b[0m in \u001b[0;36m__step\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m    249\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    250\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 251\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mthrow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    252\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    253\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_must_cancel\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mmain\u001b[0;34m(self, callback)\u001b[0m\n\u001b[1;32m    233\u001b[0m             \u001b[0mtask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_done_callback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 235\u001b[0;31m         \u001b[0;32mawait\u001b[0m \u001b[0mtask\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    237\u001b[0m     \u001b[0;32masync\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/futures.py\u001b[0m in \u001b[0;36m__await__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_asyncio_future_blocking\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 263\u001b[0;31m             \u001b[0;32myield\u001b[0m \u001b[0mself\u001b[0m  \u001b[0;31m# This tells Task to wait for completion.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    264\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"await wasn't used with future\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/tasks.py\u001b[0m in \u001b[0;36m__wakeup\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m    316\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__wakeup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    317\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 318\u001b[0;31m             \u001b[0mfuture\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    319\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    320\u001b[0m             \u001b[0;31m# This may also be a cancellation.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/futures.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    179\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__log_traceback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/tasks.py\u001b[0m in \u001b[0;36m__step\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m    247\u001b[0m                 \u001b[0;31m# We use the `send` method directly, because coroutines\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    248\u001b[0m                 \u001b[0;31m# don't have `__iter__` and `__next__` methods.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 249\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    250\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mthrow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    284\u001b[0m                     \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTwitterSearch\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    285\u001b[0m                         \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Twint:main:twitter-search'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 286\u001b[0;31m                         \u001b[0;32mawait\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtweets\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    287\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    288\u001b[0m                     \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Twint:main:no-more-tweets'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mtweets\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    224\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mtweet\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    225\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcount\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 226\u001b[0;31m                 \u001b[0;32mawait\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTweets\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtweet\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    227\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    228\u001b[0m     \u001b[0;32masync\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/output.py\u001b[0m in \u001b[0;36mTweets\u001b[0;34m(tweets, config, conn)\u001b[0m\n\u001b[1;32m    164\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTwitterSearch\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mProfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Tweets:TwitterSearch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m         \u001b[0;32mawait\u001b[0m \u001b[0mcheckData\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtweets\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m         \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Tweets:else'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/output.py\u001b[0m in \u001b[0;36mcheckData\u001b[0;34m(tweet, config, conn)\u001b[0m\n\u001b[1;32m    135\u001b[0m         \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mdatecheck\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtweet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatestamp\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\" \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtweet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimestamp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 137\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mformat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTweet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtweet\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    138\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDatabase\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    139\u001b[0m             \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':checkData:Database'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/format.py\u001b[0m in \u001b[0;36mTweet\u001b[0;34m(config, t)\u001b[0m\n\u001b[1;32m     21\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{hashtags}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\",\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhashtags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{cashtags}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\",\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcashtags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 23\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{replies}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplies_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     24\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{retweets}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mretweets_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{likes}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlikes_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;31mTypeError\u001b[0m: replace() argument 2 must be str, not int"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 424
+        },
+        "id": "g4zJVi7sy_b2",
+        "outputId": "71712dc1-cfeb-4294-f883-86fc6ce82984"
+      },
+      "source": [
+        "# add some tweets with depressed and depression tags, for a particular year\n",
+        "\n",
+        "depress_tags = [\"#depressed\", \"#depression\", \"#loneliness\", \"#hopelessness\"]\n",
+        "\n",
+        "content = {}\n",
+        "for i in range(len(depress_tags)):\n",
+        "    print(depress_tags[i])\n",
+        "    c = twint.Config()\n",
+        "    \n",
+        "    c.Format = \"Tweet id: {id} | Tweet: {tweet}\"\n",
+        "    c.Search = depress_tags[i]\n",
+        "    c.Limit = 1000\n",
+        "    c.Year = 2020\n",
+        "    c.Store_csv = True\n",
+        "    c.Store_json = True\n",
+        "    c.Output = \"/content/drive/MyDrive/NLP/Depression_Detection/dataset_depression.json\"\n",
+        "    c.Hide_output = True\n",
+        "    c.Stats = True\n",
+        "    c.Lowercase  = True\n",
+        "    c.Filter_retweets = True\n",
+        "    twint.run.Search(c)"
+      ],
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "#depressed\n",
+            "[!] No more data! Scraping will stop now.\n",
+            "found 0 deleted tweets in this search.\n",
+            "#depression\n"
+          ]
+        },
+        {
+          "output_type": "error",
+          "ename": "TypeError",
+          "evalue": "ignored",
+          "traceback": [
+            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
+            "\u001b[0;32m<ipython-input-6-d584c0441bfc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLowercase\u001b[0m  \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFilter_retweets\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m     \u001b[0mtwint\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mSearch\u001b[0;34m(config, callback)\u001b[0m\n\u001b[1;32m    408\u001b[0m     \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFollowers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    409\u001b[0m     \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mProfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 410\u001b[0;31m     \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    411\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPandas_au\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    412\u001b[0m         \u001b[0mstorage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpanda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_autoget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tweet\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(config, callback)\u001b[0m\n\u001b[1;32m    327\u001b[0m         \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 329\u001b[0;31m     \u001b[0mget_event_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTwint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    331\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nest_asyncio.py\u001b[0m in \u001b[0;36mrun_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m     68\u001b[0m                 raise RuntimeError(\n\u001b[1;32m     69\u001b[0m                     'Event loop stopped before Future completed.')\n\u001b[0;32m---> 70\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/futures.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    179\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__log_traceback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/tasks.py\u001b[0m in \u001b[0;36m__step\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m    249\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    250\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 251\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mthrow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    252\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    253\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_must_cancel\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mmain\u001b[0;34m(self, callback)\u001b[0m\n\u001b[1;32m    233\u001b[0m             \u001b[0mtask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_done_callback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 235\u001b[0;31m         \u001b[0;32mawait\u001b[0m \u001b[0mtask\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    237\u001b[0m     \u001b[0;32masync\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/futures.py\u001b[0m in \u001b[0;36m__await__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_asyncio_future_blocking\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 263\u001b[0;31m             \u001b[0;32myield\u001b[0m \u001b[0mself\u001b[0m  \u001b[0;31m# This tells Task to wait for completion.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    264\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"await wasn't used with future\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/tasks.py\u001b[0m in \u001b[0;36m__wakeup\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m    316\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__wakeup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    317\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 318\u001b[0;31m             \u001b[0mfuture\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    319\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    320\u001b[0m             \u001b[0;31m# This may also be a cancellation.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/futures.py\u001b[0m in \u001b[0;36mresult\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    179\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__log_traceback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/lib/python3.7/asyncio/tasks.py\u001b[0m in \u001b[0;36m__step\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m    247\u001b[0m                 \u001b[0;31m# We use the `send` method directly, because coroutines\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    248\u001b[0m                 \u001b[0;31m# don't have `__iter__` and `__next__` methods.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 249\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    250\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcoro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mthrow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    284\u001b[0m                     \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTwitterSearch\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    285\u001b[0m                         \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Twint:main:twitter-search'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 286\u001b[0;31m                         \u001b[0;32mawait\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtweets\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    287\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    288\u001b[0m                     \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Twint:main:no-more-tweets'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/run.py\u001b[0m in \u001b[0;36mtweets\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    224\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mtweet\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    225\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcount\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 226\u001b[0;31m                 \u001b[0;32mawait\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTweets\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtweet\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    227\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    228\u001b[0m     \u001b[0;32masync\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/output.py\u001b[0m in \u001b[0;36mTweets\u001b[0;34m(tweets, config, conn)\u001b[0m\n\u001b[1;32m    164\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTwitterSearch\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mProfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Tweets:TwitterSearch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m         \u001b[0;32mawait\u001b[0m \u001b[0mcheckData\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtweets\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m         \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':Tweets:else'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/output.py\u001b[0m in \u001b[0;36mcheckData\u001b[0;34m(tweet, config, conn)\u001b[0m\n\u001b[1;32m    135\u001b[0m         \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mdatecheck\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtweet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatestamp\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\" \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtweet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimestamp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 137\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mformat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTweet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtweet\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    138\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDatabase\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    139\u001b[0m             \u001b[0mlogme\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m':checkData:Database'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/twint/format.py\u001b[0m in \u001b[0;36mTweet\u001b[0;34m(config, t)\u001b[0m\n\u001b[1;32m     21\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{hashtags}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\",\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhashtags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{cashtags}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\",\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcashtags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 23\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{replies}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplies_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     24\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{retweets}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mretweets_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{likes}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlikes_count\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+            "\u001b[0;31mTypeError\u001b[0m: replace() argument 2 must be str, not int"
+          ]
+        }
+      ]
+    }
+  ]
+}

source_code/core/clean.py

@@ -0,0 +1,72 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Utility for cleaning raw tweet data for analysis.
+# ==============================================================================
+
+import argparse
+import warnings
+import clean_utilities as CU
+
+# Suppression of non-critical runtime warnings to ensure output clarity
+warnings.filterwarnings("ignore")
+
+def main():
+    """
+    Primary execution routine for the tweet cleaning utility.
+    
+    This script facilitates the transformation of raw unstructured text 
+    into a standardized format, essential for downstream machine learning 
+    inference and training.
+    """
+    # Configuration of the command-line argument parser
+    parser = argparse.ArgumentParser(
+        description="Twitter Depression Detection: Text Cleaning Utility"
+    )
+
+    # Definition of the mandatory positional argument for input file path
+    parser.add_argument(
+        'filename', 
+        help="Path to the raw text file containing the tweet to be sanitized"
+    )
+
+    # Parsing and validation of terminal arguments
+    args = parser.parse_args()
+
+    # Conditional logic to verify input availability before processing
+    if args.filename is not None:
+        print(f"Targeting file for preprocessing: {args.filename}")
+        
+        try:
+            # Atomic read operation for the target text file
+            with open(args.filename, 'r', encoding='utf-8') as file:
+                raw_tweet = file.read()
+                
+                # Invocation of the granular cleaning pipeline
+                # Methodology includes contraction expansion, tokenization, and lemmatization
+                print("Linguistic cleaning in progress...")
+                sanitized_tweet = CU.tweets_cleaner(raw_tweet)
+                
+                # Persisting the sanitized result to local storage
+                with open('clean_tweet.txt', 'w', encoding='utf-8') as output_file:
+                    print("Sanitization complete. Persistence target: clean_tweet.txt")
+                    output_file.write(sanitized_tweet)
+                    
+        except FileNotFoundError:
+            print(f"Error: The specified file '{args.filename}' was not discovered.")
+        except Exception as e:
+            print(f"An unexpected analytical error occurred: {e}")
+            
+    else:
+        print("Required input: Please specify a valid filename as a positional argument.")
+
+if __name__ == '__main__':
+    main()
+
+
+

source_code/core/clean_utilities.py

@@ -0,0 +1,226 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Core NLP logic for cleaning and normalizing tweet text.
+# ==============================================================================
+
+import re
+import warnings
+import nltk
+import ftfy
+from nltk.stem import WordNetLemmatizer
+from nltk.corpus import stopwords
+
+# Suppression of non-critical warnings to ensure a streamlined algorithmic log
+warnings.filterwarnings("ignore")
+
+# Dictionary of standard English contractions for lexical expansion
+# This facilitates uniform tokenization by resolving ambiguous shorthand
+CONTRACTIONS_LIST = {
+  "ain't": "am not",
+  "aren't": "are not",
+  "can't": "cannot",
+  "can't've": "cannot have",
+  "'cause": "because",
+  "could've": "could have",
+  "couldn't": "could not",
+  "couldn't've": "could not have",
+  "didn't": "did not",
+  "doesn't": "does not",
+  "don't": "do not",
+  "hadn't": "had not",
+  "hadn't've": "had not have",
+  "hasn't": "has not",
+  "haven't": "have not",
+  "he'd": "he would",
+  "he'd've": "he would have",
+  "he'll": "he will",
+  "he'll've": "he will have",
+  "he's": "he is",
+  "how'd": "how did",
+  "how'd'y": "how do you",
+  "how'll": "how will",
+  "how's": "how is",
+  "I'd": "I would",
+  "I'd've": "I would have",
+  "I'll": "I will",
+  "I'll've": "I will have",
+  "I'm": "I am",
+  "I've": "I have",
+  "isn't": "is not",
+  "it'd": "it had",
+  "it'd've": "it would have",
+  "it'll": "it will",
+  "it'll've": "it will have",
+  "it's": "it is",
+  "let's": "let us",
+  "ma'am": "madam",
+  "mayn't": "may not",
+  "might've": "might have",
+  "mightn't": "might not",
+  "mightn't've": "might not have",
+  "must've": "must have",
+  "mustn't": "must not",
+  "mustn't've": "must not have",
+  "needn't": "need not",
+  "needn't've": "need not have",
+  "o'clock": "of the clock",
+  "oughtn't": "ought not",
+  "oughtn't've": "ought not have",
+  "shan't": "shall not",
+  "sha'n't": "shall not",
+  "shan't've": "shall not have",
+  "she'd": "she would",
+  "she'd've": "she would have",
+  "she'll": "she will",
+  "she'll've": "she will have",
+  "she's": "she is",
+  "should've": "should have",
+  "shouldn't": "should not",
+  "shouldn't've": "should not have",
+  "so've": "so have",
+  "so's": "so is",
+  "that'd": "that would",
+  "that'd've": "that would have",
+  "that's": "that is",
+  "there'd": "there had",
+  "there'd've": "there would have",
+  "there's": "there is",
+  "they'd": "they would",
+  "they'd've": "they would have",
+  "they'll": "they will",
+  "they'll've": "they will have",
+  "they're": "they are",
+  "they've": "they have",
+  "to've": "to have",
+  "wasn't": "was not",
+  "we'd": "we had",
+  "we'd've": "we would have",
+  "we'll": "we will",
+  "we'll've": "we will have",
+  "we're": "we are",
+  "we've": "we have",
+  "weren't": "were not",
+  "what'll": "what will",
+  "what'll've": "what will have",
+  "what're": "what are",
+  "what's": "what is",
+  "what've": "what have",
+  "when's": "when is",
+  "when've": "when have",
+  "where'd": "where did",
+  "where's": "where is",
+  "where've": "where have",
+  "who'll": "who will",
+  "who'll've": "who will have",
+  "who's": "who is",
+  "who've": "who have",
+  "why's": "why is",
+  "why've": "why have",
+  "will've": "will have",
+  "won't": "will not",
+  "won't've": "will not have",
+  "would've": "would have",
+  "wouldn't": "would not",
+  "wouldn't've": "would not have",
+  "y'all": "you all",
+  "y'alls": "you alls",
+  "y'all'd": "you all would",
+  "y'all'd've": "you all would have",
+  "y'all're": "you all are",
+  "y'all've": "you all have",
+  "you'd": "you had",
+  "you'd've": "you would have",
+  "you'll": "you you will",
+  "you'll've": "you you will have",
+  "you're": "you are",
+  "you've": "you have"
+}
+
+# Pre-compiled regular expression for efficient contraction matching
+CONTRACTIONS_RE = re.compile('(%s)' % '|'.join(CONTRACTIONS_LIST.keys()))
+
+def expand_contractions(text: str, contractions_re=CONTRACTIONS_RE) -> str:
+    """
+    Identifies and replaces English contractions within the input text 
+    using a predefined mapping.
+    
+    Args:
+        text (str): The raw text potentially containing contractions.
+        contractions_re: Compiled regex pattern for matching contractions.
+
+    Returns:
+        str: Expanded lexical form of the input text.
+    """
+    def replace(match):
+        return CONTRACTIONS_LIST[match.group(0)]
+    return contractions_re.sub(replace, text)
+
+def tweets_cleaner(tweet: str) -> str:
+    """
+    Executes a comprehensive analytical pipeline for the linguistic 
+    normalization of microblogging content (Tweets).
+    
+    Analytical Methodology:
+        1. Case Normalization: Lowercasting to ensure uniformity.
+        2. Relevance Filtering: Exclusion of tweets consisting solely of URLs.
+        3. Noise Reduction: Removal of hashtags, mentions, and visual asset links.
+        4. Encoding Correction: Fixing malformed Unicode sequences (via ftfy).
+        5. Lexical Expansion: Resolution of linguistic contractions.
+        6. Punctuation Removal: Strategic elimination of non-alphanumeric noise.
+        7. Morphological Analysis: Removal of high-frequency stop words and 
+           application of WordNet-based lemmatization to reduce words to 
+           their base semantic roots.
+
+    Args:
+        tweet (str): Raw input tweet captured from the platform.
+
+    Returns:
+        str: Sanitized and normalized string ready for vectorization.
+    """
+    # Phase 1: Case Uniformity
+    tweet = tweet.lower()
+
+    # Phase 2: Structural Relevance Check (Filtering out pure URL content)
+    if re.match("(\w+:\/\/\S+)", tweet) is None:
+        
+        # Phase 3: Targeted entity removal (Handles Twitter-specific artifacts)
+        tweet = ' '.join(
+            re.sub(
+                "(@[A-Za-z0-9]+)|(\#[A-Za-z0-9]+)|(<Emoji:.*>)|(pic\.twitter\.com\/.*)", 
+                " ", 
+                tweet
+            ).split()
+        )
+
+        # Phase 4: Resolution of malformed character encodings
+        tweet = ftfy.fix_text(tweet)
+
+        # Phase 5: Applied contraction expansion for token consistency
+        tweet = expand_contractions(tweet)
+
+        # Phase 6: Punctuation and non-essential character pruning
+        tweet = ' '.join(re.sub("([^0-9A-Za-z \t])", " ", tweet).split())
+
+        # Phase 7: Stop-word filtration and Lemmatization
+        # Methodology: Reducing inflectional forms to a common base word (Lemma)
+        stop_words_set = set(stopwords.words('english'))
+        tokens = nltk.word_tokenize(tweet)
+
+        lemmatizer_engine = WordNetLemmatizer()
+        filtered_lexicon = [
+            lemmatizer_engine.lemmatize(word) 
+            for word in tokens 
+            if word not in stop_words_set
+        ]
+        
+        # Phase 8: Re-assembly of the normalized semantic string
+        tweet = ' '.join(filtered_lexicon)
+
+    return tweet
+

source_code/core/predict.py

@@ -0,0 +1,110 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Utility for predicting depression levels in tweets using SVM.
+# ==============================================================================
+
+import argparse
+import pickle
+import warnings
+import numpy as np
+import pandas as pd
+import spacy
+import en_core_web_lg
+import clean_utilities as CU
+
+# Suppression of non-critical runtime warnings to maintain output integrity
+warnings.filterwarnings("ignore")
+
+def main():
+    """
+    Main entry point for the prediction utility.
+    
+    This script encapsulates the end-to-end inference pipeline:
+        1. Argument Parsing: Captures input text file and model selection.
+        2. Text Preprocessing: Normalization via clean_utilities.
+        3. Feature Extraction: Generating centroid embeddings via spaCy.
+        4. Classification: Binary sentiment analysis via pre-trained SVM.
+    """
+    # Initialize the CLI argument parser with a descriptive header
+    parser = argparse.ArgumentParser(
+        description="Twitter Depression Detection: Machine Learning Inference Utility"
+    )
+
+    # Positional argument for the target tweet content (text file)
+    parser.add_argument(
+        'filename', 
+        help="Path to the text file containing the tweet for classification"
+    )
+
+    # Positional argument for the classification model type
+    parser.add_argument(
+        'model', 
+        help="Target model architecture (currently optimized for 'SVM')"
+    )
+
+    # Execution of the parsing logic
+    args = parser.parse_args()
+
+    # Pipeline validation: Ensuring input availability and model compatibility
+    if args.filename is not None and args.model == "SVM":
+        print(f"Loading input source: {args.filename}")
+        
+        try:
+            # Step 1: Data Acquisition
+            with open(args.filename, 'r', encoding='utf-8') as file:
+                raw_test_tweet = file.read()
+                print(f"Captured Content: \"{raw_test_tweet}\"")
+                
+                # Step 2: Linguistic Preprocessing
+                # Normalizes raw discourse into a tokenizable semantic format
+                print("Executing linguistic cleaning pipeline...")
+                cleaned_input = [CU.tweets_cleaner(raw_test_tweet)]
+                print(f"Normalized Form: {cleaned_input}")
+
+            # Step 3: Feature Space Transformation
+            # Utilizing dense word embeddings (spaCy 'en_core_web_lg' model)
+            print("Transforming text to 300-dimensional semantic vectors...")
+            nlp_engine = en_core_web_lg.load()
+            
+            # Generating the centroid vector representing the tweet's linguistic context
+            semantic_features = np.array([
+                np.array([token.vector for token in nlp_engine(s)]).mean(axis=0) * np.ones((300))
+                for s in cleaned_input
+            ])
+
+            # Step 4: Model Artifact Loading
+            # Loading the serialized SVM classifier from the assets directory
+            model_artifact_path = "../assets/models/model_svm1.pkl"
+            with open(model_artifact_path, 'rb') as model_file:
+                classifier = pickle.load(model_file)
+                
+            # Step 5: Algorithmic Inference
+            # The SVM determines the classification boundary for the semantic vector
+            print("Performing binary classification...")
+            prediction_bin = classifier.predict(semantic_features)
+            
+            # Step 6: Result Interpretation and User Communication
+            is_depressive = prediction_bin[0]
+            if is_depressive == 1:
+                print("\n>>> CLASSIFICATION RESULT: The analyzed content exhibits depressive characteristics.")
+            else:
+                print("\n>>> CLASSIFICATION RESULT: The analyzed content is classified as non-depressive.")
+
+        except FileNotFoundError:
+            print(f"Error: The input file {args.filename} could not be located.")
+        except Exception as e:
+            print(f"An error occurred during the inference process: {e}")
+
+    else:
+        print("Usage Error: Please provide an input file and specify 'SVM' as the target model.")
+
+if __name__ == '__main__':
+    main()
+
+

source_code/core/train.py

@@ -0,0 +1,80 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Script for training machine learning models for tweet analysis.
+# ==============================================================================
+
+import argparse
+import warnings
+import train_utilities as TU
+
+# Suppression of non-critical runtime warnings to ensure output clarity during training
+warnings.filterwarnings("ignore")
+
+def main():
+    """
+    Primary execution routine for the model training utility.
+    
+    This script facilitates the training of various machine learning 
+    architectures by providing a standardized interface for:
+        1. Dataset Ingestion: Loading and splitting training data.
+        2. Hyperparameter Configuration: Setting up model-specific parameters.
+        3. Algorithmic Training: Executing the training process via train_utilities.
+        4. Model Serialization: Persisting the resulting model for future inference.
+    """
+    # Initialize the CLI argument parser
+    parser = argparse.ArgumentParser(
+        description="Twitter Depression Detection: Model Training Utility"
+    )
+
+    # Positional argument for the training dataset path (CSV format)
+    parser.add_argument(
+        'filename', 
+        help="Path to the training dataset (TSV/CSV format with 'label' and 'clean_text')"
+    )
+
+    # Positional argument for the classification model architecture
+    # Supported: 'DT', 'LR', 'kNN', 'SVM', 'RF', 'NN', 'LSTM'
+    parser.add_argument(
+        'model', 
+        help="Target model architecture for training"
+    )
+
+    # Execution of the parsing logic
+    args = parser.parse_args()
+
+    # Deployment of the selected training pipeline based on the 'model' parameter
+    model_type = args.model
+    dataset_path = args.filename
+
+    # Pipeline selection logic
+    if model_type in ["DT", "LR", "kNN", "SVM", "RF", "NN"]:
+        # Logic for standardized Scikit-learn architectures
+        print(f"Initializing {model_type} training pipeline...")
+        
+        # Step 1: Data Acquisition and Validation Splitting
+        X_train, X_test, Y_train, Y_test = TU.load_prepare_split_df(dataset_path)
+
+        # Step 2: Algorithmic Training and Parameter Optimization
+        # The 'classification' method handles instantiation and fitting
+        trained_model = TU.classification(X_train=X_train, Y_train=Y_train, model=model_type)
+        
+        print(f"Training for {model_type} successful.")
+
+    elif model_type == "LSTM":
+        # Specialized logic for Long Short-Term Memory (LSTM) Neural Networks
+        # LSTMs are utilized here to capture long-range temporal dependencies in text
+        print("Initializing LSTM deep learning pipeline...")
+        TU.LSTM(dataset_path)
+        
+    else:
+        print(f"Error: Model architecture '{model_type}' is not currently recognized.")
+        print("Supported architectures: DT, LR, kNN, SVM, RF, NN, LSTM")
+
+if __name__ == '__main__':
+    main()

source_code/core/train_utilities.py

@@ -0,0 +1,226 @@
+# ==============================================================================
+# PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+# AUTHORS: AMEY THAKUR & MEGA SATISH
+# GITHUB (AMEY): https://github.com/Amey-Thakur
+# GITHUB (MEGA): https://github.com/msatmod
+# REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+# RELEASE DATE: June 5, 2022
+# LICENSE: MIT License
+# DESCRIPTION: Utility module for the model training pipeline.
+# ==============================================================================
+
+import pickle
+import warnings
+import numpy as np
+import pandas as pd
+import spacy
+import en_core_web_lg
+from sklearn.model_selection import train_test_split, KFold, cross_val_score, GridSearchCV
+from sklearn.metrics import confusion_matrix, accuracy_score
+from sklearn.linear_model import LogisticRegression
+from sklearn.neighbors import KNeighborsClassifier
+from sklearn.svm import SVC
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.neural_network import MLPClassifier
+from sklearn.ensemble import RandomForestClassifier
+
+# Suppression of non-critical runtime warnings to maintain algorithmic output integrity
+warnings.filterwarnings("ignore")
+
+def load_prepare_split_df(filename: str, targets=['label'], validation_size=0.3, seed=7):
+    """
+    Ingests raw data, performs feature extraction via word embeddings, 
+    and partitions the dataset for model validation.
+    
+    Methodology:
+        - TSV Ingestion: Data is loaded from the specified file.
+        - Semantic Vectorization: Utilizing spaCy's dense 300-dimensional 
+          word embeddings (centroid of token vectors).
+        - Validation Partitioning: Stratified splitting of data into 
+          training and testing subsets.
+
+    Args:
+        filename (str): Path to the TSV/CSV dataset.
+        targets (list): Column name for the dependent variable.
+        validation_size (float): Proportion of data reserved for testing.
+        seed (int): Random seed for reproducibility.
+
+    Returns:
+        tuple: (X_train, X_test, Y_train, Y_test) feature and label sets.
+    """
+    print(f"Acquiring dataset from: {filename}")
+    df_all = pd.read_csv(filename, sep='\t', encoding='utf-8')
+
+    # Step 1: Initialize the Linguistic Engine
+    nlp_engine = en_core_web_lg.load()
+
+    # Step 2: Compute Dense Word Embeddings (Feature Extraction)
+    print("Extracting semantic features via spaCy embeddings...")
+    feature_vectors = np.array([
+        np.array([token.vector for token in nlp_engine(s)]).mean(axis=0) * np.ones((300))
+        for s in df_all['clean_text']
+    ])
+    
+    # Step 3: Dataset Splitting
+    y_labels = df_all.loc[:, targets]
+    x_features = feature_vectors
+
+    x_train, x_test, y_train, y_test = train_test_split(
+        x_features, y_labels, test_size=validation_size, random_state=seed
+    )
+
+    return x_train, x_test, y_train, y_test
+
+def classification(X_train, Y_train, model=""):
+    """
+    Facilitates the training and serialization of various classification 
+    architectures.
+    
+    Architectures Supported:
+        - SVM: Support Vector Machine (Selected as the production primary).
+        - LR: Logistic Regression.
+        - DT: Decision Tree Classifier.
+        - KNN: k-Nearest Neighbors (with automated k-optimization).
+        - RF: Random Forest Classifier.
+        - NN: Multi-layer Perceptron (MLP) Neural Network.
+
+    Args:
+        X_train: Training feature set.
+        Y_train: Training label set.
+        model (str): Target architecture identifier.
+
+    Returns:
+        object: The trained Scikit-learn model instance.
+    """
+    if model == "SVM":
+        # Support Vector Machines are effective in high-dimensional semantic spaces
+        print("Initializing SVM (Support Vector Machine) training...")
+        clf = SVC(probability=True)
+        clf.fit(X_train, Y_train)
+        
+        # Performance Evaluation (Accuracy Metric)
+        train_accuracy = accuracy_score(clf.predict(X_train), Y_train)
+        print(f"Training Convergence Accuracy: {train_accuracy:.4f}")
+
+        # Persistence: Serializing the model artifact
+        save_path = "../assets/models/model_svm_pc.pkl"
+        with open(save_path, 'wb') as file:
+            pickle.dump(clf, file)
+        return clf
+
+    elif model == "LR":
+        # Logistic Regression serves as a robust baseline for linear classification
+        print("Initializing Logistic Regression training...")
+        lr_model = LogisticRegression()
+        lr_model.fit(X_train, Y_train)
+        
+        save_path = "../assets/models/model_LogReg.pkl"
+        with open(save_path, 'wb') as file:
+            pickle.dump(lr_model, file)
+        return lr_model
+
+    elif model == "DT":
+        # Decision Trees provide hierarchical decision boundaries
+        print("Initializing Decision Tree training...")
+        dt_model = DecisionTreeClassifier()
+        dt_model.fit(X_train, Y_train)
+        
+        save_path = "../assets/models/model_DTC.pkl"
+        with open(save_path, 'wb') as file:
+            pickle.dump(dt_model, file)
+        return dt_model
+
+    elif model == "KNN":
+        # kNN requires hyperparameter tuning (k value) via cross-validation
+        print("Initializing kNN training with automated k-optimization...")
+        k_values = range(1, 32, 1)
+        k_scores = []
+
+        # 10-Fold Cross-Validation for optimal k-neighbor selection
+        for k in k_values:
+            knn = KNeighborsClassifier(n_neighbors=k)
+            score = np.mean(cross_val_score(knn, X_train, Y_train, cv=10))
+            k_scores.append(score)
+            
+        optimal_k = k_values[np.argmax(k_scores)]
+        print(f"Optimized Hyperparameter discovered: k = {optimal_k}")
+
+        best_knn = KNeighborsClassifier(n_neighbors=optimal_k)
+        best_knn.fit(X_train, Y_train)
+
+        save_path = "../assets/models/model_KNN.pkl"
+        with open(save_path, 'wb') as file:
+            pickle.dump(best_knn, file)
+        return best_knn
+
+    elif model == "RF":
+        # Random Forest: Ensemble bagged decision trees for variance reduction
+        print("Initializing Random Forest training...")
+        rf_model = RandomForestClassifier()
+        rf_model.fit(X_train, Y_train)
+        
+        save_path = "../assets/models/model_RF.pkl"
+        with open(save_path, 'wb') as file:
+            pickle.dump(rf_model, file)
+        return rf_model
+
+    elif model == "NN":
+        # MLP (Multi-layer Perceptron): Basic artificial neural network
+        print("Initializing Neural Network (MLP) training...")
+        nn_model = MLPClassifier()
+        nn_model.fit(X_train, Y_train)
+        
+        save_path = "../assets/models/model_NN.pkl"
+        with open(save_path, 'wb') as file:
+            pickle.dump(nn_model, file)
+        return nn_model
+
+def LSTM(filename: str):
+    """
+    Executes a Deep Learning pipeline using Long Short-Term Memory (LSTM) 
+    recurrent neural networks for capturing temporal lingustical patterns.
+    
+    Methodology:
+        - Tokenization: Integer encoding of sequences.
+        - Padding: Uniform sequence length normalization.
+        - Architecture: Embedding layer followed by LSTM with Dropouts.
+    """
+    from keras.models import Sequential
+    from keras.layers import Dense, Embedding, LSTM
+    from keras.preprocessing.text import Tokenizer
+    from keras.preprocessing.sequence import pad_sequences
+    from keras.wrappers.scikit_learn import KerasClassifier
+
+    print(f"Acquiring data for Deep Learning (LSTM): {filename}")
+    df_dl = pd.read_csv(filename, sep='\t', encoding='utf-8')
+
+    # Step 1: Sequence Tokenization and Padding
+    vocab_size = 20000
+    max_len = 50
+    tokenizer = Tokenizer(num_words=vocab_size)
+    tokenizer.fit_on_texts(df_dl['clean_text'])
+    seqs = tokenizer.texts_to_sequences(df_dl['clean_text'])
+    x_lstm = pad_sequences(seqs, maxlen=max_len)
+    y_lstm = df_dl["label"]
+
+    # Step 2: Architecture Definition
+    print("Constructing LSTM topology...")
+    model = Sequential()
+    model.add(Embedding(vocab_size, 300, input_length=max_len))
+    model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
+    model.add(Dense(1, activation='sigmoid'))
+    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
+
+    # Step 3: Model Execution and Persistance
+    print("Commencing Deep Learning Convergence (LSTM)...")
+    # In a professional context, create_model should be passed to KerasClassifier
+    # Here we demonstrate the fundamental fit operation
+    model.fit(x_lstm, y_lstm, epochs=3, verbose=1, validation_split=0.3)
+
+    # Persistence: JSON topology and H5 weights
+    model_json = model.to_json()
+    with open("model_LSTM.json", "w") as json_file:
+        json_file.write(model_json)
+    model.save_weights("model_LSTM.h5")
+    print("Deep Learning model (LSTM) artifacts successfully persisted.")
+

source_code/notebooks/data_cleaning_exploration.py

@@ -0,0 +1,505 @@
+# -*- coding: utf-8 -*-
+"""data_cleaning_exploration.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1jU6I53BYSZ2kX-qcxcWP_1bPNYWvB24f
+
+# Data Cleaning
+
+Data cleaning is the process of detecting and removing errors and inconsistencies from the data to improve its quality. Improper data cleaning process can lead to errors, faulty analysis, distortion in dataset and eventually incompatible datasets for machine learning purposes. There is no absolute way to prescribe the exact steps in the data cleaning process because the processes will vary from dataset to dataset. My data cleaning process includes:
+
+* Check the data types
+* Check for duplicates - Primary key ('tweets.id')
+* Check missing values
+* Make text all lower case
+* Remove links and images
+* Remove hashtags
+* Remove @ mentions
+* Remove emojis
+* Remove stop words
+* Remove punctuation
+* Get rid of stuff like "what's" and making it "what is'
+* Stemming / lemmatization
+"""
+
+from google.colab import drive
+drive.mount('/content/drive')
+
+!pip install -qqq ftfy
+
+## Import required libraries
+
+## warnings
+import warnings
+warnings.filterwarnings("ignore")
+
+## for data
+import numpy as np
+import pandas as pd
+
+## for plotting
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+## for processing
+import nltk
+import re
+import ftfy
+from nltk.stem import WordNetLemmatizer 
+from nltk.corpus import stopwords
+nltk.download('stopwords')
+nltk.download('punkt')
+nltk.download('wordnet')
+nltk.download('averaged_perceptron_tagger')
+
+## for opening, manipulating, and saving many different image file f
+from PIL import Image 
+
+## WordCloud - Python linrary for creating image wordclouds
+from wordcloud import WordCloud
+from nltk import pos_tag ## For Parts of Speech tagging
+import random ## generating random numbers
+
+"""## Load the datasets"""
+
+depressive_tweets_df = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/output/depressive_tweets.csv")
+random_tweets_df = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Depression_tweets_Data/Data1/Sentiment Analysis Dataset 2.csv", 
+                               encoding = "ISO-8859-1", usecols = range(0,4), nrows = 40000)
+
+depressive_tweets_df
+
+random_tweets_df
+
+## Slicing the random tweets to have sentiment == 1 
+new_rand_df = random_tweets_df[random_tweets_df.Sentiment == 1]
+new_rand_df.reset_index(inplace=True)
+
+new_rand_df.shape
+
+new_rand_df.head()
+
+"""20952 random tweets with sentiment == 1
+
+## Data Cleaning-Processing:
+"""
+
+print(depressive_tweets_df.shape)
+print(new_rand_df.shape)
+
+## Check the data type of each column
+depressive_tweets_df.dtypes.to_frame().rename(columns={0:'data_type'})
+
+## Check the data type of each column
+new_rand_df.dtypes.to_frame().rename(columns={0:'data_type'})
+
+## Drop unnecessary columns
+depressive_tweets_df.drop(columns=['Unnamed: 0'], inplace=True)
+new_rand_df.drop(columns=['ItemID', 'index','Sentiment', 'SentimentSource'], inplace=True)
+
+"""Since we are mostly dealing with the tweets in our datasets, it is not necessary to change the data types at this stage."""
+
+## Finding unique values in each column
+for col in depressive_tweets_df:
+    print("There are ", len(depressive_tweets_df[col].unique()), "unique values in ", col)
+
+"""By considering **tweet.id** as our primary key, we have **18190** unique tweets, so we need to get rid of the duplicates."""
+
+## Finding unique values in each column
+for col in new_rand_df:
+    print("There are ", len(new_rand_df[col].unique()), "unique values in ", col)
+
+"""No duplicates in random tweets dataset"""
+
+## drop duplicate values in tweet.id
+depressive_tweets_df.drop_duplicates(subset=['tweet.id'], inplace=True)
+
+depressive_tweets_df.reset_index(inplace=True)
+
+depressive_tweets_df.shape
+
+## Find the number of Null values in each columns
+depressive_tweets_df.isnull().sum().to_frame().rename(columns={0:'Null values'})
+
+"""There are **6384** Null values in the **location** columns but since location will not be used in our analysis or as a feature in our model, we don't need to replace them."""
+
+## Find the number of Null values in each columns
+new_rand_df.isnull().sum().to_frame().rename(columns={0:'Null values'})
+
+"""No Null values in random tweets dataset."""
+
+## Drop all the columns except index, tweet.id and text
+new_dep_df = depressive_tweets_df[['text']]
+
+## Add label to both datasets (0 is non-depressive and 1 is depressive)
+new_dep_df['label'] = pd.Series([1 for x in range(len(new_dep_df.index))])
+new_rand_df['label'] = pd.Series([0 for x in range(len(new_rand_df.index))])
+
+new_dep_df
+
+## Change the column name to be aligned with depressive dataset
+new_rand_df.rename(columns={'SentimentText': 'text'}, inplace=True)
+
+new_rand_df
+
+## Combine two dataframes together
+df_all = pd.concat([new_dep_df, new_rand_df], ignore_index=True)
+
+df_all
+
+# Expand Contraction
+cList = {
+  "ain't": "am not",
+  "aren't": "are not",
+  "can't": "cannot",
+  "can't've": "cannot have",
+  "'cause": "because",
+  "could've": "could have",
+  "couldn't": "could not",
+  "couldn't've": "could not have",
+  "didn't": "did not",
+  "doesn't": "does not",
+  "don't": "do not",
+  "hadn't": "had not",
+  "hadn't've": "had not have",
+  "hasn't": "has not",
+  "haven't": "have not",
+  "he'd": "he would",
+  "he'd've": "he would have",
+  "he'll": "he will",
+  "he'll've": "he will have",
+  "he's": "he is",
+  "how'd": "how did",
+  "how'd'y": "how do you",
+  "how'll": "how will",
+  "how's": "how is",
+  "I'd": "I would",
+  "I'd've": "I would have",
+  "I'll": "I will",
+  "I'll've": "I will have",
+  "I'm": "I am",
+  "I've": "I have",
+  "isn't": "is not",
+  "it'd": "it had",
+  "it'd've": "it would have",
+  "it'll": "it will",
+  "it'll've": "it will have",
+  "it's": "it is",
+  "let's": "let us",
+  "ma'am": "madam",
+  "mayn't": "may not",
+  "might've": "might have",
+  "mightn't": "might not",
+  "mightn't've": "might not have",
+  "must've": "must have",
+  "mustn't": "must not",
+  "mustn't've": "must not have",
+  "needn't": "need not",
+  "needn't've": "need not have",
+  "o'clock": "of the clock",
+  "oughtn't": "ought not",
+  "oughtn't've": "ought not have",
+  "shan't": "shall not",
+  "sha'n't": "shall not",
+  "shan't've": "shall not have",
+  "she'd": "she would",
+  "she'd've": "she would have",
+  "she'll": "she will",
+  "she'll've": "she will have",
+  "she's": "she is",
+  "should've": "should have",
+  "shouldn't": "should not",
+  "shouldn't've": "should not have",
+  "so've": "so have",
+  "so's": "so is",
+  "that'd": "that would",
+  "that'd've": "that would have",
+  "that's": "that is",
+  "there'd": "there had",
+  "there'd've": "there would have",
+  "there's": "there is",
+  "they'd": "they would",
+  "they'd've": "they would have",
+  "they'll": "they will",
+  "they'll've": "they will have",
+  "they're": "they are",
+  "they've": "they have",
+  "to've": "to have",
+  "wasn't": "was not",
+  "we'd": "we had",
+  "we'd've": "we would have",
+  "we'll": "we will",
+  "we'll've": "we will have",
+  "we're": "we are",
+  "we've": "we have",
+  "weren't": "were not",
+  "what'll": "what will",
+  "what'll've": "what will have",
+  "what're": "what are",
+  "what's": "what is",
+  "what've": "what have",
+  "when's": "when is",
+  "when've": "when have",
+  "where'd": "where did",
+  "where's": "where is",
+  "where've": "where have",
+  "who'll": "who will",
+  "who'll've": "who will have",
+  "who's": "who is",
+  "who've": "who have",
+  "why's": "why is",
+  "why've": "why have",
+  "will've": "will have",
+  "won't": "will not",
+  "won't've": "will not have",
+  "would've": "would have",
+  "wouldn't": "would not",
+  "wouldn't've": "would not have",
+  "y'all": "you all",
+  "y'alls": "you alls",
+  "y'all'd": "you all would",
+  "y'all'd've": "you all would have",
+  "y'all're": "you all are",
+  "y'all've": "you all have",
+  "you'd": "you had",
+  "you'd've": "you would have",
+  "you'll": "you you will",
+  "you'll've": "you you will have",
+  "you're": "you are",
+  "you've": "you have"
+}
+
+c_re = re.compile('(%s)' % '|'.join(cList.keys()))
+
+def expandContractions(text, c_re=c_re):
+    def replace(match):
+        return cList[match.group(0)]
+    return c_re.sub(replace, text)
+
+## Function to perform stepwise cleaning process
+def tweets_cleaner(tweets):
+  cleaned_tweets = []
+  for tweet in tweets:
+    tweet = tweet.lower() #lowercase
+    
+    # if url links then don't append to avoid news articles
+    # also check tweet length, save those > 5 
+    if re.match("(\w+:\/\/\S+)", tweet) == None and len(tweet) > 5:
+    
+      #remove hashtag, @mention, emoji and image URLs
+      tweet = ' '.join(re.sub("(@[A-Za-z0-9]+)|(\#[A-Za-z0-9]+)|(<Emoji:.*>)|(pic\.twitter\.com\/.*)", " ", tweet).split())
+
+      #fix weirdly encoded texts
+      tweet = ftfy.fix_text(tweet)
+
+      #expand contraction
+      tweet = expandContractions(tweet)
+
+
+      #remove punctuation
+      tweet = ' '.join(re.sub("([^0-9A-Za-z \t])", " ", tweet).split())
+
+      #stop words and lemmatization
+      stop_words = set(stopwords.words('english'))
+      word_tokens = nltk.word_tokenize(tweet)
+
+      lemmatizer=WordNetLemmatizer()
+      filtered_sentence = [lemmatizer.lemmatize(word) for word in word_tokens if not word in stop_words]
+      # back to string from list
+      tweet = ' '.join(filtered_sentence) # join words with a space in between them
+
+      cleaned_tweets.append(tweet)
+
+  return cleaned_tweets
+
+"""## Word Cloud:
+
+To get the most common words used in depressive and random datasets, the POS-tag (Parts of Speech tagging) module in the NLTK library was used. Using the WordCloud library, one can generate a Word Cloud based on word frequency and superimpose these words on any image. In this case, I used the Twitter logo and Matplotlib to display the image. The Word Cloud shows the words with higher frequency in bigger text size while the "not-so" common words are in smaller text sizes.
+"""
+
+depressive_tweets_arr = [x for x in new_dep_df['text']]
+random_tweets_arr = [x for x in new_rand_df['text']]
+X_d = tweets_cleaner(depressive_tweets_arr)
+X_r = tweets_cleaner(random_tweets_arr)
+
+## function to obtain adjectives from tweets
+def getadjectives(tweet):
+    tweet = nltk.word_tokenize(tweet)  # convert string to tokens
+    tweet = [word for (word, tag) in pos_tag(tweet)
+             if tag == "JJ"]  # pos_tag module in NLTK library
+    return " ".join(tweet)  # join words with a space in between them
+
+"""### Depressive Tweets Exploration"""
+
+## Apply getadjectives function to the processed tweets 
+## Extract all tweets into one long string with each word separate with a "space"
+tweets_long_string = [getadjectives(x) for x in X_d]
+tweets_long_string = " ".join(tweets_long_string)
+
+# Import Twitter Logo
+image = np.array(Image.open('/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/logo.jpeg'))
+    
+fig = plt.figure() # Instantiate the figure object
+fig.set_figwidth(14) # set width
+fig.set_figheight(18) # set height
+
+plt.imshow(image, cmap=plt.cm.gray, interpolation='bilinear') # Display data as an image
+plt.axis('off') # Remove axis
+plt.show() # Display image
+
+## Create function to generate the blue colour for the Word CLoud
+
+def blue_color_func(word, font_size, position, orientation, random_state=None,**kwargs):
+    return "hsl(210, 100%%, %d%%)" % random.randint(50, 70)
+
+## Instantiate the Twitter word cloud object
+twitter_wc = WordCloud(background_color='white', max_words=1500, mask=image)
+
+## generate the word cloud
+twitter_wc.generate(tweets_long_string)
+
+## display the word cloud
+fig = plt.figure()
+fig.set_figwidth(14)  # set width
+fig.set_figheight(18)  # set height
+
+plt.imshow(twitter_wc.recolor(color_func=blue_color_func, random_state=3),
+           interpolation="bilinear")
+plt.axis('off')
+plt.show()
+
+twitter_wc.to_file("/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/wordcloud.png") #save to a png file
+
+"""**Analyzing Top Words in the Word Cloud for depressive dataset**"""
+
+## Combine all words in depressive into a list
+tweets_long_string = [getadjectives(x) for x in X_d]
+tweets_list=[]
+for item in tweets_long_string:
+    item = item.split()
+    for i in item:
+        tweets_list.append(i)
+
+# Use the Built-in Python Collections module to determine Word frequency
+from collections import Counter
+counts = Counter(tweets_list)
+df = pd.DataFrame.from_dict(counts, orient='index').reset_index()
+df.columns = ['Words', 'Count']
+df.sort_values(by='Count', ascending=False, inplace=True)
+
+df.head(10)  # Check dataframe first 10 rows
+
+"""### Random Tweets Exploration
+
+"""
+
+## Apply getadjectives function to the processed tweets 
+## Extract all tweets into one long string with each word separate with a "space"
+tweets_long_string_rand = [getadjectives(x) for x in X_r]
+tweets_long_string_rand = " ".join(tweets_long_string_rand)
+
+# Import Twitter Logo
+image = np.array(Image.open('/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/logo.jpeg'))
+    
+fig = plt.figure() # Instantiate the figure object
+fig.set_figwidth(14) # set width
+fig.set_figheight(18) # set height
+
+plt.imshow(image, cmap=plt.cm.gray, interpolation='bilinear') # Display data as an image
+plt.axis('off') # Remove axis
+plt.show() # Display image
+
+## Create function to generate the blue colour for the Word CLoud
+
+def blue_color_func(word, font_size, position, orientation, random_state=None,**kwargs):
+    return "hsl(210, 100%%, %d%%)" % random.randint(50, 70)
+
+## Instantiate the Twitter word cloud object
+twitter_wc = WordCloud(background_color='white', max_words=1500, mask=image)
+
+## generate the word cloud
+twitter_wc.generate(tweets_long_string_rand)
+
+## display the word cloud
+fig = plt.figure()
+fig.set_figwidth(14)  # set width
+fig.set_figheight(18)  # set height
+
+plt.imshow(twitter_wc.recolor(color_func=blue_color_func, random_state=3),
+           interpolation="bilinear")
+plt.axis('off')
+plt.show()
+
+twitter_wc.to_file("/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/wordcloud_rand.png") #save to a png file
+
+"""**Analyzing Top Words in the Word Cloud for random dataset**"""
+
+## Combine all words in depressive into a list
+tweets_long_string_rand = [getadjectives(x) for x in X_r]
+tweets_list_rand=[]
+for item in tweets_long_string_rand:
+    item = item.split()
+    for i in item:
+        tweets_list_rand.append(i)
+
+## Use the Built-in Python Collections module to determine Word frequency
+from collections import Counter
+counts = Counter(tweets_list_rand)
+df_rand = pd.DataFrame.from_dict(counts, orient='index').reset_index()
+df_rand.columns = ['Words', 'Count']
+df_rand.sort_values(by='Count', ascending=False, inplace=True)
+
+df_rand.head(10)  # Check dataframe first 10 rows
+
+"""## Data Analysis:"""
+
+## distribution of classes for prediction
+def create_distribution(dataFile):
+  return sns.countplot(x='label', data=dataFile, palette='hls')
+
+create_distribution(df_all)
+
+"""Depreesive and random (Non-depressive) tweets are almost evenly distributed.
+
+**Finding distribution of tweet lengths**
+"""
+
+dep_line_lengths = [len(statement) for statement in new_dep_df['text']]
+plt.plot(dep_line_lengths)
+plt.show()
+
+rand_line_lengths = [len(statement) for statement in new_rand_df['text']]
+plt.plot(dep_line_lengths)
+plt.show()
+
+"""From the distributions above, it is clear that there is no outliers in our depressive and random datasets.
+
+## Cleaning combined dataset and save it
+"""
+
+tweets_arr = [x for x in df_all['text']]
+
+corpus = tweets_cleaner(tweets_arr)
+
+corpus[:10]
+
+## Adding clean tweets as a new column
+df_all['clean_text'] = corpus
+
+"""We have to remove those rows with tweets that has been completely deleted in the cleaning process."""
+
+# replace field that's entirely space (or empty) with NaN
+df_all.replace(r'^\s*$', np.nan, regex=True, inplace=True)
+
+df_all[df_all['clean_text'].isnull()]
+
+## Deleting the rows with nan 
+df_all.dropna(subset=['clean_text'], inplace=True)
+
+## Double_check for nan
+df_all[df_all['clean_text'].isnull()]
+
+## Save cleaned_dataset
+df_all.to_csv('/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/processed_data/processed_data.csv',
+              sep='\t', encoding='utf-8',index=False)

source_code/notebooks/data_gathering_twint.py

@@ -0,0 +1,80 @@
+# -*- coding: utf-8 -*-
+"""Data_Gathering_Twint.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1zV6s2FhvQCmyNh0uyknfm47WATAOihbU
+"""
+
+from google.colab import drive
+drive.mount('/content/drive')
+
+!git clone https://github.com/twintproject/twint.git
+
+import os
+os.chdir("/content/twint")
+
+!pip freeze > requirements.txt
+
+!pip install .
+
+!pip install -U git+https://github.com/cyxv/twint.git@master
+
+!pip install nest_asyncio
+
+!pip3 install twint
+
+# Import required libraries
+import nest_asyncio
+nest_asyncio.apply()
+import pandas as pd
+import twint
+import pandas as pd
+import re
+
+# add some tweets with depressed and depression tags, for a particular year
+
+depress_tags = ["#depressed", "#anxiety", "#depression", "#suicide", "#mentalhealth"
+                "#loneliness", "#hopelessness", "#itsokaynottobeokay"]
+
+content = {}
+for i in range(len(depress_tags)):
+    print(depress_tags[i])
+    c = twint.Config()
+    
+    c.Format = "Tweet id: {id} | Tweet: {tweet}"
+    c.Search = depress_tags[i]
+    c.Limit = 1000
+    c.Year = 2019
+    c.Lang = "en"
+    c.Store_csv = True
+    c.Store_Object = True
+    c.Output = "/content/drive/MyDrive/NLP/Depression_Detection/depressive_en_2019.csv"
+    c.Hide_output = True
+    c.Stats = True
+    c.Lowercase  = True
+    c.Filter_retweets = True
+    twint.run.Search(c)
+
+# add some tweets with depressed and depression tags, for a particular year
+
+depress_tags = ["#depressed", "#depression", "#loneliness", "#hopelessness"]
+
+content = {}
+for i in range(len(depress_tags)):
+    print(depress_tags[i])
+    c = twint.Config()
+    
+    c.Format = "Tweet id: {id} | Tweet: {tweet}"
+    c.Search = depress_tags[i]
+    c.Limit = 1000
+    c.Year = 2020
+    c.Store_csv = True
+    c.Store_json = True
+    c.Output = "/content/drive/MyDrive/NLP/Depression_Detection/dataset_depression.json"
+    c.Hide_output = True
+    c.Stats = True
+    c.Lowercase  = True
+    c.Filter_retweets = True
+    twint.run.Search(c)

source_code/notebooks/data_gathering_twitter_API.py

@@ -0,0 +1,388 @@
+# -*- coding: utf-8 -*-
+"""Twitter_API.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1UAilj_PmxYbwHsc_s79d9UyBvawBVZAS
+
+# Tweet mining using Twitter API via Tweepy:
+
+In this notebook I am using Tweepy python library to  tweets using relevant hashtags. I was able to retrieve around 19000 unique tweets via twitter API. At the end, all the datasets with different depressive hashtags will be combined, cleaned and saved as depressive_tweets.csv.
+"""
+
+from google.colab import drive
+drive.mount('/content/drive')
+
+"""## Tweets mining"""
+
+!pip install -qqq tweepy
+
+## Import required libraries
+import tweepy
+from tweepy.streaming import StreamListener
+from tweepy import OAuthHandler
+from tweepy import Stream
+import csv
+import pandas as pd
+
+## Access to twitter API cunsumer_key and access_secret
+#import config.ipynb
+
+## Twitter API related information
+consumer_key = config.API_KEY
+consumer_secret = config.API_KEY_SECRET
+access_key= config.ACCESS_TOKEN
+access_secret = config.ACCESS_TOKEN_SECRET
+
+auth = tweepy.OAuthHandler(consumer_key, consumer_secret) # Pass in Consumer key and secret for authentication by API
+auth.set_access_token(access_key, access_secret) # Pass in Access key and secret for authentication by API
+api = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True) # Sleeps when API limit is reached
+
+## depress_tags = ["#depressed", "#anxiety", "#depression", "#suicide", "#mentalhealth"
+##                "#loneliness", "#hopelessness", "#itsokaynottobeokay", "#sad"]
+
+"""## "#depressed""""
+
+## Create a function for tweets mining
+def tweets_mining1(search_query1, num_tweets1, since_id_num1):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list1 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query1, lang="en", since_id=since_id_num1, 
+                                                    tweet_mode='extended').items(num_tweets1)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list1[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_depressed_1.csv','a', newline='', encoding='utf-8') as csvFile1:
+      csv_writer1 = csv.writer(csvFile1, delimiter=',') # create an instance of csv object
+      csv_writer1.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words1 = "#depressed" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query1 = search_words1 + " -filter:links AND -filter:retweets AND -filter:replies" 
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_depressed_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining1(search_query1, 1000, latest_tweet)
+
+df_depressed_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_depressed_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_depressed_1
+
+## Finding unique values in each column
+for col in df_depressed_1:
+    print("There are ", len(df_depressed_1[col].unique()), "unique values in ", col)
+
+"""### Anxiety and suicide """
+
+## Create a function for tweets mining
+def tweets_mining2(search_query2, num_tweets2, since_id_num2):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list2 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query2, lang="en", since_id=since_id_num2, 
+                                                    tweet_mode='extended').items(num_tweets2)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list2[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_anxiety_1.csv','a', newline='', encoding='utf-8') as csvFile2:
+      csv_writer2 = csv.writer(csvFile2, delimiter=',') # create an instance of csv object
+      csv_writer2.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words2 = "#anxiety" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query2 = search_words2 + " -filter:links AND -filter:retweets AND -filter:replies"
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_anxiety_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining2(search_query2, 2000, latest_tweet)
+
+df_anxiety_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_anxiety_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_anxiety_1
+
+## Finding unique values in each column
+for col in df_anxiety_1:
+    print("There are ", len(df_anxiety_1[col].unique()), "unique values in ", col)
+
+"""## "#Suicide""""
+
+## Create a function for tweets mining
+def tweets_mining3(search_query3, num_tweets3, since_id_num3):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list3 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query3, lang="en", since_id=since_id_num3, 
+                                                    tweet_mode='extended').items(num_tweets3)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list3[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_suicide_1.csv','a', newline='', encoding='utf-8') as csvFile3:
+      csv_writer3 = csv.writer(csvFile3, delimiter=',') # create an instance of csv object
+      csv_writer3.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words3 = "#suicide" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query3 = search_words3 + " -filter:links AND -filter:retweets AND -filter:replies" 
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_suicide_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining3(search_query3, 10000, latest_tweet)
+
+df_suicide_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_suicide_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_suicide_1
+
+"""## "#hopelessness""""
+
+## Create a function for tweets mining
+def tweets_mining4(search_query4, num_tweets4, since_id_num4):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list4 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query4, lang="en", since_id=since_id_num4, 
+                                                    tweet_mode='extended').items(num_tweets4)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list4[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_hopeless_1.csv','a', newline='', encoding='utf-8') as csvFile4:
+      csv_writer4 = csv.writer(csvFile4, delimiter=',') # create an instance of csv object
+      csv_writer4.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words4 = "#hopelessness" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query4 = search_words4 + " -filter:links AND -filter:retweets AND -filter:replies"
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_hopeless_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining4(search_query4, 10000, latest_tweet)
+
+df_hopeless_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_hopeless_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_hopeless_1
+
+"""## "#mentalhealth""""
+
+## Create a function for tweets mining
+def tweets_mining5(search_query5, num_tweets5, since_id_num5):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list5 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query5, lang="en", since_id=since_id_num5, 
+                                                    tweet_mode='extended').items(num_tweets5)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list5[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_mentalhealth_1.csv','a', newline='', encoding='utf-8') as csvFile5:
+      csv_writer5 = csv.writer(csvFile5, delimiter=',') # create an instance of csv object
+      csv_writer5.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words5 = "#mentalhealth" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query5 = search_words5 + " -filter:links AND -filter:retweets AND -filter:replies" 
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_mentalhealth_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0])
+tweets_mining5(search_query5, 1000, latest_tweet)
+
+df_mentalhealth_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_mentalhealth_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_mentalhealth_1
+
+"""## "#loneliness""""
+
+## Create a function for tweets mining
+def tweets_mining6(search_query6, num_tweets6, since_id_num6):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list6 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query6, lang="en", since_id=since_id_num6, 
+                                                    tweet_mode='extended').items(num_tweets6)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list6[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_loneliness_1.csv','a', newline='', encoding='utf-8') as csvFile6:
+      csv_writer6 = csv.writer(csvFile6, delimiter=',') # create an instance of csv object
+      csv_writer6.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words6 = "#loneliness" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query6 = search_words6 + " -filter:links AND -filter:retweets AND -filter:replies" 
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_loneliness_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0])
+tweets_mining6(search_query6, 10000, latest_tweet)
+
+df_loneliness_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_loneliness_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_loneliness_1
+
+"""## "#itsokaynottobeokay""""
+
+## Create a function for tweets mining
+def tweets_mining7(search_query7, num_tweets7, since_id_num7):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list7 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query7, lang="en", since_id=since_id_num7, 
+                                                    tweet_mode='extended').items(num_tweets7)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list7[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_itsoknottobeok_1 copy.csv','a', newline='', encoding='utf-8') as csvFile7:
+      csv_writer7 = csv.writer(csvFile7, delimiter=',') # create an instance of csv object
+      csv_writer7.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words7 = "#itsokaynottobeokay" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query7 = search_words7 + " -filter:links AND -filter:retweets AND -filter:replies"
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_itsoknottobeok_1 copy.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining7(search_query7, 2000, latest_tweet)
+
+df_itsok_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_itsoknottobeok_1 copy.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_itsok_1
+
+"""## "#depression""""
+
+## Create a function for tweets mining
+def tweets_mining8(search_query8, num_tweets8, since_id_num8):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list8 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query8, lang="en", since_id=since_id_num8, 
+                                                    tweet_mode='extended').items(num_tweets8)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list8[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_depression_1.csv','a', newline='', encoding='utf-8') as csvFile8:
+      csv_writer8 = csv.writer(csvFile8, delimiter=',') # create an instance of csv object
+      csv_writer8.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words8 = "#depression" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query8 = search_words8 + " -filter:links AND -filter:retweets AND -filter:replies"
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_depression_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining8(search_query8, 1000, latest_tweet)
+
+df_depression_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_depression_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_depression_1
+
+## Finding unique values in each column
+for col in df_depression_1:
+    print("There are ", len(df_depression_1[col].unique()), "unique values in ", col)
+
+"""## "#sad""""
+
+## Create a function for tweets mining
+def tweets_mining9(search_query9, num_tweets9, since_id_num9):
+  # Collect tweets using the Cursor object
+  # Each item in the iterator has various attributes that you can access to get information about each tweet
+  tweet_list9 = [tweets for tweets in tweepy.Cursor(api.search, q=search_query9, lang="en", since_id=since_id_num9, 
+                                                    tweet_mode='extended').items(num_tweets9)]
+  
+  # Begin scraping the tweets individually:
+  for tweet in tweet_list9[::-1]:
+    tweet_id = tweet.id # get Tweet ID result
+    created_at = tweet.created_at # get time tweet was created
+    text = tweet.full_text # retrieve full tweet text
+    location = tweet.user.location # retrieve user location
+    retweet = tweet.retweet_count # retrieve number of retweets
+    favorite = tweet.favorite_count # retrieve number of likes
+    with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_sad_1.csv','a', newline='', encoding='utf-8') as csvFile9:
+      csv_writer9 = csv.writer(csvFile9, delimiter=',') # create an instance of csv object
+      csv_writer9.writerow([tweet_id, created_at, text, location, retweet, favorite]) # write each row
+
+search_words9 = "#sad" # Specifying exact phrase to search
+# Exclude Links, retweets, replies
+search_query9 = search_words9 + " -filter:links AND -filter:retweets AND -filter:replies" 
+with open('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_sad_1.csv', encoding='utf-8') as data:
+    latest_tweet = int(list(csv.reader(data))[-1][0]) 
+tweets_mining9(search_query9, 2000, latest_tweet)
+
+df_sad_1 = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/tweets_sad_1.csv",
+                 names=['tweet.id', "created_at","text", "location", "retweet", "favorite"])
+
+df_sad_1
+
+"""# Combining all the tweets"""
+
+import glob
+
+path = r'/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API'  # use your path
+all_files = glob.glob(path + "/*.csv")
+
+tweets = []
+
+for filename in all_files:
+    df = pd.read_csv(filename, 
+                     names=['tweet.id', "created_at","text", "location", "retweet", "favorite"]) # Convert each csv to a dataframe
+    tweets.append(df)
+
+tweets_df = pd.concat(tweets, ignore_index=True) # Merge all dataframes
+#tweets_df.columns=['tweet.id', "created_at","text", "location", "retweet", "favorite"]
+tweets_df.head()
+
+tweets_df
+
+tweets_df.to_csv('/content/drive/MyDrive/NLP/Depression_Detection/Data_fetch_API/output/depressive_tweets.csv')
+
+"""## Data cleaning
+
+Data cleaning is one of the essential steps because without a proper cleaning procedure you will have errors in your analysis and eventually your data-driven results. Here I try to eliminate duplicates tweets by using the Primary key ('tweets.id'), checked for empty rows and replaced “NaN” if there is any.
+"""
+
+tweets_df.shape #Get number of rows and columns
+
+## Check the data type of each column
+tweets_df.dtypes.to_frame().rename(columns={0:'data_type'})
+
+## Finding unique values in each column
+for col in tweets_df:
+    print("There are ", len(tweets_df[col].unique()), "unique values in ", col)

source_code/notebooks/modeling.py

@@ -0,0 +1,378 @@
+# -*- coding: utf-8 -*-
+"""modeling.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1x78fRDZAuK5FaSTKHPGy8eSbZ_gYAFr6
+"""
+
+from google.colab import drive
+drive.mount('/content/drive')
+
+#!pip install -qqq h5py
+
+#!pip install --upgrade -qqq gensim
+
+!python -m spacy download en_core_web_lg
+
+!pip install -U SpaCy==2.2.0
+
+## Import required libraries
+
+## warnings
+import warnings
+warnings.filterwarnings("ignore")
+
+## for data
+import numpy as np
+import pandas as pd
+
+## for plotting
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+## TF-IDF 
+from sklearn.feature_extraction.text import TfidfVectorizer
+
+## T-Sne
+from yellowbrick.text import TSNEVisualizer
+from sklearn import manifold
+
+## Train-Test Split
+from sklearn.model_selection import train_test_split
+
+## Feature selection
+from sklearn import feature_selection
+
+## libraraies for classification
+from sklearn.pipeline import Pipeline
+import sklearn.metrics as skm
+from sklearn.metrics import confusion_matrix, accuracy_score
+from sklearn.linear_model import LogisticRegression
+from sklearn.neighbors import KNeighborsClassifier
+from sklearn.svm import SVC
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.neural_network import MLPClassifier
+from sklearn.ensemble import RandomForestClassifier
+
+## for saving model
+import pickle
+
+## for explainer
+#from lime import lime_text
+
+## detokenization
+from nltk.tokenize.treebank import TreebankWordDetokenizer
+
+## for word embedding with gensim
+import gensim
+import gensim.downloader as gensim_api
+from gensim.models import Word2Vec
+from gensim.models import KeyedVectors
+from keras.preprocessing.text import Tokenizer
+from keras.preprocessing.sequence import pad_sequences
+
+## for word embedding with Spacy
+import spacy
+import en_core_web_lg
+
+## for deep learning
+from keras.models import load_model
+from keras.models import Model, Sequential
+from keras.callbacks import EarlyStopping, ModelCheckpoint
+from keras.layers import Conv1D, Dense, Input, LSTM, Embedding, Dropout, Activation, MaxPooling1D
+from tensorflow.keras import models, layers, preprocessing as kprocessing
+from tensorflow.keras import backend as K
+import tensorflow as tf
+import keras
+from keras.layers import Lambda
+import tensorflow as tf
+from keras.models import model_from_json
+
+## for bert language model
+#import transformers
+
+"""## Loading the dataset:"""
+
+df_all = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/processed_data/processed_data.csv",
+                     sep='\t', encoding='utf-8')
+
+df_all
+
+"""## Classification models as well as LSTM with pretrained model(Spacy):
+
+In order to run a supervised learning model, we first need to convert the clean_text into feature representation.
+"""
+
+nlp = en_core_web_lg.load()
+
+## word-embedding
+all_vectors = pd.np.array([pd.np.array([token.vector for token in nlp(s)]).mean(axis=0) * pd.np.ones((300)) \
+                           for s in df_all['clean_text']])
+
+# split out validation dataset for the end
+Y= df_all["label"]
+X = all_vectors
+
+from sklearn.model_selection import train_test_split, KFold, cross_val_score, GridSearchCV
+validation_size = 0.3
+seed = 7
+X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=validation_size, random_state=seed)
+
+# test options for classification
+num_folds = 10
+seed = 7
+scoring = 'accuracy'
+
+## spot check the algorithms
+models = []
+models.append(('LR', LogisticRegression()))
+models.append(('KNN', KNeighborsClassifier()))
+models.append(('CART', DecisionTreeClassifier()))
+models.append(('SVM', SVC()))
+## Neural Network
+models.append(('NN', MLPClassifier()))
+## Ensable Models 
+models.append(('RF', RandomForestClassifier()))
+
+## Running the classification models
+results = []
+names = []
+kfold_results = []
+test_results = []
+train_results = []
+for name, model in models:
+    kfold = KFold(n_splits=num_folds, random_state=seed)
+    cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring)
+    results.append(cv_results)
+    names.append(name)
+    #msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
+    #print(msg)
+   
+   # Full Training period
+    res = model.fit(X_train, Y_train)
+    train_result = accuracy_score(res.predict(X_train), Y_train)
+    train_results.append(train_result)
+    
+    # Test results
+    test_result = accuracy_score(res.predict(X_test), Y_test)
+    test_results.append(test_result)    
+    
+    msg = "%s: %f (%f) %f %f" % (name, cv_results.mean(), cv_results.std(), train_result, test_result)
+    print(msg)
+    print(confusion_matrix(res.predict(X_test), Y_test))
+    #print(classification_report(res.predict(X_test), Y_test))
+
+# compare algorithms
+from matplotlib import pyplot
+fig = pyplot.figure()
+ind = np.arange(len(names))  # the x locations for the groups
+width = 0.35  # the width of the bars
+fig.suptitle('Algorithm Comparison')
+ax = fig.add_subplot(111)
+pyplot.bar(ind - width/2, train_results,  width=width, label='Train Error')
+pyplot.bar(ind + width/2, test_results, width=width, label='Test Error')
+fig.set_size_inches(15,8)
+pyplot.legend()
+ax.set_xticks(ind)
+ax.set_xticklabels(names)
+pyplot.show()
+
+"""The best model with the highest accuracy is **Support Vector Machine(SVM)** with **85.79**% accuracy on test dataset. Logistic Regression performed good as well but we see overfitting problem with CART, NN and RF.
+
+### LSTM model:
+"""
+
+### Create sequence
+vocabulary_size = 20000
+tokenizer = Tokenizer(num_words= vocabulary_size)
+tokenizer.fit_on_texts(df_all['clean_text'])
+sequences = tokenizer.texts_to_sequences(df_all['clean_text'])
+X_LSTM = pad_sequences(sequences, maxlen=50)
+
+## Split the data into train and test
+Y_LSTM = df_all["label"]
+X_train_LSTM, X_test_LSTM, Y_train_LSTM, Y_test_LSTM = train_test_split(X_LSTM, \
+                       Y_LSTM, test_size=validation_size, random_state=seed)
+
+from keras.wrappers.scikit_learn import KerasClassifier
+def create_model(input_length=50):
+    model = Sequential()
+    model.add(Embedding(20000, 300, input_length=50))
+    model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
+    model.add(Dense(1, activation='sigmoid'))
+    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])    
+    return model    
+model_LSTM = KerasClassifier(build_fn=create_model, epochs=3, verbose=1, validation_split=0.4)
+model_LSTM.fit(X_train_LSTM, Y_train_LSTM)
+
+train_result_LSTM = accuracy_score(model_LSTM.predict(X_train_LSTM), Y_train_LSTM)
+# Test results
+test_result_LSTM = accuracy_score(model_LSTM.predict(X_test_LSTM), Y_test_LSTM)
+
+print("train result:", train_result_LSTM)
+print("test result:", test_result_LSTM)
+
+confusion_matrix(model_LSTM.predict(X_test_LSTM), Y_test_LSTM)
+
+"""### Compare all the models:"""
+
+train_results.append(train_result_LSTM);test_results.append(test_result_LSTM)
+names.append("LSTM")
+
+# compare algorithms
+from matplotlib import pyplot
+fig = pyplot.figure()
+ind = np.arange(len(names))  # the x locations for the groups
+width = 0.35  # the width of the bars
+fig.suptitle('Algorithm Comparison')
+ax = fig.add_subplot(111)
+pyplot.bar(ind - width/2, train_results,  width=width, label='Train Error')
+pyplot.bar(ind + width/2, test_results, width=width, label='Test Error')
+fig.set_size_inches(15,8)
+pyplot.legend()
+ax.set_xticks(ind)
+ax.set_xticklabels(names)
+pyplot.show()
+plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/classification_comparision.png')
+
+"""## Evaluate the performance:
+
+*   **Accuracy:** the fraction of predictions the model got right.
+*   **Confusion Matrix:** a summary table that breaks down the number of correct and incorrect predictions by each class.
+*   **ROC:** a plot that illustrates the true positive rate against the false positive rate at various threshold settings. The area under the curve (AUC) indicates the probability that the classifier will rank a randomly chosen positive observation higher than a randomly chosen negative one.
+*  **Precision:** the fraction of relevant instances among the retrieved instances.
+*  **Recall:** the fraction of the total amount of relevant instances that were actually retrieved.
+"""
+
+def conf_matrix_acc(y_true, y_pred):
+  ## Plot confusion matrix
+  cm = confusion_matrix(y_true, y_pred)
+  fig, ax = plt.subplots()
+  sns.heatmap(cm, annot=True, fmt='d', ax=ax, cmap=plt.cm.Blues, 
+            cbar=False)
+  ax.set(xlabel="Pred", ylabel="True", xticklabels=classes, 
+       yticklabels=classes, title="Confusion matrix")
+  plt.yticks(rotation=0)
+  print("=========================================")
+  print(f'Accuracy score is : {accuracy_score(y_true, y_pred)}')
+  print("=========================================")
+  print("Detail:")
+  print(skm.classification_report(y_true, y_pred))
+
+## Plot ROC and precision-recall curve
+def roc_precision_auc():
+  fig, ax = plt.subplots(nrows=1, ncols=2)
+  ## Plot roc
+  for i in range(len(classes)):
+      fpr, tpr, thresholds = skm.roc_curve(y_test_array[:,i],  
+                            probs[:,i])
+      ax[0].plot(fpr, tpr, lw=3, 
+                label='{0} (area={1:0.2f})'.format(classes[i], 
+                                skm.auc(fpr, tpr))
+                 )
+  ax[0].plot([0,1], [0,1], color='navy', lw=3, linestyle='--')
+  ax[0].set(xlim=[-0.05,1.0], ylim=[0.0,1.05], 
+            xlabel='False Positive Rate', 
+            ylabel="True Positive Rate (Recall)", 
+            title="Receiver operating characteristic")
+  ax[0].legend(loc="lower right")
+  ax[0].grid(True)
+
+  ## Plot precision-recall curve
+  for i in range(len(classes)):
+    precision, recall, thresholds = skm.precision_recall_curve(
+                y_test_array[:,i], probs[:,i])
+    ax[1].plot(recall, precision, lw=3, 
+               label='{0} (area={1:0.2f})'.format(classes[i], 
+                                  skm.auc(recall, precision))
+              )
+  ax[1].set(xlim=[0.0,1.05], ylim=[0.0,1.05], xlabel='Recall', 
+            ylabel="Precision", title="Precision-Recall curve")
+  ax[1].legend(loc="best")
+  ax[1].grid(True)
+  plt.show()
+  #plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/ROC_Precision_LR.png')
+  #plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/ROC_Precision_SVM.png')  
+  ## AUC score
+  print(f'AUC score is : {skm.roc_auc_score(Y_test, probs[:,1])}')
+
+"""## Support Vector Machine(SVM) with word embedding:"""
+
+nlp = en_core_web_lg.load()
+
+## word-embedding
+all_vectors = pd.np.array([pd.np.array([token.vector for token in nlp(s)]).mean(axis=0) * pd.np.ones((300)) \
+                           for s in df_all['clean_text']])
+
+# split out validation dataset for the end
+Y= df_all["label"]
+X = all_vectors
+
+from sklearn.model_selection import train_test_split, KFold, cross_val_score, GridSearchCV
+validation_size = 0.3
+seed = 7
+X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=validation_size, random_state=seed)
+
+# test options for classification
+num_folds = 10
+seed = 7
+scoring = 'accuracy'
+
+#Create a svm Classifier
+clf = SVC(probability=True)
+
+## Running the svm Classifier
+   
+# Full Training period
+res = clf.fit(X_train, Y_train)
+train_result = accuracy_score(res.predict(X_train), Y_train)
+test_result = accuracy_score(res.predict(X_test), Y_test)
+
+print("train_result:", "test_resuld:", train_result, test_result, sep=" ")
+
+## Save the Modle to file in the current working directory
+SVM = "/content/drive/MyDrive/NLP/Depression_Detection/modeling/model_svm1.pkl"  
+
+with open(SVM, 'wb') as file:  
+    pickle.dump(clf, file)
+
+## Load the Model back from file
+with open(SVM, 'rb') as file:  
+    clf = pickle.load(file)
+
+clf
+
+## Test results
+## 
+y_pred_svm = res.predict(X_test) 
+classes = np.unique(Y_test.to_list())
+y_test_array = pd.get_dummies(Y_test, drop_first=False).values
+probs = res.predict_proba(X_test)
+conf_matrix_acc(Y_test.to_list(),y_pred_svm)
+roc_precision_auc()
+
+"""## Exploring False positive and False negative:"""
+
+## creating lists of true values and predictions
+y_test_1 = [x for x in y_test]
+y_pred_lr_1 = [x for x in y_pred_lr]
+
+## Find the indices of wrong predictions
+idx = [] 
+for i in range(len(y_test_1)):
+  if y_test_1[i] != y_pred_lr_1[i]:
+    idx.append(i)
+    i+=1
+
+print('There are", {} "wrong preditions", len(idx))
+
+wrong_arr = cv.inverse_transform(X_test_tfidf[idx])
+
+## detokenize the wrong array
+detokenized = [TreebankWordDetokenizer().detokenize(x) for x in wrong_arr]
+
+detokenized[:50]
+
+"""There is no specific patterns between false positive and false negative predictions."""

source_code/notebooks/old_models.py

@@ -0,0 +1,637 @@
+# -*- coding: utf-8 -*-
+"""old_models.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1Oc7A5TaGLg1qkYXzf0qLGIe0_ZxyAnXE
+
+This notebook contains Feature selection with Chi-Square test, Logistic Regression with TFIDF as well as Bidirectional LSTM with gensim to classifies a given tweet into depressive or non-depressive ones.
+"""
+
+from google.colab import drive
+drive.mount('/content/drive')
+
+## Import required libraries
+
+## warnings
+import warnings
+warnings.filterwarnings("ignore")
+
+## for data
+import numpy as np
+import pandas as pd
+
+## for plotting
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+## TF-IDF 
+from sklearn.feature_extraction.text import TfidfVectorizer
+
+## T-Sne
+from yellowbrick.text import TSNEVisualizer
+from sklearn import manifold
+
+## Train-Test Split
+from sklearn.model_selection import train_test_split
+
+## Feature selection
+from sklearn import feature_selection
+
+## libraraies for classification
+from sklearn.pipeline import Pipeline
+import sklearn.metrics as skm
+from sklearn.metrics import confusion_matrix, accuracy_score
+from sklearn.linear_model import LogisticRegression
+from sklearn.neighbors import KNeighborsClassifier
+from sklearn.svm import SVC
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.neural_network import MLPClassifier
+from sklearn.ensemble import RandomForestClassifier
+
+## for saving model
+import pickle
+
+## detokenization
+from nltk.tokenize.treebank import TreebankWordDetokenizer
+
+## for word embedding with gensim
+import gensim
+import gensim.downloader as gensim_api
+from gensim.models import Word2Vec
+from gensim.models import KeyedVectors
+from keras.preprocessing.text import Tokenizer
+from keras.preprocessing.sequence import pad_sequences
+
+## for word embedding with Spacy
+# import spacy
+# import en_core_web_lg
+
+## for deep learning
+from keras.models import load_model
+from keras.models import Model, Sequential
+from keras.callbacks import EarlyStopping, ModelCheckpoint
+from keras.layers import Conv1D, Dense, Input, LSTM, Embedding, Dropout, Activation, MaxPooling1D
+from tensorflow.keras import models, layers, preprocessing as kprocessing
+from tensorflow.keras import backend as K
+import tensorflow as tf
+import keras
+from keras.layers import Lambda
+import tensorflow as tf
+from keras.models import model_from_json
+
+## for bert language model
+#import transformers
+
+"""## Loading the dataset:"""
+
+df_all = pd.read_csv("/content/drive/MyDrive/NLP/Depression_Detection/data_cleaning/processed_data/processed_data.csv",
+                     sep='\t', encoding='utf-8')
+
+df_all
+
+"""## Feature selection
+
+In order to drop some columns and reduce the matrix dimensionality, we can carry out some Feature Selection, the process of selecting a subset of relevant variables. I will proceed as follows:
+
+
+
+1.   treat each category as binary (for example, the “depressive” category is 1 for the depressive tweets and 0 for non_depressive);
+2.   perform a Chi-Square test to determine whether a feature and the (binary) target are independent;
+3.   keep only the features with a certain p-value from the Chi-Square test.
+
+This snippet of code is derived from https://towardsdatascience.com/text-classification-with-nlp-tf-idf-vs-word2vec-vs-bert-41ff868d1794
+"""
+
+y = y_train
+X_names = cv.get_feature_names()
+p_value_limit = 0.95
+df_features = pd.DataFrame()
+for cat in np.unique(y):
+    chi2, p = feature_selection.chi2(X_train_tfidf, y==cat)
+    df_features = df_features.append(pd.DataFrame(
+                   {"feature":X_names, "score":1-p, "y":cat}))
+    df_features = df_features.sort_values(["y","score"], 
+                    ascending=[True,False])
+    df_features = df_features[df_features["score"]>p_value_limit]
+X_names = df_features["feature"].unique().tolist()
+
+print(len(X_names))
+
+"""I reduced the number of features from 20018 to 688 by keeping the most statistically relevant ones. Let’s print some:"""
+
+for cat in np.unique(y):
+   print("# {}:".format(cat))
+   print("  . selected features:",
+         len(df_features[df_features["y"]==cat]))
+   print("  . top features:", ",".join(df_features[df_features["y"]==cat]["feature"].values[:10]))
+   print(" ")
+
+"""## Logistic Regression with TFIDF:
+
+### Spliting data to train and test datasets:
+"""
+
+## split dataset to train and test
+X_train, X_test, y_train, y_test = train_test_split(df_all['clean_text'], df_all['label'], test_size=0.3, random_state= 42)
+
+X_train.shape, X_test.shape, y_train.shape, y_test.shape
+
+"""### TF-IDF
+
+TF-IDF (term frequency and inverse document frequency):
+"""
+
+## Creating the TF-IDF model
+cv = TfidfVectorizer()
+cv.fit(X_train.to_list())
+dic_vocabulary = cv.vocabulary_
+
+X_train_tfidf = cv.transform(X_train.to_list())
+
+X_test_tfidf = cv.transform(X_test.to_list())
+
+cv.inverse_transform(X_test_tfidf[0])
+
+X_train_tfidf.shape
+
+# ## Adding clean tweets to a list called corpus
+# corpus = []
+# corpus = [x for x in df_train['clean_text']] 
+# # corpus = df_train["clean_text"]
+
+"""The feature matrix X_train_tfidf has a shape of 16,464 (Number of documents in training) x 20018 (Length of vocabulary) and it’s pretty sparse:"""
+
+sns.heatmap(X_train_tfidf.todense()[:,np.random.randint(0,X_train_tfidf.shape[1],100)]==0, vmin=0, vmax=1, cbar=False).set_title('Sparse Matrix Sample')
+
+"""In order to know the position of a certain word, we can look it up in the vocabulary:"""
+
+word = "mental"
+dic_vocabulary[word]
+
+"""Build a scikit-learn pipeline: a sequential application of a list of transformations and a final estimator. Putting the Tf-Idf vectorizer and Logistic Regression classifier in a pipeline allows us to transform and predict test data in just one step."""
+
+# classifier = LogisticRegression(solver='liblinear', penalty='l1')
+
+# ## pipeline
+# model = Pipeline([("vectorizer", cv), 
+#                   ("classifier", classifier)])
+# ## train classifier
+# model["classifier"].fit(X_train, y_train)
+# ## test
+# predicted = model.predict(X_test)
+# predicted_prob = model.predict_proba(X_test)
+
+# ## creating the instance of the models
+lr = LogisticRegression(solver='liblinear', penalty='l1')
+## fitting the model
+print(lr.fit(X_train_tfidf, y_train.to_list()))
+
+## Save the Modle to file in the current working directory
+LogisticReg = "/content/drive/MyDrive/NLP/Depression_Detection/modeling/model_LogReg.pkl"  
+
+with open(LogisticReg, 'wb') as file:  
+    pickle.dump(lr, file)
+
+## Load the Model back from file
+with open(LogisticReg, 'rb') as file:  
+    lr = pickle.load(file)
+
+lr
+
+## Test
+y_pred_lr = lr.predict(X_test_tfidf)
+probs = lr.predict_proba(X_test_tfidf)
+classes = np.unique(y_test.to_list())
+y_test_array = pd.get_dummies(y_test, drop_first=False).values
+
+"""## Evaluate the performance:
+
+*   **Accuracy:** the fraction of predictions the model got right.
+*   **Confusion Matrix:** a summary table that breaks down the number of correct and incorrect predictions by each class.
+*   **ROC:** a plot that illustrates the true positive rate against the false positive rate at various threshold settings. The area under the curve (AUC) indicates the probability that the classifier will rank a randomly chosen positive observation higher than a randomly chosen negative one.
+*  **Precision:** the fraction of relevant instances among the retrieved instances.
+*  **Recall:** the fraction of the total amount of relevant instances that were actually retrieved.
+"""
+
+def conf_matrix_acc(y_true, y_pred):
+  ## Plot confusion matrix
+  cm = confusion_matrix(y_true, y_pred)
+  fig, ax = plt.subplots()
+  sns.heatmap(cm, annot=True, fmt='d', ax=ax, cmap=plt.cm.Blues, 
+            cbar=False)
+  ax.set(xlabel="Pred", ylabel="True", xticklabels=classes, 
+       yticklabels=classes, title="Confusion matrix")
+  plt.yticks(rotation=0)
+  print("=========================================")
+  print(f'Accuracy score is : {accuracy_score(y_true, y_pred)}')
+  print("=========================================")
+  print("Detail:")
+  print(skm.classification_report(y_true, y_pred))
+
+## Plot ROC and precision-recall curve
+def roc_precision_auc():
+  fig, ax = plt.subplots(nrows=1, ncols=2)
+  ## Plot roc
+  for i in range(len(classes)):
+      fpr, tpr, thresholds = skm.roc_curve(y_test_array[:,i],  
+                            probs[:,i])
+      ax[0].plot(fpr, tpr, lw=3, 
+                label='{0} (area={1:0.2f})'.format(classes[i], 
+                                skm.auc(fpr, tpr))
+                 )
+  ax[0].plot([0,1], [0,1], color='navy', lw=3, linestyle='--')
+  ax[0].set(xlim=[-0.05,1.0], ylim=[0.0,1.05], 
+            xlabel='False Positive Rate', 
+            ylabel="True Positive Rate (Recall)", 
+            title="Receiver operating characteristic")
+  ax[0].legend(loc="lower right")
+  ax[0].grid(True)
+
+  ## Plot precision-recall curve
+  for i in range(len(classes)):
+    precision, recall, thresholds = skm.precision_recall_curve(
+                y_test_array[:,i], probs[:,i])
+    ax[1].plot(recall, precision, lw=3, 
+               label='{0} (area={1:0.2f})'.format(classes[i], 
+                                  skm.auc(recall, precision))
+              )
+  ax[1].set(xlim=[0.0,1.05], ylim=[0.0,1.05], xlabel='Recall', 
+            ylabel="Precision", title="Precision-Recall curve")
+  ax[1].legend(loc="best")
+  ax[1].grid(True)
+  plt.show()
+  #plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/ROC_Precision_LR.png')
+  plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/ROC_Precision_SVM.png')  
+  ## AUC score
+  print(f'AUC score is : {skm.roc_auc_score(Y_test, probs[:,1])}')
+
+conf_matrix_acc(y_test.to_list(),y_pred_lr)
+
+roc_precision_auc()
+
+
+
+"""## Bidirectional LSTM:
+
+In Python, you can load a pre-trained Word Embedding model from genism-data like this:
+"""
+
+nlp_pre = gensim_api.load("word2vec-google-news-300")
+
+word = "anxiety"
+fig = plt.figure()
+## word embedding
+tot_words = [word] + [tupla[0] for tupla in 
+                 nlp_pre.most_similar(word, topn=20)]
+X = nlp_pre[tot_words]
+## pca to reduce dimensionality from 300 to 3
+pca = manifold.TSNE(perplexity=40, n_components=3, init='pca')
+X = pca.fit_transform(X)
+## create dtf
+dtf_ = pd.DataFrame(X, index=tot_words, columns=["x","y","z"])
+dtf_["input"] = 0
+dtf_["input"].iloc[0:1] = 1
+## plot 3d
+from mpl_toolkits.mplot3d import Axes3D
+ax = fig.add_subplot(111, projection='3d')
+ax.scatter(dtf_[dtf_["input"]==0]['x'], 
+           dtf_[dtf_["input"]==0]['y'], 
+           dtf_[dtf_["input"]==0]['z'], c="black")
+ax.scatter(dtf_[dtf_["input"]==1]['x'], 
+           dtf_[dtf_["input"]==1]['y'], 
+           dtf_[dtf_["input"]==1]['z'], c="red")
+ax.set(xlabel=None, ylabel=None, zlabel=None, xticklabels=[], 
+       yticklabels=[], zticklabels=[])
+for label, row in dtf_[["x","y","z"]].iterrows():
+    x, y, z = row
+    ax.text(x, y, z, s=label)
+
+"""Instead of using a pre-trained model, I am going to fit my own Word2Vec on the training data corpus with gensim. Before fitting the model, the corpus needs to be transformed into a list of lists of n-grams. In this particular case, I’ll try to capture unigrams (“york”), bigrams (“new york”), and trigrams (“new york city”)."""
+
+## split dataset
+dtf_train, dtf_test = train_test_split(df_all, test_size=0.3)
+## get target
+y_train = dtf_train["label"].values
+y_test = dtf_test["label"].values
+
+corpus = []
+corpus = [x for x in dtf_train['clean_text']]
+
+## create list of lists of unigrams
+lst_corpus = []
+for string in corpus:
+   lst_words = str(string).split()
+   lst_grams = [" ".join(lst_words[i:i+1]) 
+               for i in range(0, len(lst_words), 1)]
+   lst_corpus.append(lst_grams)
+
+## detect bigrams and trigrams
+bigrams_detector = gensim.models.phrases.Phrases(lst_corpus, 
+                 delimiter=" ".encode(), min_count=5, threshold=10)
+bigrams_detector = gensim.models.phrases.Phraser(bigrams_detector)
+trigrams_detector = gensim.models.phrases.Phrases(bigrams_detector[lst_corpus], 
+            delimiter=" ".encode(), min_count=5, threshold=10)
+trigrams_detector = gensim.models.phrases.Phraser(trigrams_detector)
+
+"""When fitting the Word2Vec, you need to specify:
+
+* the target size of the word vectors, I’ll use 300;
+* the window, or the maximum distance between the current and predicted word within a sentence, I’ll use the mean length of text in the corpus;
+* the training algorithm, I’ll use skip-grams (sg=1) as in general it has better results.
+"""
+
+## fit w2v
+nlp = gensim.models.word2vec.Word2Vec(lst_corpus, size=300,   
+            window=8, min_count=1, sg=1, iter=30)
+
+"""We have our embedding model, so we can select any word from the corpus and transform it into a vector."""
+
+word = "anxiety"
+nlp[word].shape
+
+"""We can even use it to visualize a word and its context into a smaller dimensional space (2D or 3D) by applying any dimensionality reduction algorithm (i.e. TSNE)."""
+
+word = "anxiety"
+fig = plt.figure()
+## word embedding
+tot_words = [word] + [tupla[0] for tupla in 
+                 nlp.most_similar(word, topn=20)]
+X = nlp[tot_words]
+## pca to reduce dimensionality from 300 to 3
+pca = manifold.TSNE(perplexity=40, n_components=3, init='pca')
+X = pca.fit_transform(X)
+## create dtf
+dtf_ = pd.DataFrame(X, index=tot_words, columns=["x","y","z"])
+dtf_["input"] = 0
+dtf_["input"].iloc[0:1] = 1
+## plot 3d
+from mpl_toolkits.mplot3d import Axes3D
+ax = fig.add_subplot(111, projection='3d')
+ax.scatter(dtf_[dtf_["input"]==0]['x'], 
+           dtf_[dtf_["input"]==0]['y'], 
+           dtf_[dtf_["input"]==0]['z'], c="black")
+ax.scatter(dtf_[dtf_["input"]==1]['x'], 
+           dtf_[dtf_["input"]==1]['y'], 
+           dtf_[dtf_["input"]==1]['z'], c="red")
+ax.set(xlabel=None, ylabel=None, zlabel=None, xticklabels=[], 
+       yticklabels=[], zticklabels=[])
+for label, row in dtf_[["x","y","z"]].iterrows():
+    x, y, z = row
+    ax.text(x, y, z, s=label)
+
+"""The word vectors can be used in a neural network as weights in the follwing procedure:
+1. Transform the corpus into padded sequences of word ids to get a feature matrix.
+2. Create an embedding matrix so that the vector of the word with id N is located at the Nth row.
+3. Build a neural network with an embedding layer that weighs every word in the sequences with the corresponding vector.
+
+**Feature Engineering:** by transforming the same preprocessed corpus (list of lists of n-grams) given to the Word2Vec into a list of sequences using tensorflow/keras:
+"""
+
+## tokenize text
+tokenizer = kprocessing.text.Tokenizer(lower=True, split=' ', 
+                     oov_token="NaN", 
+                     filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n')
+tokenizer.fit_on_texts(lst_corpus)
+dic_vocabulary = tokenizer.word_index
+
+## create sequence
+lst_text2seq= tokenizer.texts_to_sequences(lst_corpus)
+
+## padding sequence
+X_train = kprocessing.sequence.pad_sequences(lst_text2seq, 
+                    maxlen=35, padding="post", truncating="post")
+
+X_train.shape
+
+"""The feature matrix X_train has a shape of 16559 x 35 (Number of sequences x Sequences max length). Let’s visualize it:"""
+
+sns.heatmap(X_train==0, vmin=0, vmax=1, cbar=False)
+plt.show()
+
+"""Every text in the corpus is now an id sequence with length 35. For instance, if a text had 20 tokens in it, then the sequence is composed of 20 ids + 15 0s, which is the padding element (while the id for word not in the vocabulary is 1)
+
+Let’s print how a text from the train set has been transformed into a sequence with the padding and the vocabulary.
+"""
+
+i = 8
+
+## list of text: ["I like this", ...]
+len_txt = len(dtf_train["clean_text"].iloc[i].split())
+print("from: ", dtf_train["clean_text"].iloc[i], "| len:", len_txt)
+
+## sequence of token ids: [[1, 2, 3], ...]
+len_tokens = len(X_train[i])
+print("to: ", X_train[i], "| len:", len(X_train[i]))
+
+## vocabulary: {"I":1, "like":2, "this":3, ...}
+print("check: ", dtf_train["clean_text"].iloc[i].split()[0], 
+      " -- idx in vocabulary -->", 
+      dic_vocabulary[dtf_train["clean_text"].iloc[i].split()[0]])
+
+print("vocabulary: ", dict(list(dic_vocabulary.items())[0:5]), "... (padding element, 0)")
+
+corpus = dtf_test["clean_text"]
+
+## create list of n-grams
+lst_corpus = []
+for string in corpus:
+    lst_words = str(string).split()
+    lst_grams = [" ".join(lst_words[i:i+1]) for i in range(0, 
+                 len(lst_words), 1)]
+    lst_corpus.append(lst_grams)
+    
+## detect common bigrams and trigrams using the fitted detectors
+lst_corpus = list(bigrams_detector[lst_corpus])
+lst_corpus = list(trigrams_detector[lst_corpus])
+## text to sequence with the fitted tokenizer
+lst_text2seq = tokenizer.texts_to_sequences(lst_corpus)
+
+## padding sequence
+X_test = kprocessing.sequence.pad_sequences(lst_text2seq, maxlen=35,
+             padding="post", truncating="post")
+
+X_test.shape
+
+sns.heatmap(X_test==0, vmin=0, vmax=1, cbar=False)
+plt.show()
+
+"""We’ve got our X_train and X_test, now we need to create the embedding matrix that will be used as a weight matrix in the neural network."""
+
+## start the matrix (length of vocabulary x vector size) with all 0s
+embeddings = np.zeros((len(dic_vocabulary)+1, 300))
+for word,idx in dic_vocabulary.items():
+    ## update the row with vector
+    try:
+        embeddings[idx] =  nlp[word]
+    ## if word not in model then skip and the row stays all 0s
+    except:
+        pass
+
+embeddings.shape
+
+"""That code generates a matrix of shape 20,050 x 300 (Length of vocabulary extracted from the corpus x Vector size). It can be navigated by word id, which can be obtained from the vocabulary."""
+
+word = "anxiety"
+print("dic[word]:", dic_vocabulary[word], "|idx")
+print("embeddings[idx]:", embeddings[dic_vocabulary[word]].shape, 
+      "|vector")
+
+"""### Deep Learning:
+
+It’s finally time to build a deep learning model. I’m going to use the embedding matrix in the first Embedding layer of the neural network that I will build and train to classify the news. Each id in the input sequence will be used as the index to access the embedding matrix. The output of this Embedding layer will be a 2D matrix with a word vector for each word id in the input sequence (Sequence length x Vector size). Let’s use the sentence “I like this article” as an example:
+
+My neural network shall be structured as follows:
+
+* An Embedding layer that takes the sequences as input and the word vectors as weights, just as described before.
+
+* A simple Attention layer that won’t affect the predictions but it’s going to capture the weights of each instance and allow us to build a nice explainer (it isn't necessary for the predictions, just for the explainability, so you can skip it).
+
+* Two layers of Bidirectional LSTM to model the order of words in a sequence in both directions.
+
+* Two final dense layers that will predict the probability of each category.
+"""
+
+## code attention layer
+def attention_layer(inputs, neurons):
+    x = layers.Permute((2,1))(inputs)
+    x = layers.Dense(neurons, activation="softmax")(x)
+    x = layers.Permute((2,1), name="attention")(x)
+    x = layers.multiply([inputs, x])
+    return x
+
+## input
+x_in = layers.Input(shape=(35,))
+## embedding
+x = layers.Embedding(input_dim=embeddings.shape[0],  
+                     output_dim=embeddings.shape[1], 
+                     weights=[embeddings],
+                     input_length=35, trainable=False)(x_in)
+## apply attention
+x = attention_layer(x, neurons=35)
+## 2 layers of bidirectional lstm
+x = layers.Bidirectional(layers.LSTM(units=35, dropout=0.2, 
+                         return_sequences=True))(x)
+x = layers.Bidirectional(layers.LSTM(units=35, dropout=0.2))(x)
+## final dense layers
+x = layers.Dense(64, activation='relu')(x)
+y_out = layers.Dense(1, activation='sigmoid')(x)
+## compile
+model = models.Model(x_in, y_out)
+model.compile(loss='binary_crossentropy',
+              optimizer='adam', metrics=['accuracy'])
+
+model.summary()
+
+## encode y
+dic_y_mapping = {n:label for n,label in 
+                 enumerate(np.unique(y_train))}
+inverse_dic = {v:k for k,v in dic_y_mapping.items()}
+y_train = np.array([inverse_dic[y] for y in y_train])
+## train
+training = model.fit(x=X_train, y=y_train, batch_size=256, 
+                     epochs=30, shuffle=True, verbose=0, 
+                     validation_split=0.3)
+
+## plot loss and accuracy
+metrics = [k for k in training.history.keys() if ("loss" not in k) and ("val" not in k)]
+fig, ax = plt.subplots(nrows=1, ncols=2, sharey=True)
+ax[0].set(title="Training")
+ax11 = ax[0].twinx()
+ax[0].plot(training.history['loss'], color='black')
+ax[0].set_xlabel('Epochs')
+ax[0].set_ylabel('Loss', color='black')
+for metric in metrics:
+    ax11.plot(training.history[metric], label=metric)
+ax11.set_ylabel("Score", color='steelblue')
+ax11.legend()
+ax[1].set(title="Validation")
+ax22 = ax[1].twinx()
+ax[1].plot(training.history['val_loss'], color='black')
+ax[1].set_xlabel('Epochs')
+ax[1].set_ylabel('Loss', color='black')
+for metric in metrics:
+     ax22.plot(training.history['val_'+metric], label=metric)
+ax22.set_ylabel("Score", color="steelblue")
+plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/loss_accuracy_LSTM_3.png')
+plt.show()
+
+# serialize model to JSON
+model_json = model.to_json()
+with open("/content/drive/MyDrive/NLP/Depression_Detection/modeling/model.json", "w") as json_file:
+    json_file.write(model_json)
+# serialize weights to HDF5
+model.save_weights("/content/drive/MyDrive/NLP/Depression_Detection/modeling/model.h5")
+print("Saved model to disk")
+
+loaded_model = model_from_json(open("/content/drive/MyDrive/NLP/Depression_Detection/modeling/model.json", "r").read(),
+                              custom_objects={'tf': tf})
+json_file.close() 
+# load weights into new model
+loaded_model.load_weights("/content/drive/MyDrive/NLP/Depression_Detection/modeling/model.h5")
+print("Loaded model from disk")
+
+labels_pred = model.predict(X_test)
+labels_pred = np.round(labels_pred.flatten())
+accuracy = accuracy_score(y_test, labels_pred)
+classes = np.unique(y_test)
+print("Accuracy: %.2f%%" % (accuracy*100))
+
+def conf_matrix_acc2(y_true, y_pred):
+  ## Plot confusion matrix
+  cm = confusion_matrix(y_test, y_pred)
+  fig, ax = plt.subplots()
+  sns.heatmap(cm, annot=True, fmt='d', ax=ax, cmap=plt.cm.Blues, 
+            cbar=False)
+  ax.set(xlabel="Pred", ylabel="True", xticklabels=classes, 
+       yticklabels=classes, title="Confusion matrix")
+  plt.yticks(rotation=0)
+  print("=========================================")
+  print(f'Accuracy score is : {accuracy_score(y_true, y_pred)}')
+  print("=========================================")
+  print("Detail:")
+  print(skm.classification_report(y_test, y_pred))
+
+conf_matrix_acc2(y_test, labels_pred)
+
+# classes = np.unique(y_test)
+# y_test_array = pd.get_dummies(y_test, drop_first=False).values
+# predicted_prob = model.predict_on_batch(X_test)
+
+# ## Plot ROC and precision-recall curve
+# def roc_precision_auc2():
+#   fig, ax = plt.subplots(nrows=1, ncols=2)
+#   ## Plot roc
+#   for i in range(len(classes)):
+#       fpr, tpr, thresholds = skm.roc_curve(y_test_array[:,i],  
+#                             predicted_prob[:,i])
+#       ax[0].plot(fpr, tpr, lw=3, 
+#                 label='{0} (area={1:0.2f})'.format(classes[i], 
+#                                 skm.auc(fpr, tpr))
+#                  )
+#   ax[0].plot([0,1], [0,1], color='navy', lw=3, linestyle='--')
+#   ax[0].set(xlim=[-0.05,1.0], ylim=[0.0,1.05], 
+#             xlabel='False Positive Rate', 
+#             ylabel="True Positive Rate (Recall)", 
+#             title="Receiver operating characteristic")
+#   ax[0].legend(loc="lower right")
+#   ax[0].grid(True)
+
+#   ## Plot precision-recall curve
+#   for i in range(len(classes)):
+#     precision, recall, thresholds = skm.precision_recall_curve(
+#                 y_test_array[:,i], probs[:,i])
+#     ax[1].plot(recall, precision, lw=3, 
+#                label='{0} (area={1:0.2f})'.format(classes[i], 
+#                                   skm.auc(recall, precision))
+#               )
+#   ax[1].set(xlim=[0.0,1.05], ylim=[0.0,1.05], xlabel='Recall', 
+#             ylabel="Precision", title="Precision-Recall curve")
+#   ax[1].legend(loc="best")
+#   ax[1].grid(True)
+#   plt.show()
+#   #plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/ROC_Precision_LR.png')
+#   #plt.savefig('/content/drive/MyDrive/NLP/Depression_Detection/modeling/ROC_Precision_LSTM.png')  
+#   ## AUC score
+#   print(f'AUC score is : {skm.roc_auc_score(y_test, probs[:,1])}')

source_code/notebooks/testing.py

@@ -0,0 +1,283 @@
+# -*- coding: utf-8 -*-
+"""testing.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1MCstbEJ_U20yRJDGRmZTjIpGTCzTFL_o
+"""
+
+from google.colab import drive
+drive.mount('/content/drive')
+
+!pip install -qqq ftfy
+
+!pip install -qqq json_file
+
+!python -m spacy download en_core_web_lg
+
+!pip install -U SpaCy==2.2.0
+
+## Import required libraries
+
+## warnings
+import warnings
+warnings.filterwarnings("ignore")
+
+## for data
+import numpy as np
+import pandas as pd
+
+## for plotting
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+## Bag of Words
+from sklearn.feature_extraction.text import CountVectorizer
+
+## TF-IDF 
+from sklearn.feature_extraction.text import TfidfVectorizer
+
+## Train-Test Split
+from sklearn.model_selection import train_test_split
+
+## for processing
+import nltk
+import re
+import ftfy
+from nltk.stem import WordNetLemmatizer 
+from nltk.corpus import stopwords
+nltk.download('stopwords')
+nltk.download('punkt')
+nltk.download('wordnet')
+nltk.download('averaged_perceptron_tagger')
+
+## Feature selection
+from sklearn import feature_selection
+
+## Support vector machine
+from sklearn.pipeline import Pipeline
+import sklearn.metrics as skm
+from sklearn.metrics import confusion_matrix, accuracy_score
+from sklearn.svm import SVC
+
+## for saving and loading model
+import pickle
+
+## for word embedding with Spacy
+import spacy
+import en_core_web_lg
+
+# ## for word embedding
+# import gensim
+# import gensim.downloader as gensim_api
+# from gensim.models import Word2Vec
+# from gensim.models import KeyedVectors
+# from keras.preprocessing.text import Tokenizer
+# from keras.preprocessing.sequence import pad_sequences
+
+# ## for deep learning
+# from keras.models import load_model
+# from keras.models import Model, Sequential
+# from keras.callbacks import EarlyStopping, ModelCheckpoint
+# from keras.layers import Conv1D, Dense, Input, LSTM, Embedding, Dropout, Activation, MaxPooling1D
+# from tensorflow.keras import models, layers, preprocessing as kprocessing
+# from tensorflow.keras import backend as K
+# from keras.models import model_from_json
+# from keras.layers import Lambda
+# import tensorflow as tf
+# import json
+# import json_file
+
+# Expand Contraction
+cList = {
+  "ain't": "am not",
+  "aren't": "are not",
+  "can't": "cannot",
+  "can't've": "cannot have",
+  "'cause": "because",
+  "could've": "could have",
+  "couldn't": "could not",
+  "couldn't've": "could not have",
+  "didn't": "did not",
+  "doesn't": "does not",
+  "don't": "do not",
+  "hadn't": "had not",
+  "hadn't've": "had not have",
+  "hasn't": "has not",
+  "haven't": "have not",
+  "he'd": "he would",
+  "he'd've": "he would have",
+  "he'll": "he will",
+  "he'll've": "he will have",
+  "he's": "he is",
+  "how'd": "how did",
+  "how'd'y": "how do you",
+  "how'll": "how will",
+  "how's": "how is",
+  "I'd": "I would",
+  "I'd've": "I would have",
+  "I'll": "I will",
+  "I'll've": "I will have",
+  "I'm": "I am",
+  "I've": "I have",
+  "isn't": "is not",
+  "it'd": "it had",
+  "it'd've": "it would have",
+  "it'll": "it will",
+  "it'll've": "it will have",
+  "it's": "it is",
+  "let's": "let us",
+  "ma'am": "madam",
+  "mayn't": "may not",
+  "might've": "might have",
+  "mightn't": "might not",
+  "mightn't've": "might not have",
+  "must've": "must have",
+  "mustn't": "must not",
+  "mustn't've": "must not have",
+  "needn't": "need not",
+  "needn't've": "need not have",
+  "o'clock": "of the clock",
+  "oughtn't": "ought not",
+  "oughtn't've": "ought not have",
+  "shan't": "shall not",
+  "sha'n't": "shall not",
+  "shan't've": "shall not have",
+  "she'd": "she would",
+  "she'd've": "she would have",
+  "she'll": "she will",
+  "she'll've": "she will have",
+  "she's": "she is",
+  "should've": "should have",
+  "shouldn't": "should not",
+  "shouldn't've": "should not have",
+  "so've": "so have",
+  "so's": "so is",
+  "that'd": "that would",
+  "that'd've": "that would have",
+  "that's": "that is",
+  "there'd": "there had",
+  "there'd've": "there would have",
+  "there's": "there is",
+  "they'd": "they would",
+  "they'd've": "they would have",
+  "they'll": "they will",
+  "they'll've": "they will have",
+  "they're": "they are",
+  "they've": "they have",
+  "to've": "to have",
+  "wasn't": "was not",
+  "we'd": "we had",
+  "we'd've": "we would have",
+  "we'll": "we will",
+  "we'll've": "we will have",
+  "we're": "we are",
+  "we've": "we have",
+  "weren't": "were not",
+  "what'll": "what will",
+  "what'll've": "what will have",
+  "what're": "what are",
+  "what's": "what is",
+  "what've": "what have",
+  "when's": "when is",
+  "when've": "when have",
+  "where'd": "where did",
+  "where's": "where is",
+  "where've": "where have",
+  "who'll": "who will",
+  "who'll've": "who will have",
+  "who's": "who is",
+  "who've": "who have",
+  "why's": "why is",
+  "why've": "why have",
+  "will've": "will have",
+  "won't": "will not",
+  "won't've": "will not have",
+  "would've": "would have",
+  "wouldn't": "would not",
+  "wouldn't've": "would not have",
+  "y'all": "you all",
+  "y'alls": "you alls",
+  "y'all'd": "you all would",
+  "y'all'd've": "you all would have",
+  "y'all're": "you all are",
+  "y'all've": "you all have",
+  "you'd": "you had",
+  "you'd've": "you would have",
+  "you'll": "you you will",
+  "you'll've": "you you will have",
+  "you're": "you are",
+  "you've": "you have"
+}
+
+c_re = re.compile('(%s)' % '|'.join(cList.keys()))
+
+def expandContractions(text, c_re=c_re):
+    def replace(match):
+        return cList[match.group(0)]
+    return c_re.sub(replace, text)
+
+## Function to perform stepwise cleaning process
+def tweets_cleaner(tweet):
+  cleaned_tweets = []
+  tweet = tweet.lower() #lowercase
+    
+  # if url links then don't append to avoid news articles
+  # also check tweet length, save those > 5 
+  if re.match("(\w+:\/\/\S+)", tweet) == None and len(tweet) > 5:
+    
+    #remove hashtag, @mention, emoji and image URLs
+    tweet = ' '.join(re.sub("(@[A-Za-z0-9]+)|(\#[A-Za-z0-9]+)|(<Emoji:.*>)|(pic\.twitter\.com\/.*)", " ", tweet).split())
+
+    #fix weirdly encoded texts
+    tweet = ftfy.fix_text(tweet)
+
+    #expand contraction
+    tweet = expandContractions(tweet)
+
+
+    #remove punctuation
+    tweet = ' '.join(re.sub("([^0-9A-Za-z \t])", " ", tweet).split())
+
+    #stop words and lemmatization
+    stop_words = set(stopwords.words('english'))
+    word_tokens = nltk.word_tokenize(tweet)
+
+    lemmatizer=WordNetLemmatizer()
+    filtered_sentence = [lemmatizer.lemmatize(word) for word in word_tokens if not word in stop_words]
+    # back to string from list
+    tweet = ' '.join(filtered_sentence) # join words with a space in between them
+
+    cleaned_tweets.append(tweet)
+
+  return cleaned_tweets
+
+nlp = en_core_web_lg.load()
+
+## Load the model
+SVM = "/content/drive/MyDrive/NLP/Depression_Detection/modeling/model_svm.pkl" 
+with open(SVM, 'rb') as file:  
+    clf = pickle.load(file)
+
+clf
+
+test_tweet = "I hate my life"
+
+corpus = tweets_cleaner(test_tweet)
+
+corpus
+
+## word-embedding
+test = pd.np.array([pd.np.array([token.vector for token in nlp(s)]).mean(axis=0) * pd.np.ones((300)) \
+                           for s in corpus])
+
+labels_pred = clf.predict(test)
+
+labels_pred[0]
+
+# loaded_model = model_from_json(open("/content/drive/MyDrive/NLP/Depression_Detection/modeling/model.json", "r").read(),
+#                               custom_objects={'tf': tf}) 
+# # load weights into new model
+# loaded_model.load_weights("/content/drive/MyDrive/NLP/Depression_Detection/modeling/model.h5")
+# print("Loaded model from disk")

source_code/requirements.txt

@@ -0,0 +1,42 @@
+blis==0.4.1
+certifi==2021.10.8
+charset-normalizer==2.0.7
+click==8.0.3
+cycler==0.11.0
+cymem==2.0.6
+fonttools==4.28.1
+ftfy==6.0.3
+idna==3.3
+joblib==1.1.0
+kiwisolver==1.3.2
+matplotlib==3.5.0
+murmurhash==1.0.6
+nltk==3.6.5
+numpy==1.21.4
+packaging==21.2
+pandas==1.3.4
+Pillow==8.4.0
+plac==0.9.6
+preshed==3.0.6
+pyparsing==2.4.7
+python-dateutil==2.8.2
+pytz==2021.3
+regex==2021.11.10
+requests==2.26.0
+scikit-learn==1.0.1
+scipy==1.7.2
+seaborn==0.11.2
+setuptools-scm==6.3.2
+six==1.16.0
+spacy==2.2.0
+srsly==1.0.5
+thinc==7.1.1
+threadpoolctl==3.0.0
+tomli==1.2.2
+tqdm==4.62.3
+urllib3==1.26.7
+wasabi==0.8.2
+wcwidth==0.2.5
+wordcloud==1.8.1
+Flask==1.1.2
+Flask-Bootstrap==3.3.7.1

source_code/static/overlay.css

@@ -0,0 +1,99 @@
+/* Cinematic Overlay Styles */
+.cinematic-overlay {
+    position: fixed;
+    top: 0;
+    left: 0;
+    width: 100vw;
+    height: 100vh;
+    background: rgba(29, 161, 242, 0.98);
+    /* Classic Twitter Blue #1DA1F2 */
+    z-index: 9999;
+    display: none;
+    align-items: center;
+    justify-content: center;
+    flex-direction: column;
+    opacity: 0;
+    transition: opacity 0.8s ease-in-out;
+    backdrop-filter: blur(10px);
+}
+
+.cinematic-overlay.active {
+    opacity: 1;
+}
+
+.overlay-content {
+    text-align: center;
+    color: var(--clr-white);
+}
+
+.overlay-logo {
+    font-size: 7rem;
+    color: #ffffff;
+    margin-bottom: 30px;
+    opacity: 0;
+    transform: scale(0.5);
+    transition: all 1s cubic-bezier(0.175, 0.885, 0.32, 1.275);
+}
+
+.cinematic-overlay.active .overlay-logo {
+    opacity: 1;
+    transform: scale(1);
+    filter: drop-shadow(0 0 20px rgba(29, 174, 255, 0.6));
+}
+
+.overlay-title {
+    font-family: 'Play', sans-serif;
+    font-size: 3rem;
+    font-weight: 700;
+    letter-spacing: 2px;
+    margin-bottom: 10px;
+    opacity: 0;
+    transform: translateY(20px);
+    transition: all 0.8s ease 0.4s;
+    color: #ffffff;
+    /* Solid white for max contrast */
+    text-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+}
+
+.cinematic-overlay.active .overlay-title {
+    opacity: 1;
+    transform: translateY(0);
+}
+
+.overlay-author {
+    font-family: 'Play', sans-serif;
+    font-size: 1rem;
+    font-size: 1rem;
+    color: #ffffff;
+    /* Solid white */
+    text-shadow: 0 1px 2px rgba(0, 0, 0, 0.2);
+    text-transform: uppercase;
+    letter-spacing: 3px;
+    margin-top: 40px;
+    margin-bottom: 15px;
+    opacity: 0;
+    transition: opacity 1s ease 0.8s;
+}
+
+.cinematic-overlay.active .overlay-author {
+    opacity: 1;
+}
+
+.overlay-names {
+    font-size: 1.5rem;
+    font-weight: 300;
+    opacity: 0;
+    transform: scale(0.9);
+    transition: all 1s ease 1s;
+}
+
+.cinematic-overlay.active .overlay-names {
+    opacity: 1;
+    transform: scale(1);
+}
+
+.overlay-names .separator {
+    color: #ffffff;
+    margin: 0 15px;
+    font-weight: 700;
+}

source_code/static/security.js

@@ -0,0 +1,26 @@
+/**
+ * security.js
+ * Implements low-level security features and Easter eggs.
+ */
+
+// Disable Right Click
+document.addEventListener('contextmenu', function (e) {
+    e.preventDefault();
+});
+
+// Disable F12, Ctrl+Shift+I, Ctrl+Shift+J, Ctrl+U
+document.onkeydown = function (e) {
+    if (e.keyCode == 123) {
+        return false;
+    }
+    if (e.ctrlKey && e.shiftKey && (e.keyCode == 'I'.charCodeAt(0) || e.keyCode == 'J'.charCodeAt(0))) {
+        return false;
+    }
+    if (e.ctrlKey && e.keyCode == 'U'.charCodeAt(0)) {
+        return false;
+    }
+}
+
+// Easter Egg - Console Warning
+console.log("%cStop snooping around!", "color: red; font-family: sans-serif; font-size: 4.5em; font-weight: bolder; text-shadow: #000 1px 1px;");
+console.log("%cThis is a project by Amey Thakur & Mega Satish.", "color: #1daeff; font-family: sans-serif; font-size: 1.5em;");

source_code/static/styles.css

@@ -0,0 +1,237 @@
+/* ==============================================================================
+ * PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+ * AUTHORS: AMEY THAKUR & MEGA SATISH
+ * GITHUB (AMEY): https://github.com/Amey-Thakur
+ * GITHUB (MEGA): https://github.com/msatmod
+ * REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+ * RELEASE DATE: June 5, 2022
+ * LICENSE: MIT License
+ * DESCRIPTION: Global style sheet with a professional and personal design.
+ * ============================================================================== */
+
+@import url('https://fonts.googleapis.com/css2?family=Play:wght@400;700&display=swap');
+
+:root {
+	/* Color Palette */
+	--clr-navy: #0A192F;
+	--clr-slate: #8892B0;
+	--clr-light-slate: #A8B2D1;
+	--clr-white: #E6F1FF;
+	--clr-ivory: #F8F9FA;
+	--clr-accent: #64FFDA;
+	--clr-accent-dark: #1daeff;
+	--clr-bg: #F4F7FB;
+
+	/* Typography */
+	--font-main: 'Play', sans-serif;
+
+	/* Layout */
+	--header-height: 80px;
+	--max-width: 900px;
+	--transition: all 0.3s cubic-bezier(0.645, 0.045, 0.355, 1);
+}
+
+* {
+	box-sizing: border-box;
+	margin: 0;
+	padding: 0;
+}
+
+body {
+	background-color: var(--clr-bg);
+	color: var(--clr-navy);
+	font-family: var(--font-main);
+	line-height: 1.6;
+	overflow-x: hidden;
+	user-select: none;
+}
+
+.container {
+	max-width: var(--max-width);
+	margin: 0 auto;
+	padding: 0 40px;
+}
+
+/* Header & Typography */
+h1,
+h2,
+h3 {
+	font-family: var(--font-main);
+	color: var(--clr-navy);
+}
+
+.hero-section {
+	padding: 80px 0 40px;
+	text-align: center;
+}
+
+.hero-section h1 {
+	font-size: 3rem;
+	margin-bottom: 20px;
+	font-weight: 700;
+}
+
+.hero-section p {
+	font-size: 1.2rem;
+	color: var(--clr-slate);
+	max-width: 600px;
+	margin: 0 auto;
+}
+
+/* Analysis Card */
+.analysis-card {
+	background: white;
+	border-radius: 12px;
+	box-shadow: 0 10px 30px -15px rgba(2, 12, 27, 0.1);
+	padding: 40px;
+	margin-bottom: 30px;
+}
+
+.input-group {
+	margin-top: 30px;
+}
+
+textarea.analysis-input {
+	width: 100%;
+	min-height: 150px;
+	padding: 20px;
+	border: 2px solid #E2E8F0;
+	border-radius: 8px;
+	font-family: var(--font-main);
+	font-size: 1.1rem;
+	transition: var(--transition);
+	resize: vertical;
+	outline: none;
+}
+
+textarea.analysis-input:focus {
+	border-color: var(--clr-accent-dark);
+	box-shadow: 0 0 0 4px rgba(29, 174, 255, 0.1);
+}
+
+.btn-primary {
+	display: inline-block;
+	background-color: var(--clr-accent-dark);
+	color: white;
+	padding: 15px 35px;
+	border-radius: 6px;
+	font-weight: 600;
+	text-decoration: none;
+	border: 2px solid var(--clr-accent-dark);
+	cursor: pointer;
+	transition: var(--transition);
+	margin-top: 20px;
+	font-size: 1rem;
+}
+
+.btn-primary:hover {
+	background-color: white;
+	color: var(--clr-accent-dark);
+	transform: translateY(-3px);
+	box-shadow: 0 5px 15px rgba(29, 174, 255, 0.3);
+}
+
+/* Result Section */
+.result-display {
+	text-align: center;
+	padding: 40px 0;
+}
+
+.result-badge {
+	display: inline-block;
+	padding: 10px 25px;
+	border-radius: 50px;
+	font-weight: 700;
+	text-transform: uppercase;
+	letter-spacing: 1px;
+	margin-top: 20px;
+}
+
+.badge-depressive {
+	background-color: #FFF5F5;
+	color: #C53030;
+	border: 1px solid #FEB2B2;
+}
+
+.badge-non-depressive {
+	background-color: #F0FFF4;
+	color: #276749;
+	border: 1px solid #9AE6B4;
+}
+
+/* Footer & Authorship */
+footer {
+	padding: 60px 0;
+	text-align: center;
+	border-top: 1px solid #E2E8F0;
+	margin-top: 60px;
+}
+
+.authorship {
+	color: var(--clr-slate);
+	font-size: 0.9rem;
+}
+
+.authorship a {
+	color: var(--clr-accent-dark);
+	text-decoration: none;
+	font-weight: 600;
+}
+
+.authorship a:hover {
+	text-decoration: underline;
+}
+
+.metadata {
+	margin-top: 10px;
+	font-size: 0.8rem;
+	color: var(--clr-light-slate);
+}
+
+/* Animations */
+@keyframes fadeIn {
+	from {
+		opacity: 0;
+		transform: translateY(20px);
+	}
+
+	to {
+		opacity: 1;
+		transform: translateY(0);
+	}
+}
+
+.animate-fade {
+	animation: fadeIn 0.8s ease forwards;
+}
+
+/* Brain Icon Animation */
+.brain-trigger {
+	color: var(--clr-accent-dark);
+	margin-bottom: 20px;
+	cursor: pointer;
+	transition: all 0.5s ease;
+	filter: drop-shadow(0 0 5px rgba(29, 174, 255, 0.3));
+}
+
+.brain-trigger:hover {
+	animation: brainPulse 1.5s infinite ease-in-out;
+	color: #4dc4ff;
+	/* Slightly brighter on hover */
+	filter: drop-shadow(0 0 15px rgba(29, 174, 255, 0.8));
+}
+
+@keyframes brainPulse {
+	0% {
+		transform: scale(1);
+	}
+
+	50% {
+		transform: scale(1.15);
+		filter: drop-shadow(0 0 25px rgba(29, 174, 255, 0.9));
+	}
+
+	100% {
+		transform: scale(1);
+	}
+}

source_code/templates/404.html

@@ -0,0 +1,137 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>404 - Page Not Found | Tweet Depression Detection</title>
+    <!-- Simple Brain Icon as Favicon -->
+    <link rel="icon" href="{{url_for('.static', filename='brain.svg')}}" type="image/svg+xml">
+
+    <!-- Modern Typography & Iconography -->
+    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
+    <link rel="stylesheet" href="{{url_for('.static', filename='styles.css')}}">
+    <script src="{{url_for('.static', filename='security.js')}}"></script>
+
+    <style>
+        body {
+            overflow-x: hidden;
+            display: flex;
+            flex-direction: column;
+            min-height: 100vh;
+        }
+
+        .error-container {
+            flex: 1;
+            display: flex;
+            flex-direction: column;
+            justify-content: center;
+            align-items: center;
+            text-align: center;
+            padding: 20px;
+        }
+
+        .error-code {
+            font-size: 10rem;
+            color: var(--clr-accent-dark);
+            font-weight: 700;
+            margin: 0;
+            line-height: 1;
+            position: relative;
+            animation: glitch 1s infinite alternate-reverse;
+            text-shadow: 2px 2px 0 var(--clr-accent), -2px -2px 0 #ff0055;
+        }
+
+        .error-message {
+            font-size: 1.5rem;
+            color: var(--clr-slate);
+            margin-top: 10px;
+            margin-bottom: 40px;
+            letter-spacing: 1px;
+        }
+
+        @keyframes glitch {
+            0% {
+                transform: skew(0deg);
+            }
+
+            20% {
+                transform: skew(-2deg);
+            }
+
+            40% {
+                transform: skew(2deg);
+            }
+
+            60% {
+                transform: skew(-1deg);
+            }
+
+            80% {
+                transform: skew(3deg);
+            }
+
+            100% {
+                transform: skew(0deg);
+            }
+        }
+
+        .broken-icon {
+            font-size: 5rem;
+            color: var(--clr-light-slate);
+            margin-bottom: 20px;
+            animation: float 3s ease-in-out infinite;
+        }
+
+        @keyframes float {
+            0% {
+                transform: translateY(0px) rotate(0deg);
+            }
+
+            50% {
+                transform: translateY(-15px) rotate(5deg);
+            }
+
+            100% {
+                transform: translateY(0px) rotate(0deg);
+            }
+        }
+    </style>
+</head>
+
+<body>
+
+    <header class="hero-section container animate-fade">
+        <i class="fas fa-brain fa-3x" style="color: #1daeff; margin-bottom: 20px;"></i>
+        <h1>Tweet Depression Detection</h1>
+    </header>
+
+    <main class="container animate-fade" style="animation-delay: 0.2s;">
+        <div class="error-container">
+            <i class="fas fa-unlink broken-icon"></i>
+            <h1 class="error-code">404</h1>
+            <p class="error-message">Oops! This tweet seems to have disappeared.</p>
+
+            <a href="{{ url_for('index') }}" class="btn-primary">
+                <i class="fas fa-home" style="margin-right: 8px;"></i> Return to Home
+            </a>
+        </div>
+    </main>
+
+    <footer class="container"
+        style="padding: 60px 0 40px; text-align: center; border-top: 1px solid rgba(136, 146, 176, 0.1);">
+        <p style="color: var(--clr-slate); font-size: 0.95rem; margin-bottom: 12px;">
+            Developed by <a href="https://github.com/Amey-Thakur"
+                style="color: var(--clr-accent-dark); font-weight: 700;">Amey Thakur</a> & <a
+                href="https://github.com/msatmod" style="color: var(--clr-accent-dark); font-weight: 700;">Mega
+                Satish</a>
+        </p>
+        <p style="color: var(--clr-light-slate); font-size: 0.85rem; letter-spacing: 0.5px;">
+            © 2022 • MIT License • <a href="https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS"
+                target="_blank" style="margin-left: 5px;"><i class="fab fa-github"></i> GitHub</a>
+        </p>
+    </footer>
+
+</body>
+
+</html>

source_code/templates/index.html

@@ -0,0 +1,136 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Tweet Depression Detection | AMEY & MEGA</title>
+    <!-- Simple Brain Icon as Favicon -->
+    <link rel="icon" href="{{url_for('.static', filename='brain.svg')}}" type="image/svg+xml">
+
+    <!-- Modern Typography & Iconography -->
+    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
+    <link rel="stylesheet" href="{{url_for('.static', filename='styles.css')}}">
+    <link rel="stylesheet" href="{{url_for('.static', filename='overlay.css')}}">
+    <script src="{{url_for('.static', filename='security.js')}}"></script>
+</head>
+
+<!-- 
+==============================================================================
+PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+AUTHORS: AMEY THAKUR & MEGA SATISH
+GITHUB (AMEY): https://github.com/Amey-Thakur
+GITHUB (MEGA): https://github.com/msatmod
+REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+RELEASE DATE: June 5, 2022
+LICENSE: MIT License
+DESCRIPTION: Main interface for tweet analysis.
+==============================================================================
+-->
+
+<body>
+
+    <header class="hero-section container animate-fade">
+        <i class="fas fa-brain fa-3x brain-trigger" onclick="triggerCinematic()"></i>
+        <h1>Tweet Depression Detection</h1>
+        <p>Using Machine Learning to predict sentiment in tweets.</p>
+    </header>
+
+    <main class="container animate-fade" style="animation-delay: 0.2s;">
+        <section class="analysis-card">
+            <h3><i class="fab fa-twitter" style="margin-right: 10px; color: #1daeff;"></i> Try it Out</h3>
+            <p style="font-size: 0.9rem; color: #8892B0; margin-bottom: 20px;">Paste a tweet below to see how our model
+                classifies its sentiment.</p>
+
+            <form id="analysisForm" action="{{ url_for('predict')}}" method="POST">
+                <div class="input-group">
+                    <textarea name="tweet" class="analysis-input"
+                        placeholder="Paste tweet content here for sentiment analysis..." required></textarea>
+                </div>
+                <div style="text-align: right;">
+                    <button type="submit" class="btn-primary">
+                        <i class="fab fa-twitter" style="margin-right: 8px;"></i> Analyze Tweet
+                    </button>
+                </div>
+            </form>
+
+            <audio id="tweetSound" src="{{url_for('.static', filename='tweet-sound.mp3')}}" preload="auto"></audio>
+
+            <script>
+                // Cinematic Interaction Logic
+                function triggerCinematic() {
+                    const overlay = document.getElementById('cinematicOverlay');
+                    const sound = document.getElementById('tweetSound');
+
+                    overlay.style.display = 'flex';
+                    // Force reflow
+                    void overlay.offsetWidth;
+                    overlay.classList.add('active');
+
+                    setTimeout(() => {
+                        sound.play().catch(err => console.log("Audio playback failed:", err));
+                    }, 200);
+                }
+
+                function closeOverlay() {
+                    const overlay = document.getElementById('cinematicOverlay');
+                    overlay.classList.remove('active');
+                    setTimeout(() => {
+                        overlay.style.display = 'none';
+                    }, 800);
+                }
+
+                document.getElementById('analysisForm').addEventListener('submit', function (e) {
+                    const form = this;
+                    const sound = document.getElementById('tweetSound');
+
+                    e.preventDefault();
+                    sound.play().catch(err => console.log("Audio playback failed:", err));
+
+                    setTimeout(() => {
+                        form.submit();
+                    }, 400); // 400ms delay to let the chirp start
+                });
+            </script>
+        </section>
+
+        <section class="analysis-card" style="background: rgba(29, 174, 255, 0.05);">
+            <h4>How it Works</h4>
+            <p style="font-size: 0.95rem;">This project uses an <strong>SVM (Support Vector Machine)</strong> model
+                combined with <strong>spaCy</strong> word embeddings to process and classify text. It was developed to
+                experiment with modern Machine Learning workflows.</p>
+        </section>
+    </main>
+
+    <footer class="container"
+        style="padding: 60px 0 40px; text-align: center; border-top: 1px solid rgba(136, 146, 176, 0.1);">
+        <p style="color: var(--clr-slate); font-size: 0.95rem; margin-bottom: 12px;">
+            Developed by <a href="https://github.com/Amey-Thakur"
+                style="color: var(--clr-accent-dark); font-weight: 700;">Amey Thakur</a> & <a
+                href="https://github.com/msatmod" style="color: var(--clr-accent-dark); font-weight: 700;">Mega
+                Satish</a>
+        </p>
+        <p style="color: var(--clr-light-slate); font-size: 0.85rem; letter-spacing: 0.5px;">
+            © 2022 • MIT License • <a href="https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS"
+                target="_blank" style="margin-left: 5px;"><i class="fab fa-github"></i> GitHub</a>
+        </p>
+    </footer>
+
+    <!-- Cinematic Overlay (Moved to root to avoid transform conflicts) -->
+    <div id="cinematicOverlay" class="cinematic-overlay" onclick="closeOverlay()">
+        <div class="overlay-content">
+            <!-- Cinematic Icon -->
+            <i class="fab fa-twitter overlay-logo"></i>
+            <h1 class="overlay-title">Tweet Depression Detection</h1>
+            <p class="overlay-author">Developed by</p>
+            <div class="overlay-names">
+                <span>Amey Thakur</span>
+                <span class="separator">&</span>
+                <span>Mega Satish</span>
+            </div>
+        </div>
+    </div>
+
+</body>
+
+</html>

source_code/templates/result.html

@@ -0,0 +1,90 @@
+<!-- 
+==============================================================================
+PROJECT: DEPRESSION-DETECTION-USING-TWEETS
+AUTHORS: AMEY THAKUR & MEGA SATISH
+GITHUB (AMEY): https://github.com/Amey-Thakur
+GITHUB (MEGA): https://github.com/msatmod
+REPOSITORY: https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS
+RELEASE DATE: June 5, 2022
+LICENSE: MIT License
+DESCRIPTION: Result page for the tweet analysis.
+==============================================================================
+-->
+
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Analysis Result | AMEY & MEGA </title>
+    <link rel="icon" href="{{url_for('.static', filename='chart-bar.svg')}}" type="image/svg+xml">
+
+    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
+    <link rel="stylesheet" href="{{url_for('.static', filename='styles.css')}}">
+    <script src="{{url_for('.static', filename='security.js')}}"></script>
+</head>
+
+<body>
+
+    <header class="hero-section container animate-fade">
+        <i class="fas fa-chart-line fa-3x" style="color: #1daeff; margin-bottom: 20px;"></i>
+        <h1>Analysis Result</h1>
+        <p>Here is what our model predicted for the tweet you provided.</p>
+    </header>
+
+    <main class="container animate-fade" style="animation-delay: 0.2s;">
+        <section class="analysis-card">
+            <h3><i class="fas fa-quote-left" style="color: var(--clr-slate); margin-right: 10px;"></i> The Tweet
+            </h3>
+            <div
+                style="background: #F8F9FA; padding: 25px; border-radius: 8px; margin: 20px 0; border-left: 4px solid var(--clr-slate);">
+                <p
+                    style="font-family: var(--font-serif); font-style: italic; font-size: 1.1rem; color: var(--clr-navy);">
+                    {{ name }}
+                </p>
+            </div>
+        </section>
+
+        <section class="analysis-card result-display">
+            <h3>Our Prediction</h3>
+
+            {% if prediction == 0 %}
+            <div class="result-badge badge-non-depressive">
+                <i class="fas fa-check-circle"></i> Non-Depressive
+            </div>
+            <p style="margin-top: 25px; color: var(--clr-slate);">Our model didn't find any significant signs of
+                depression in this text.</p>
+            {% elif prediction == 1 %}
+            <div class="result-badge badge-depressive">
+                <i class="fas fa-exclamation-triangle"></i> Depressive
+            </div>
+            <p style="margin-top: 25px; color: var(--clr-slate);">Our model identified patterns that are often
+                associated with depression in this text.</p>
+            {% endif %}
+
+            <div style="margin-top: 40px; border-top: 1px solid #E2E8F0; padding-top: 30px;">
+                <a href="{{ url_for('index') }}" class="btn-primary">
+                    <i class="fab fa-twitter" style="margin-right: 8px;"></i> Analyze Another Tweet
+                </a>
+            </div>
+        </section>
+    </main>
+
+    <footer class="container"
+        style="padding: 60px 0 40px; text-align: center; border-top: 1px solid rgba(136, 146, 176, 0.1);">
+        <p style="color: var(--clr-slate); font-size: 0.95rem; margin-bottom: 12px;">
+            Developed by <a href="https://github.com/Amey-Thakur"
+                style="color: var(--clr-accent-dark); font-weight: 700;">Amey Thakur</a> & <a
+                href="https://github.com/msatmod" style="color: var(--clr-accent-dark); font-weight: 700;">Mega
+                Satish</a>
+        </p>
+        <p style="color: var(--clr-light-slate); font-size: 0.85rem; letter-spacing: 0.5px;">
+            © 2022 • MIT License • <a href="https://github.com/Amey-Thakur/DEPRESSION-DETECTION-USING-TWEETS"
+                target="_blank" style="margin-left: 5px;"><i class="fab fa-github"></i> GitHub</a>
+        </p>
+    </footer>
+
+</body>
+
+</html>

source_code/test_app.py

@@ -0,0 +1,20 @@
+import requests
+
+def test_app():
+    url = "http://127.0.0.1:5000/predict"
+    data = {"tweet": "I feel very sad and hopeless today"}
+    try:
+        response = requests.post(url, data=data)
+        if response.status_code == 200:
+            print("Successfully connected to the server.")
+            if "Outcome: Depressive" in response.text:
+                print("Prediction test PASSED: Correctly identified depressive sentiment.")
+            else:
+                print("Prediction test FAILED: Outcome not found in response.")
+        else:
+            print(f"Server returned status code: {response.status_code}")
+    except Exception as e:
+        print(f"Error connecting to server: {e}")
+
+if __name__ == "__main__":
+    test_app()