| # 🚀 Predicting Viral News: A Data Science Pipeline | |
| **Author:** Matan Kriel | |
| ## 📌 Project Overview | |
| Predicting which articles will go "viral" is the holy grail for publishers. This project builds a complete end-to-end Data Science pipeline to analyze the **Online News Popularity** dataset. | |
| The goal was to transform raw data into actionable insights by: | |
| 1. **Engineering Features:** Creating a custom "Article Vibe" feature using Clustering. | |
| 2. **Regression Analysis:** Attempting to predict the exact share count. | |
| 3. **Classification Analysis:** Successfully predicting if an article will be a "Hit" (>1400 shares) or a "Flop". | |
| --- | |
| ## 🛠️ The Dataset | |
| * **Source:** UCI Machine Learning Repository (Online News Popularity). | |
| * **Size:** ~39,000 articles. | |
| * **Features:** 61 columns (Content, Sentiment, Time, Keywords). | |
| * **Target:** `shares` (Number of social media shares). | |
| --- | |
| ## 🧹 Phase 1: Data Handling & EDA | |
| We began by cleaning the dataset (stripping whitespace from columns, removing duplicates) and performing a **Time-Based Split**. Since the goal is to predict *future* performance, we sorted data by date (`timedelta`) to prevent "data leakage" from the future into the training set. | |
| ### Correlation Analysis | |
| We analyzed the relationship between content features (images, links, sentiment) and the target variable. | |
|  | |
| *Above: Correlation Heatmap showing feature relationships.* | |
| **📉 Insight:** As seen in the heatmap, the linear correlation between individual features (like `n_tokens_content` or `num_imgs`) and `shares` is extremely low (max ~0.06). This suggests that virality is **non-linear** and complex, justifying the need for advanced tree-based models over simple linear regression. | |
| --- | |
| ## 📊 Phase 2: Regression Model Strategy | |
| To tackle the difficult task of predicting exact share counts, we designed a rigorous comparison of three distinct regression algorithms. This allowed us to establish a baseline before attempting complex feature engineering. | |
| **The 3 Models Compared:** | |
| 1. **Linear Regression (Baseline):** A simple linear model to establish the minimum performance benchmark. | |
| 2. **Random Forest Regressor:** Selected for its ability to handle non-linear relationships and interactions between features (e.g., *Sentiment* vs. *Subjectivity*). | |
| 3. **Gradient Boosting Regressor:** Selected as the "Challenger" model, known for high precision in Kaggle-style competitions. | |
|  | |
| *Results of this comparison are detailed in Phase 4.* | |
| --- | |
| ## 🧪 Phase 3: Feature Engineering (Clustering) | |
| To capture the subtle "tone" of an article—which raw numbers often miss—we engineered a new feature called `cluster_vibe`. We used **K-Means Clustering** to group articles based on two dimensions: | |
| 1. **Sentiment:** (Positive vs. Negative) | |
| 2. **Subjectivity:** (Opinion vs. Fact) | |
| ### Choosing the Optimal 'k' | |
| We used the Elbow Method and Silhouette Analysis to determine the best number of clusters. | |
|  | |
| *Above: Elbow Method (Left) and Silhouette Score (Right).* | |
| **Decisions & Logic:** | |
| * **The Elbow:** We observed a distinct "bend" in the WCSS curve at **k=4**. | |
| * **Silhouette Score:** While k=2 had a higher score, it was too broad (just Pos/Neg). k=4 maintained a strong score (~0.32) while providing necessary granularity (e.g., *Positive-Opinion, Neutral-Fact, etc.*). | |
| * **Action:** We assigned every article to one of these 4 clusters and added it as a categorical feature to improve our models. | |
| --- | |
| ## 📉 Phase 4: Regression Results | |
| With our features engineered, we evaluated the three models defined in Phase 2 using **RMSE** (Root Mean Squared Error) and **R2 Score**. | |
| **🏆 Result:** | |
| * All models struggled to predict the exact number (Low R2 scores across the board). | |
| * **Gradient Boosting** performed best, minimizing the error more than the Linear Baseline. | |
| * **Pivot Decision:** We concluded that predicting the *exact* share count is inherently noisy due to massive viral outliers. We decided to pivot to **Classification** to solve a more actionable business problem: *"Will this be popular or not?"* | |
|  | |
| --- | |
| ## 🚀 Phase 5: Classification Analysis (The Solution) | |
| **Goal:** Classify articles as **Viral** (1) or **Not Viral** (0). | |
| **Threshold:** Median split (>1400 shares). | |
| We repeated the comparison process, pitting **Logistic Regression** (Baseline) against **Random Forest** and **Gradient Boosting**. | |
| ### Model Showdown | |
|  | |
| *Above: ROC Curves comparing the 3 models.* | |
| **🏆 Result:** | |
| * **Gradient Boosting** was the clear winner with an **AUC of ~0.75**. | |
| * It significantly outperformed the Baseline (AUC 0.64), proving the model successfully learned complex non-linear patterns. | |
| ### What Drives Virality? (Interpretation) | |
| We analyzed which features the model found most important. | |
|  | |
| **💡 Key Insights:** | |
| * **#1 Predictor (kw_avg_avg):** The historical performance of keywords is the strongest predictor. If a topic was popular in the past, it is likely to be popular again. This suggests a "Caching Effect" in audience interest. | |
| * **Content vs. Context:** Structural features (like `is_weekend` or `num_imgs`) mattered less than the specific keywords used. | |
| * **Cluster Vibe:** While our engineered cluster feature helped group articles, historical metrics overpowered it in the final decision trees. | |
| --- | |
| ## ⚖️ Phase 6: Final Evaluation | |
| We ran the winning **Gradient Boosting Classifier** on the **Test Set** (the "Future" data held out from the start). | |
| * **Final AUC:** ~0.75 | |
| * **Conclusion:** The model is robust and generalizes well to unseen data. It is ready for deployment. | |
|  | |
| --- | |
| ## 🎮 Bonus: The Viral-O-Meter | |
| To demonstrate the model's utility, we built an interactive **Gradio Dashboard** embedded in the notebook. This allows non-technical stakeholders (e.g., editors) to input article metrics and receive a real-time prediction on whether their draft will go viral. | |
| --- | |
| ## 📂 Files in this Repo | |
| * `notebook.ipynb`: The complete Python code for the pipeline. | |
| * `gradient_boosting_viral_predictor.joblib`: The saved final model. | |
| * `README.md`: Project documentation. | |
| Video Link: https://youtu.be/Al665qltkDg | |
| --- | |
| license: mit | |
| --- | |