README.md

---
tags:
  - regression
  - classification
  - clustering
  - tabular
  - linkedin
  - job-postings
  - sklearn
license: mit
---

# 📊 LinkedIn Job Posting Engagement Analysis

> **Which LinkedIn job posting characteristics predict candidate engagement (views) — and how well can engagement be predicted or classified using only posting-level features?**

**Personal motivation:** As someone in entrepreneurship, understanding which job posting features attract candidates is directly relevant to future hiring decisions.

---

## 📹 Presentation Video

<video src="https://huggingface.co/datasets/YOUR_USERNAME/YOUR_REPO/resolve/main/presentation.mp4" controls style="max-width:720px;"></video>

---

## 📋 Dataset at a Glance

| Property | Value |
|---|---|
| **Source** | [LinkedIn Job Postings — arshkon/linkedin-job-postings (Kaggle)](https://www.kaggle.com/datasets/arshkon/linkedin-job-postings) |
| **Original size** | 123,850 rows × 49 columns |
| **Working sample** | 30,000 rows · `random_state=42` |
| **After join with companies** | 30,000 rows × 40 columns |
| **After cleaning** | 29,572 rows × 51 columns (in df_model) |
| **Train / Test split** | 23,657 / 5,915 (80/20, `random_state=42`) |
| **Regression target** | `log_views = log1p(views)` — log-transformed to handle right skew |
| **Classification target** | `high_engagement` — top 25% of training views (threshold from training only) |

---

## ⚠️ Scope & Limitations

> LinkedIn's algorithm, sponsored status, and company follower counts drive the **majority of view variance** and are **unobservable** in this dataset. Models use posting-level features only. The practical goal is **ranking postings by predicted engagement**, not exact point prediction. Results show associations, not causal relationships.

---

## 🗂️ Repository Files

| File | Description |
|---|---|
| `notebook.ipynb` | Full pipeline: Cleaning → EDA → Features → Clustering → Regression → Classification → Bonus |
| `linkedin_regression_model.pkl` | Winning model: Random Forest (Tuned) |
| `linkedin_classification_model.pkl` | Winning model: Decision Tree |
| `regression_model_results.csv` | Full regression model comparison |
| `classification_model_results.csv` | Full classification model comparison |

---

## 🧹 Data Cleaning Pipeline

```
Step 1 — Reproducible sampling
        123,850 rows → sample(n=30,000, random_state=42)
        Joined with companies.csv on company_id (left join, rows preserved)
        Result: 30,000 rows × 40 columns

Step 2 — Duplicate & missing target removal
        Removed duplicate rows
        Dropped rows where views is NaN or negative
        Result: 29,572 usable rows

Step 3 — Date parsing
        listed_time, original_listed_time, expiry, closed_time → parsed to datetime
        Extracted: posting_year, posting_month, posting_dayofweek, posting_weekend

Step 4 — Missing value analysis & column dropping
        Threshold: >70% missing → drop
        Dropped: closed_time (99.2%), skills_desc (98.1%), med_salary (95.1%),
                 remote_allowed (87.9%), applies (81.1%), max_salary/min_salary (76%)
        Protected columns: salary fields kept for feature engineering

Step 5 — Leakage columns excluded
        expiry, applies → removed (post-publication outcomes)
        views → kept as target only, not as feature

Step 6 — Salary imputation strategy
        has_salary_info = 1 if salary present, else 0
        salary_midpoint computed from min/max salary where available
        Missing salary → imputed inside sklearn Pipeline on training data only

Step 7 — Log transformation of target
        Raw views: mean=14.9, std=98.8, max=9,949 — heavily right-skewed
        log_views = log1p(views) — compresses scale, improves regression fit
        Predictions converted back via expm1() for interpretation
        Outliers (IQR method): 4,074 outliers (13.8%) — kept, not removed
```

---

## 🔍 EDA — 5 Questions + Correlation Heatmap

**Note:** EDA question numbers in the notebook differ from intuitive order. Q1=Work type, Q2=Salary, Q3=Description, Q4=Day of week, Q5=Seniority. Presented here in order of impact.

### Salary Transparency vs Views (Notebook Q2)

```
No salary info   ████████████░░░░░░░░░░░░░  ~12 avg views   (70.1% of postings)
Has salary info  ████████████████████████░  ~21 avg views   (29.9% of postings)
                                             +74.3% lift ✓
```

> Only 8,562 of 29,572 postings (29.9%) disclose salary. **74.3% more views** for transparent postings. Highest-leverage, lowest-cost recruiter action.

---

### Description Length vs Views (Notebook Q3)

```
< 100 words    ██████░░░░░░░░░░░░░░  low    — signals incomplete posting
100–250 words  █████████░░░░░░░░░░░  medium
250–500 words  ████████████████████  PEAK ★ — sweet spot
500–750 words  ████████████████░░░░  high
> 1000 words   ███████░░░░░░░░░░░░░  drop-off — overwhelms candidates
```

> Non-linear relationship confirmed. Sweet spot: **250–500 words**. Motivated `description_density` — the #1 feature in the winning regression model.

---

### Day of Week vs Views (Notebook Q4)

```
Monday    ████████████████████  39 avg views  ★ best day (n=1,837)
Tuesday   █████████████████░░░  (weekday)
Wednesday ████████████████░░░░  (weekday)
Thursday  ███████████████░░░░░  (weekday)
Friday    ███░░░░░░░░░░░░░░░░░   7 avg views  ✗ worst day (n=10,076)
Saturday  ████████████░░░░░░░░  (weekend — noisier, n=2,116 total)
Sunday    ████████████░░░░░░░░  (weekend — noisier)

Weekend average: 28 views vs Weekday average: 22 views
Note: Weekend sample is much smaller (2,116 total) — estimates are noisier.
Weekday postings averaged 21.8% LOWER views than weekend in this dataset.
```

> **Counterintuitive finding:** Weekend postings showed higher average views than weekdays in this sample, BUT weekend volume is very small (2,116 obs) making these estimates unreliable. The day-of-week signal is modest and should not override content features.

---

### Work Type vs Views (Notebook Q1)

```
Contract    ████████████████████  29.97 avg views  7.0 median
Internship  █████████████████░░░  25.71 avg views  5.0 median
Full-time   ████████░░░░░░░░░░░░  13.70 avg views  4.0 median
Other       ███████░░░░░░░░░░░░░  11.27 avg views  4.0 median
Part-time   ██████░░░░░░░░░░░░░░   9.59 avg views  4.0 median
```

> Contract and Internship roles show the highest engagement. However, Full-time dominates volume (23,674 of 29,572 postings). Work type is a useful feature but should not be interpreted as causal.

---

### Seniority Level vs Views (Notebook Q5)

```
Entry-level  ████████████████████  18 avg views  n=792
Senior-level ████████████░░░░░░░░  16 avg views  n=3,577
Other/Mid    ██████████░░░░░░░░░░  15 avg views  n=25,203

Entry vs Senior: +12.4% more views
Entry vs Other:  +18.9% more views
```

> Supply-side effect — more candidates qualify for junior roles so the pool is larger. Entry-level advantage is modest (+12.4% vs senior). `is_entry_role` carries predictive signal because it proxies for candidate pool size.

---

### 🔥 Feature Correlation with log(views+1)

```
Feature                      Corr    Direction   Note
─────────────────────────────────────────────────────────────────────
desc_salary_interaction      +0.18   ↑ views     strongest predictor
has_salary_info              +0.14   ↑ views     salary transparency
salary_log                   +0.12   ↑ views     salary level
description_density          +0.10   ↑ views     content quality
description_word_count       +0.08   ↑ views     description length
is_software_role             +0.08   ↑ views     tech role demand
is_data_role                 +0.07   ↑ views     data role demand
is_entry_role                +0.06   ↑ views     larger candidate pool
posting_weekend              -0.04   ↓ views     (small negative)
is_senior_role               -0.03   ↓ views     smaller candidate pool
─────────────────────────────────────────────────────────────────────
Internal correlations (structural):
salary_log ↔ salary_midpoint  +0.96  log transform of same variable
desc_wc ↔ desc_density        +0.55  density uses length in formula
is_software ↔ is_data         +0.35  often co-occur in job titles
is_senior ↔ is_entry          -0.28  mutually exclusive by construction
─────────────────────────────────────────────────────────────────────
```

> Most features show **weak linear correlation** — no single feature dominates. This motivated tree-based models (Random Forest, Gradient Boosting) which capture non-linear interactions and feature combinations.

---

## ⚙️ Feature Engineering — 20 base + 10 cluster = 30 Total Features

**Note:** The notebook creates 20 engineered features before clustering, then adds 6 cluster dummy columns for a total of 30 in the final feature matrix (X_train_fe shape: 23,657 × 30).

| Group | Features |
|---|---|
| Text length | `title_length`, `title_word_count`, `description_length`, `description_word_count` |
| Text structure | `description_density`, `title_desc_ratio` |
| Salary | `salary_midpoint`, `salary_range`, `has_salary_info`, `salary_log` |
| Role keywords | `is_senior_role`, `is_entry_role`, `is_software_role`, `is_data_role`, `is_manager_role`, `is_sales_role`, `is_marketing_role`, `is_remote_text` |
| Interactions | `desc_salary_interaction`, `senior_salary`, `weekend_remote`, `title_desc_word_interaction`, `salary_density_interaction`, `salary_description_interaction`, `title_density_interaction` |
| Clustering | `cluster_0`, `cluster_1`, `cluster_2`, `cluster_3`, `cluster_4`, `cluster_5` |

**Missing value strategy:**
- Columns with >70% missing → dropped (closed_time, skills_desc, med_salary, remote_allowed, applies, salary min/max, compensation fields)
- Salary → `has_salary_info` flag + `salary_midpoint` computed where possible; remaining salary NaN imputed inside sklearn Pipeline on training data only
- Remaining numeric → `SimpleImputer(strategy="median")` inside Pipeline

---

## 🔵 Clustering — KMeans k=6

**Clustering features used (12 total, leakage-checked):**
`title_word_count`, `description_word_count`, `salary_log`, `description_density`, `has_salary_info`, `is_senior_role`, `is_entry_role`, `is_software_role`, `is_data_role`, `is_manager_role`, `is_sales_role`, `is_marketing_role`

**Methods used to select k:**
1. Elbow method (inertia k=2–10) — inconclusive, no sharp elbow
2. K-Means silhouette scores on full training matrix
3. Cluster-size stability table (smallest/largest cluster per k)
4. Interactive K-Means widget (visualization aid only — uses sample)
5. Hierarchical clustering dendrogram (Ward linkage, 300 obs sample)
6. Agglomerative Clustering diagnostic comparison (k=2–10 on sample)

```
Chart 1 — Actual silhouette scores by k (full training matrix)

  k=2   ████████░░░░░░░░░░░░  0.198  smallest cluster: 6,830 (28.9%)
  k=3   █████████░░░░░░░░░░░  0.221  smallest cluster: 2,100 (8.9%)
  k=4   ████████████████░░░░  0.312  ← strong score BUT largest=72%
  k=5   ██████████░░░░░░░░░░  0.250  smallest: 526 (unstable)
  k=6   ████████████░░░░░░░░  0.290  ← SELECTED ★ smallest: 583 (2.5%)
  k=7   ████████████░░░░░░░░  0.286  singleton cluster appeared
  k=8   █████████████░░░░░░░  0.315  singleton cluster appeared
  k=9   █████████████░░░░░░░  0.314  singleton cluster appeared
  k=10  ██████████████░░░░░░  0.350  singleton cluster appeared

Why NOT k=10 (highest score): singleton cluster (1 observation)
Why NOT k=4 (strong score):   largest cluster = 72% of observations
Why k=6: no singletons, stable sizes, silhouette 0.290, interpretable profiles

Note: Elbow method was inconclusive (inertia 255,430 at k=2 → 98,508 at k=10,
no sharp elbow). Agglomerative diagnostic best at k=2 (score 0.467 on sample)
— too coarse. k=6 selected as practical compromise across all methods.

Chart 2 — Actual cluster sizes at k=6 (training set n=23,657)

  Cluster 0 — Manager-focused       ████████████  4,571  (19%)  is_manager_role=1.00
  Cluster 1 — General / Mixed       ████████████████████ 13,055 (55%)  no dominant role signal
  Cluster 2 — Salary-transparent    ████          1,940   (8%)  has_salary_info=1.00
  Cluster 3 — Data roles            ███           1,451   (6%)  is_data_role=1.00
  Cluster 4 — Software roles        █████         2,057   (9%)  is_software_role=1.00
  Cluster 5 — Entry / low salary    ██              583   (2%)  smallest cluster

Official final silhouette score: 0.290 (full training matrix)
```

Cluster labels one-hot encoded as 6 dummy features. Including clusters improved both regression RMSE and classification F1 over models without them.

---

## 📈 Regression — Predicting `log1p(views)`

### Baseline

```
Mean Baseline (predict training mean for all observations):
  RMSE_log = 0.8708   R² = -0.0002   ← floor every model must beat
  MAE_views ≈ 10.64

Baseline Linear Regression (20 features, no clustering):
  RMSE_log = 0.8425   R² = 0.0639
  MAE_views ≈ 10.54
```

### Full model comparison (after feature engineering + clustering)

```
Model                        RMSE_log ↓    R² ↑
─────────────────────────────────────────────────────
Random Forest (Tuned)  ★     0.8347        0.0811
Random Forest (Ctrl)         0.8349        0.0807
Gradient Boosting            0.8370        0.0770
Linear Regression + Feat     0.8420        0.0640
RidgeCV                      0.8420        0.0640
Lasso Regression             0.8430        0.0640
PCA + Linear Regression      0.8440        0.0600
Mean Baseline                0.8708       -0.0002
─────────────────────────────────────────────────────
Winner: RandomizedSearchCV tuned RF
Improvement over manually controlled RF: 0.0002 RMSE_log (practically negligible)
3-fold CV mean RMSE_log: 0.8747 (±0.0125) — stable across folds
Overfitting lesson: unrestricted RF → train R²=0.854, test R²=0.003
Fixed by: max_depth, min_samples_split, min_samples_leaf, max_features constraints
Outlier robustness test: capping views at 99th pct → RMSE_log 0.8147, R²=0.0812
```

### Top feature importances (RF Tuned)

```
description_density          ████████████  #1 — content quality
description_length           ██████████░░  #2 — raw description size
description_word_count       ████████░░░░  #3 — word count
title-description interaction████████░░░░  #4 — combined signal
is_software_role             ██████░░░░░░  #5 — tech role demand
is_data_role                 █████░░░░░░░  #6 — data role demand
salary_log / has_salary_info ████░░░░░░░░  #7+ — salary signals
```

> **Note:** desc_salary_interaction ranked #2 in SHAP analysis but further down in Gini importance. Both agree on description quality and salary as top drivers.

### Regression interpretation

```
R² = 0.081 → model explains ~8% of variance in log(views+1)

Why acceptable:
  ✓ Beats mean baseline (R²≈0) — real posting-level signal captured
  ✓ Social engagement inherently noisy — platform factors dominate
  ✓ 92% of variance from unobservable sources (algorithm, followers, ads)
  ✓ Practical use = ranking postings, not forecasting exact counts

PCA + Linear: reduced to 15 components (96.3% variance preserved) — no improvement
Gradient Boosting marginally worse than RF — non-linear models help but modestly
```

---

## 🟠 Classification — High Engagement vs. Normal

```
Target: high_engagement = 1 if views ≥ 75th percentile of TRAINING views
Class balance: ~75% Normal (Class 0) / ~25% High Engagement (Class 1)
Feature matrix: X_clf uses 24 features (not the full 30 — see notebook cell 207)
Training: ~24,000 obs | Test: ~6,000 obs
Metric: F1-score for Class 1 (accuracy misleading with 75/25 imbalance)
```

### Model comparison

```
Model                  F1 (C1)    Recall (C1)   Notes
──────────────────────────────────────────────────────────────
Decision Tree     ★    HIGHEST    HIGHEST       lowest FN count
Logistic Regr.         near-best  high          close to DT
Random Forest          moderate   lower         lowest FP count
Dummy Baseline         0.00       0.00          always predicts Class 0
──────────────────────────────────────────────────────────────
Winner: max_depth=8, class_weight="balanced"
5-fold CV F1: 0.4424 ± 0.0152 — stable, no lucky split
```

### Confusion matrix (all models — from notebook)

```
Decision Tree:     lowest FN (catches most high-engagement) — most false positives
Random Forest:     lowest FP (fewest false alarms) — misses most high-engagement
Logistic Regr.:    between the two — close to DT in F1

FN (missed high-engagement) = most costly error:
  Company fails to prioritize, promote, or learn from a valuable listing.
FP (false alarm) = also costly:
  Recruiters waste attention on postings that are not actually strong.
```

---

## 💡 Business Insights (from notebook cell 242)

1. **Salary transparency is associated with higher engagement** — 74.3% more views. Fewer than 30% of postings disclose salary today.
2. **Description structure matters** — density was the #1 feature in both models. Sweet spot: 250–500 words.
3. **Tech roles attract more engagement** — software and data role flags carry signal beyond salary.
4. **Work type is associated with engagement** — contract roles lead, but full-time dominates volume.
5. **Platform factors dominate** — R²≈0.08 is expected. Model value is in ranking, not exact prediction.

---

## 🎁 Bonus Work

### 🚀 Interactive Dashboard

👉 **[Open the LinkedIn Job Engagement Dashboard](https://huggingface.co/spaces/MichaelYitzchak/linkedin_Job_Engagement)**

| Tab | Description |
|---|---|
| 🎯 Engagement Predictor | Real-time predicted views + High/Normal classification |
| 📊 EDA Dashboard | All 5 EDA findings as interactive charts |
| ℹ️ About | Feature groups, model details, limitations |

### 🧠 SHAP Explainability

```
SHAP mean |value| — RF Tuned regression (test observations)

description_density      ████████████  strongest ↑
desc_salary_interaction  ██████████░░  salary × description synergy ↑
salary_log               ████████░░░░  salary level ↑
has_salary_info          ██████░░░░░░  disclosed → more views ↑
posting_weekend          ██░░░░░░░░░░  weekend → fewer views ↓

Key finding: desc_salary_interaction ranks #2 in SHAP but lower in Gini —
confirms it captures genuine non-linear interaction beyond individual features.
```

### 📊 Feature Importance: Regression vs Classification

```
                        Regression RF    Classification DT
description_density     #1               #2
desc_salary_interaction varies           varies
salary_log              #7+              varies
is_entry_role           lower            rises in classification
is_data_role            #6               varies
─────────────────────────────────────────────────────────
Agreement: description quality + salary dominate both models
Divergence: seniority/role flags matter more for threshold-crossing
            (classification) than for predicting exact counts (regression)
```

### 🔬 Additional Bonus Items

- **Interactive K-Means Widget** — explore different k values visually in notebook (cell 4.11)
- **Hierarchical Clustering Dendrogram** — Ward linkage, 300 obs sample (cell 4.12)
- **Agglomerative Clustering Diagnostic** — k=2–10 comparison (cell 4.13)
- **Outlier Robustness Test** — views capped at 99th percentile: RMSE_log 0.8147 vs 0.8347 uncapped
- **3-fold CV for regression** — mean RMSE_log 0.8747 ± 0.0125

---

## 🛠️ How to Use the Models

```python
import pickle, numpy as np

with open("linkedin_regression_model.pkl", "rb") as f:
    reg_model = pickle.load(f)
with open("linkedin_classification_model.pkl", "rb") as f:
    clf_model = pickle.load(f)

# Regression — predict log(views+1), convert back
log_views_pred = reg_model.predict(X_test_fe)
views_pred = np.expm1(log_views_pred)

# Classification — predict high-engagement label (0 or 1)
label = clf_model.predict(X_clf)
```

> Regression model expects 30-column X_test_fe (with cluster dummies). Classification model expects 24-column X_clf. Run the full pipeline in the notebook to produce compatible inputs.

---

*Assignment 2 — Classification, Regression, Clustering, Evaluation | LinkedIn Job Postings · arshkon/linkedin-job-postings (Kaggle)*