README.md

# 🇮🇹 Italian Tech Salaries — Salary Prediction

> **Assignment #2 | Classification, Regression, Clustering & Evaluation**
> **Course:** Data Science | Reichman University
> **Student:** Adi Prog | [adiprog14](https://huggingface.co/adiprog14)
> **Dataset:** [datapizza-ai-lab/salaries](https://huggingface.co/datasets/datapizza-ai-lab/salaries)

---

## 🎬 Presentation Video

<video src="https://huggingface.co/adiprog14/tech-salary-predictor/resolve/main/presentation.mp4" controls="controls" style="max-width: 720px;"></video>
 *Can't play the video? [Click here to watch](YOUR_VIDEO_LINK_HERE)*

---

## 📌 Project Overview

**Research Question:**
> *What are the strongest predictors of salary in the Italian tech market — and does AI adoption actually pay off?*

**Goal:** Build a machine learning pipeline to **predict the gross annual salary (RAL in EUR)** of Italian tech workers, using their professional profile, education, company details, and AI usage habits.

| Property | Value |
|---|---|
| Dataset | Datapizza Italian Tech Salaries |
| Source | [salaries.datapizza.tech](https://salaries.datapizza.tech) |
| Rows | ~26,000 anonymous survey responses |
| Features | 20 (numeric + categorical + list-based) |
| Target | `salary` — gross annual salary in EUR |
| License | CC-BY-NC-4.0 |

---

## 📁 Repository Files

| File | Description |
|---|---|
| `README.md` | Full project documentation (this file) |
| `Assignment2_Salaries.ipynb` | Complete Python notebook — all code, outputs, and analysis |
| `regression_model.pkl` | Winning regression model (Gradient Boosting) |
| `classification_model.pkl` | Winning classification model (Gradient Boosting) |
| `presentation.mp4` | 4–6 minute video walkthrough |

---

## 🔍 Part 1 — Dataset

The **Datapizza Salaries** dataset is a crowd-sourced anonymous survey of tech workers across Italy, collected and updated weekly by [Datapizza](https://datapizza.tech).

**Feature breakdown:**

| Type | Features |
|---|---|
| Numeric | `monthsOfExperience`, `age`, `aiUsageFrequency`, `salary` |
| Categorical | `jobTitle`, `educationType`, `companyIndustry`, `companySize`, `province`, `workMode`, `gender`, `editor` |
| List-based | `technologies`, `aiTechnologies`, `aiTaskTypes`, `aiUpdateSources`, `admiredTechnologies` |

> **Note:** Submissions before November 2024 contain only partial data — fields like `educationType`, `age`, and `aiUsageFrequency` were added in a later survey version. This is handled during cleaning.

---

## 📊 Part 2 — Exploratory Data Analysis

### 2.1 Data Cleaning

| Step | Action | Reason |
|---|---|---|
| Invalid rows | Kept only `valid == True` | Remove incomplete/test submissions |
| Salary outliers | Kept 5,000–300,000 EUR | Remove clearly erroneous entries |
| Age outliers | Kept 16–70 years | Remove non-realistic working ages |
| Experience cap | Capped at 480 months (40 years) | Remove data entry errors |
| Duplicates | Removed exact duplicate rows | List columns required special string-conversion handling |
| Dropped columns | `valid`, `submittedAt`, `rating` | Not predictive for salary |

> **Technical note:** List-type columns (`technologies`, `aiTechnologies`, etc.) cannot be directly hashed by pandas. Duplicates were detected by converting list columns to sorted strings before comparison — then the mask was applied to the original dataframe.

---

### 2.2 Descriptive Statistics

| Metric | salary (EUR) | Experience (months) | Age | AI Usage (1–5) |
|---|---|---|---|---|
| Mean | ~44,000 | ~72 | ~33 | ~3.2 |
| Median | ~40,000 | ~60 | ~32 | ~3.0 |
| Min | 5,000 | 1 | 16 | 1 |
| Max | 300,000 | 480 | 70 | 5 |

---

### 2.3 Research Questions & Visualizations

**Q1: What does the salary distribution look like?**

![Salary Distribution](salary_distribution.png)

The salary distribution is heavily **right-skewed** — most workers earn between 25k–60k EUR/year, with a long tail of high earners. The log-transformed version reveals a near-normal shape, suggesting a log-transform could help linear models.

**Key insight:** The majority of Italian tech workers earn under 60k EUR. Very high salaries (>100k) exist but are rare, mostly concentrated in senior leadership or specialized AI/cloud roles.

---

**Q2: Which job titles earn the most?**

![Salary by Job Title](salary_by_jobtitle.png)

Roles commanding the highest median salaries include `solutions_architect`, `it_manager`, and `machine_learning_engineer`. At the bottom sit `data_analyst`, `junior_developer`, and `intern`.

**Key insight:** Architecture and management roles consistently outperform pure coding roles — the scope of responsibility matters more than the technical stack alone.

---

**Q3: Does experience correlate with salary?**

![Experience vs Salary](experience_vs_salary.png)

The Pearson correlation between `monthsOfExperience` and `salary` is approximately **0.35** — a moderate positive relationship. The high variance in the scatter plot shows that experience alone does not determine salary.

**Key insight:** Experience is necessary but not sufficient. Two workers with the same seniority can earn very different amounts depending on role, company size, and location.

---

**Q4: Does work mode affect salary?**

![Salary by Work Mode](salary_by_workmode.png)

| Work Mode | Median Salary |
|---|---|
| Remote | Highest |
| Hybrid | Middle |
| Onsite | Lowest |

**Key insight:** Remote workers earn a clear premium. This likely reflects that remote roles attract senior or specialized talent, and that companies hiring remotely compete in a national or international talent market.

---

**Q5: Does education level impact salary?**

![Salary by Education](salary_by_education.png)

A clear step-up pattern exists across education levels. The gap between High School and Master's degree is approximately **10,000 EUR/year** in median salary. The Master's-to-PhD jump is smaller, since many senior tech roles don't require a PhD.

**Key insight:** Education pays off in the Italian tech market, with each level adding measurable salary uplift.

---

**Q6: Does AI usage frequency correlate with salary?**

![AI Usage vs Salary](ai_usage_vs_salary.png)

Workers who report higher AI tool usage (scale 1–5) tend to earn more, with a consistent upward trend.

**Key insight:** AI adoption correlates with higher salary — but this is likely a **proxy for role sophistication**. Roles like ML Engineer naturally involve more AI usage and also pay more. The causality is indirect.

---

**Q7: Which company sizes pay best?**

![Salary by Company Size](salary_by_companysize.png)

Mid-to-large companies (201–5000 employees) pay the most. Very small companies (1–10) pay the least, likely reflecting startup equity-over-salary trade-offs not visible in this data.

---

**Q8: Correlation Heatmap**

![Correlation Heatmap](correlation_heatmap.png)

Among numeric features, `monthsOfExperience` has the strongest correlation with `salary`. `age` and `aiUsageFrequency` show weaker but positive associations.

---

### 2.4 EDA Summary

| Finding | Implication for Modeling |
|---|---|
| Salary is right-skewed | Log-transform may help linear models |
| Experience → salary (~0.35 correlation) | Strong baseline numeric feature |
| Remote work premium exists | `workMode` is a valuable feature |
| Education shows stepwise effect | Ordinal encoding is appropriate |
| AI usage → salary | Worth engineering as count/binary features |
| Larger companies → higher pay | Ordinal company size encoding useful |
| List columns contain rich information | Need count-based feature extraction |

---

## ⚙️ Part 3 — Baseline Regression Model

### Setup

Before any feature engineering, we established a benchmark using the simplest available features:

- **Features:** `monthsOfExperience` + one-hot encoded `jobTitle`, `educationType`, `companySize`, `workMode`, `companyIndustry`, `gender`
- **Missing categoricals** → filled with `'unknown'`
- **Missing numeric** → filled with column median
- **Train/Test split:** 80% / 20%, `random_state=42`
- **Model:** `LinearRegression()` — default scikit-learn parameters, no tuning

### Baseline Results

| Metric | Value |
|---|---|
| MAE | 7,638 EUR |
| RMSE | 11,953 EUR |
| R² | 0.3586 |

![Baseline Evaluation](baseline_evaluation.png)

The **Actual vs Predicted** scatter shows significant spread around the perfect-prediction diagonal. The model struggles with extreme salaries — it tends to overestimate low earners and underestimate high earners. The **residuals** are roughly centered at zero but with heavy tails, confirming that linear assumptions miss the non-linear patterns in salary data.

### Feature Importance — Coefficients

![Baseline Coefficients](baseline_coefficients.png)

The largest positive coefficients come from senior job titles and remote work mode. The largest negative coefficients come from junior roles and high school education. This is directionally correct and validates the EDA findings.

### Baseline Limitations

| Limitation | Impact |
|---|---|
| Ignores list features (technologies, AI tools) | Misses important skill-based signals |
| Assumes linear relationships | Salary is non-linear in reality |
| Simple missing value imputation | May introduce bias |
| Cannot capture feature interactions | e.g. senior + remote + big company → very high salary |

> These limitations directly motivate the feature engineering work in Part 4.

---

## 🛠️ Part 4 — Feature Engineering

Feature engineering is the most impactful step in the pipeline. We built 14 new features from the raw data, organized into four categories.

### 4.1 Numeric Transformations

| Feature | Formula | Rationale |
|---|---|---|
| `years_of_experience` | `months / 12` | More interpretable unit |
| `is_senior` | `1 if years >= 5 else 0` | Captures the junior/senior threshold effect |
| `experience_squared` | `years²` | Models diminishing salary returns at high seniority |

### 4.2 List-Based Features

The dataset contains list columns (e.g. `technologies = ["Python", "AWS", "SQL"]`). We extracted count-based and binary signals from them:

| Feature | Description |
|---|---|
| `num_technologies` | Count of technologies used |
| `num_ai_tools` | Count of AI tools used |
| `num_ai_tasks` | Count of task types where AI is applied |
| `num_ai_sources` | Count of AI news sources followed |
| `uses_premium_ai` | Binary: uses Claude / Cursor / GitHub Copilot / Claude Code |

> **Why `uses_premium_ai`?** These tools are associated with more senior/specialized developers and higher-paying roles.

### 4.3 Ordinal Encodings

Rather than one-hot encoding ordered categories (which loses the ordering), we mapped them to integers:

| Feature | Mapping |
|---|---|
| `companySize_ord` | 1-10→1, 11-50→2, 51-200→3, ... 5001+→7 |
| `education_ord` | High School→1, Bachelor→2, Master→3, PhD→4 |
| `workMode_ord` | onsite→1, hybrid→2, remote→3 |
| `is_remote` | Binary flag for remote work |

### 4.4 Clustering — K-Means

We applied K-Means on the core numeric features to discover natural worker segments, then used the results as additional model features.

**Elbow Method — choosing k:**

![Elbow Method](elbow_method.png)

The inertia curve bends noticeably around **k=4**, selecting 4 clusters as the optimal number.

**Cluster Visualization (PCA 2D):**

![Cluster PCA](cluster_pca.png)

The PCA projection confirms reasonably separated clusters — K-Means found meaningful structure. The two principal components capture a significant portion of variance in the worker feature space.

**Cluster Salary Profiles:**

![Cluster Salary](cluster_salary.png)

| Cluster | Likely Profile | Salary Level |
|---|---|---|
| 0 | Junior generalists — low experience, broad tech stack | Lowest |
| 1 | Senior specialists — high experience, focused stack | Highest |
| 2 | AI-heavy roles — high AI usage, often remote | Above average |
| 3 | Mid-level hybrid — moderate experience, standard roles | Middle |

**Cluster features added:**
- `cluster_id` — which natural segment the worker belongs to
- `dist_to_centroid` — distance to the cluster center (measures how "typical" the worker is)

### 4.5 Final Feature Set Summary

| Feature | Type | Origin |
|---|---|---|
| `years_of_experience` | Numeric | Derived |
| `is_senior` | Binary | Derived |
| `experience_squared` | Numeric | Derived |
| `num_technologies` | Numeric | List extraction |
| `num_ai_tools` | Numeric | List extraction |
| `num_ai_tasks` | Numeric | List extraction |
| `num_ai_sources` | Numeric | List extraction |
| `uses_premium_ai` | Binary | List extraction |
| `companySize_ord` | Ordinal | Encoded |
| `education_ord` | Ordinal | Encoded |
| `workMode_ord` | Ordinal | Encoded |
| `is_remote` | Binary | Encoded |
| `cluster_id` | Categorical | K-Means |
| `dist_to_centroid` | Numeric | K-Means |
| `jobTitle_*` | One-hot | Encoded |
| `companyIndustry_*` | One-hot | Encoded |
| `gender_*` | One-hot | Encoded |

---

## 🤖 Part 5 — Three Improved Regression Models

Using the full engineered feature set, we trained and compared three models.

**Setup:**
- Train/Test split: 80/20, `random_state=42`
- Scaling: StandardScaler applied for Linear Regression; tree models use raw features

### Model 1 — Linear Regression (Engineered Features)

Same algorithm as the baseline, now with all engineered features. This isolates the **impact of feature engineering alone** on linear model performance.

### Model 2 — Random Forest Regressor

- `n_estimators=200`, `random_state=42`
- Ensemble of 200 decision trees — handles non-linearity and feature interactions
- Robust to remaining salary outliers

### Model 3 — Gradient Boosting Regressor

- `n_estimators=200`, `learning_rate=0.1`, `random_state=42`
- Trees built sequentially — each one corrects the errors of the previous
- Generally achieves the best bias-variance trade-off on structured tabular data

### Results Comparison

![Model Comparison Regression](model_comparison_regression.png)

| Model | MAE (EUR) | RMSE (EUR) | R² |
|---|---|---|---|
| Baseline LR | 7,638 | 11,953 | 0.3586 |
| LR (engineered) | 7,381 | 11,670 | 0.3886 |
| Random Forest | 7,797 | 12,412 | 0.3084 |
| **Gradient Boosting** | **7,306** | **11,668** | **0.3888** |

### Feature Importance (Gradient Boosting)

![Feature Importance](feature_importance.png)

The top predictors according to Gradient Boosting:
- `years_of_experience` and `experience_squared` — seniority dominates
- Job title dummies (senior roles) — role type is the second biggest driver
- `cluster_id` — the K-Means cluster assignment adds real predictive value
- `companySize_ord` — larger companies pay more
- `is_remote` / `workMode_ord` — the remote premium is captured

> **Notably, `cluster_id` and `dist_to_centroid` appear in the top importances**, validating that the unsupervised clustering step added genuine signal to the supervised model.

### 🏆 Winning Regression Model: Gradient Boosting Regressor

| Reason | Explanation |
|---|---|
| Non-linearity | Salary relationships are non-linear — trees handle this naturally |
| Sequential correction | Each boosting round reduces remaining errors |
| Feature interactions | Senior + remote + large company → captured automatically |
| Outlier robustness | Less sensitive than Linear Regression to extreme salaries |

---

## 🎯 Part 7 — Regression to Classification

### Strategy: Quantile Binning (3 Classes)

The continuous `salary` target was converted into 3 tiers using quantile thresholds:

| Class | Label | Salary Range |
|---|---|---|
| 0 | Low | Bottom 33% |
| 1 | Mid | Middle 33% |
| 2 | High | Top 33% |

**Why quantile binning?**
Quantile binning guarantees **balanced classes** — approximately equal representation in each tier. This avoids the class imbalance problem that fixed-threshold splits can create.

### Class Distribution

![Class Distribution](class_distribution.png)

The three classes are nearly equal in size (~33% each), confirming balanced classes. No oversampling or class weighting is required.

---

## 🔎 Part 8 — Three Classification Models

### Precision vs Recall

In this salary classification task, **recall matters more**, specifically for the **High** salary class.

A **False Negative** — predicting Low/Mid when the worker is actually High — is more costly than a False Positive:
- An HR tool that misclassifies a high-earner as mid-level → under-compensation → talent loss
- A recruiting system that misses high-salary candidates → misallocated budget

> **False Negatives are the more critical error** in this domain.

### Model 1 — Logistic Regression

- Trained on StandardScaler-normalized features
- Linear decision boundaries between salary classes
- Fast and interpretable, but constrained by linearity

### Model 2 — Random Forest Classifier

- `n_estimators=200`, `random_state=42`
- Handles non-linear class boundaries
- Ensemble reduces variance compared to a single tree

### Model 3 — Gradient Boosting Classifier

- `n_estimators=200`, `learning_rate=0.1`, `random_state=42`
- Focuses on misclassified samples in each sequential round
- Best recall on the High salary class

### Confusion Matrices

![Confusion Matrix](confusion_matrix.png)

Observations from the confusion matrices:
- **Most errors occur at the Mid class boundaries** — this is expected, since Mid salaries overlap with both Low and High at the edges
- **Gradient Boosting** makes the fewest off-diagonal mistakes overall
- **Logistic Regression** struggles most with the Mid class due to linear boundaries
- **High class recall** is best in Gradient Boosting — matching our stated priority

### Results Comparison

| Model | Accuracy | F1 Macro |
|---|---|---|
| Logistic Regression | 0.6241 | 0.6224 |
| Random Forest | 0.6007 | 0.5980 |
| **Gradient Boosting** | **0.6241** | **0.6221** |

### 🏆 Winning Classification Model: Gradient Boosting Classifier

| Reason | Explanation |
|---|---|
| Best F1 Macro | Performs well across all three salary tiers |
| Best High-class recall | Aligns with our priority metric |
| Non-linear boundaries | Captures complex salary tier separations |
| Sequential boosting | Reduces bias vs Random Forest |

---

## 💡 Key Takeaways & Lessons Learned

### What worked well

- **Feature engineering had the biggest single impact** — especially ordinal encodings and list-count features. The improvement from baseline LR to engineered LR shows this clearly.
- **K-Means clustering added genuine predictive value** — `cluster_id` appeared in the top feature importances, validating the unsupervised step.
- **Gradient Boosting won on both tasks** — regression and classification — which is consistent with its known strength on structured tabular data.
- **Quantile binning** produced balanced classes without any oversampling tricks.

### What was challenging

- **List-type columns required custom handling** throughout — from duplicate detection (hashing issue) to feature extraction (no direct encoding possible).
- **High missing rates** in AI-related columns meant many rows had partial data — imputation choices affected model quality.
- **Salary variance is genuinely high** — even with good features, salary prediction has inherent noise due to individual negotiation and company-specific pay bands not captured in the survey.

### Domain insights

- **Experience is the strongest predictor**, but role and company size matter almost as much
- **Remote work pays a real premium** — consistent across all analyses
- **AI adoption correlates with salary** — a signal of role sophistication, not a direct cause
- **The Italian tech market shows significant salary disparity** by region (Milano dominates), which this model does not fully exploit since `province` was dropped due to high cardinality

---

## 🚀 How to Use the Models

```python
import pickle
import pandas as pd
import numpy as np

# Load regression model
with open('regression_model.pkl', 'rb') as f:
    reg_model = pickle.load(f)

# Load classification model
with open('classification_model.pkl', 'rb') as f:
    clf_model = pickle.load(f)

# After applying the same feature engineering pipeline from the notebook:
# predicted_salary = reg_model.predict(X_engineered)   # EUR value
# predicted_class  = clf_model.predict(X_engineered)   # 0=Low, 1=Mid, 2=High
```

> **Important:** Both models expect the fully engineered feature set described in Part 4. Raw dataset columns cannot be fed directly.

---

## 📚 References

- Dataset: [Datapizza Salaries](https://huggingface.co/datasets/datapizza-ai-lab/salaries) — CC-BY-NC-4.0
- scikit-learn documentation: [sklearn.org](https://scikit-learn.org)
- HuggingFace Datasets: [huggingface.co/docs/datasets](https://huggingface.co/docs/datasets)
- Course: Data Science, Reichman University, 2025–2026