adiprog14/tech-salary-predictor

🇮🇹 Italian Tech Salaries — Salary Prediction

Assignment #2 | Classification, Regression, Clustering & Evaluation Course: Data Science | Reichman University Student: Adi Prog | adiprog14 Dataset: datapizza-ai-lab/salaries

---

🎬 Presentation Video

Can't play the video? Click here to watch

---

📌 Project Overview

Research Question:

What are the strongest predictors of salary in the Italian tech market — and does AI adoption actually pay off?

Goal: Build a machine learning pipeline to predict the gross annual salary (RAL in EUR) of Italian tech workers, using their professional profile, education, company details, and AI usage habits.

Property	Value
Dataset	Datapizza Italian Tech Salaries
Source	salaries.datapizza.tech
Rows	~26,000 anonymous survey responses
Features	20 (numeric + categorical + list-based)
Target	salary — gross annual salary in EUR
License	CC-BY-NC-4.0

---

📁 Repository Files

File	Description
README.md	Full project documentation (this file)
Assignment2_Salaries.ipynb	Complete Python notebook — all code, outputs, and analysis
regression_model.pkl	Winning regression model (Gradient Boosting)
classification_model.pkl	Winning classification model (Gradient Boosting)
presentation.mp4	4–6 minute video walkthrough

---

🔍 Part 1 — Dataset

The Datapizza Salaries dataset is a crowd-sourced anonymous survey of tech workers across Italy, collected and updated weekly by Datapizza.

Feature breakdown:

Type	Features
Numeric	monthsOfExperience, age, aiUsageFrequency, salary
Categorical	jobTitle, educationType, companyIndustry, companySize, province, workMode, gender, editor
List-based	technologies, aiTechnologies, aiTaskTypes, aiUpdateSources, admiredTechnologies

Note: Submissions before November 2024 contain only partial data — fields like educationType, age, and aiUsageFrequency were added in a later survey version. This is handled during cleaning.

---

📊 Part 2 — Exploratory Data Analysis

2.1 Data Cleaning

Step	Action	Reason
Invalid rows	Kept only valid == True	Remove incomplete/test submissions
Salary outliers	Kept 5,000–300,000 EUR	Remove clearly erroneous entries
Age outliers	Kept 16–70 years	Remove non-realistic working ages
Experience cap	Capped at 480 months (40 years)	Remove data entry errors
Duplicates	Removed exact duplicate rows	List columns required special string-conversion handling
Dropped columns	valid, submittedAt, rating	Not predictive for salary

Technical note: List-type columns (technologies, aiTechnologies, etc.) cannot be directly hashed by pandas. Duplicates were detected by converting list columns to sorted strings before comparison — then the mask was applied to the original dataframe.

---

2.2 Descriptive Statistics

Metric	salary (EUR)	Experience (months)	Age	AI Usage (1–5)
Mean	~44,000	~72	~33	~3.2
Median	~40,000	~60	~32	~3.0
Min	5,000	1	16	1
Max	300,000	480	70	5

---

2.3 Research Questions & Visualizations

Q1: What does the salary distribution look like?

The salary distribution is heavily right-skewed — most workers earn between 25k–60k EUR/year, with a long tail of high earners. The log-transformed version reveals a near-normal shape, suggesting a log-transform could help linear models.

Key insight: The majority of Italian tech workers earn under 60k EUR. Very high salaries (>100k) exist but are rare, mostly concentrated in senior leadership or specialized AI/cloud roles.

---

Q2: Which job titles earn the most?

Roles commanding the highest median salaries include solutions_architect, it_manager, and machine_learning_engineer. At the bottom sit data_analyst, junior_developer, and intern.

Key insight: Architecture and management roles consistently outperform pure coding roles — the scope of responsibility matters more than the technical stack alone.

---

Q3: Does experience correlate with salary?

The Pearson correlation between monthsOfExperience and salary is approximately 0.35 — a moderate positive relationship. The high variance in the scatter plot shows that experience alone does not determine salary.

Key insight: Experience is necessary but not sufficient. Two workers with the same seniority can earn very different amounts depending on role, company size, and location.

---

Q4: Does work mode affect salary?

Work Mode	Median Salary
Remote	Highest
Hybrid	Middle
Onsite	Lowest

Key insight: Remote workers earn a clear premium. This likely reflects that remote roles attract senior or specialized talent, and that companies hiring remotely compete in a national or international talent market.

---

Q5: Does education level impact salary?

A clear step-up pattern exists across education levels. The gap between High School and Master's degree is approximately 10,000 EUR/year in median salary. The Master's-to-PhD jump is smaller, since many senior tech roles don't require a PhD.

Key insight: Education pays off in the Italian tech market, with each level adding measurable salary uplift.

---

Q6: Does AI usage frequency correlate with salary?

Workers who report higher AI tool usage (scale 1–5) tend to earn more, with a consistent upward trend.

Key insight: AI adoption correlates with higher salary — but this is likely a proxy for role sophistication. Roles like ML Engineer naturally involve more AI usage and also pay more. The causality is indirect.

---

Q7: Which company sizes pay best?

Mid-to-large companies (201–5000 employees) pay the most. Very small companies (1–10) pay the least, likely reflecting startup equity-over-salary trade-offs not visible in this data.

---

Q8: Correlation Heatmap

Among numeric features, monthsOfExperience has the strongest correlation with salary. age and aiUsageFrequency show weaker but positive associations.

---

2.4 EDA Summary

Finding	Implication for Modeling
Salary is right-skewed	Log-transform may help linear models
Experience → salary (~0.35 correlation)	Strong baseline numeric feature
Remote work premium exists	workMode is a valuable feature
Education shows stepwise effect	Ordinal encoding is appropriate
AI usage → salary	Worth engineering as count/binary features
Larger companies → higher pay	Ordinal company size encoding useful
List columns contain rich information	Need count-based feature extraction

---

⚙️ Part 3 — Baseline Regression Model

Setup

Before any feature engineering, we established a benchmark using the simplest available features:

• Features: monthsOfExperience + one-hot encoded jobTitle, educationType, companySize, workMode, companyIndustry, gender
• Missing categoricals → filled with 'unknown'
• Missing numeric → filled with column median
• Train/Test split: 80% / 20%, random_state=42
• Model: LinearRegression() — default scikit-learn parameters, no tuning

Baseline Results

Metric	Value
MAE	7,638 EUR
RMSE	11,953 EUR
R²	0.3586

The Actual vs Predicted scatter shows significant spread around the perfect-prediction diagonal. The model struggles with extreme salaries — it tends to overestimate low earners and underestimate high earners. The residuals are roughly centered at zero but with heavy tails, confirming that linear assumptions miss the non-linear patterns in salary data.

Feature Importance — Coefficients

The largest positive coefficients come from senior job titles and remote work mode. The largest negative coefficients come from junior roles and high school education. This is directionally correct and validates the EDA findings.

Baseline Limitations

Limitation	Impact
Ignores list features (technologies, AI tools)	Misses important skill-based signals
Assumes linear relationships	Salary is non-linear in reality
Simple missing value imputation	May introduce bias
Cannot capture feature interactions	e.g. senior + remote + big company → very high salary

These limitations directly motivate the feature engineering work in Part 4.

---

🛠️ Part 4 — Feature Engineering

Feature engineering is the most impactful step in the pipeline. We built 14 new features from the raw data, organized into four categories.

4.1 Numeric Transformations

Feature	Formula	Rationale
years_of_experience	months / 12	More interpretable unit
is_senior	1 if years >= 5 else 0	Captures the junior/senior threshold effect
experience_squared	years²	Models diminishing salary returns at high seniority

4.2 List-Based Features

The dataset contains list columns (e.g. technologies = ["Python", "AWS", "SQL"]). We extracted count-based and binary signals from them:

Feature	Description
num_technologies	Count of technologies used
num_ai_tools	Count of AI tools used
num_ai_tasks	Count of task types where AI is applied
num_ai_sources	Count of AI news sources followed
uses_premium_ai	Binary: uses Claude / Cursor / GitHub Copilot / Claude Code

Why uses_premium_ai? These tools are associated with more senior/specialized developers and higher-paying roles.

4.3 Ordinal Encodings

Rather than one-hot encoding ordered categories (which loses the ordering), we mapped them to integers:

Feature	Mapping
companySize_ord	1-10→1, 11-50→2, 51-200→3, ... 5001+→7
education_ord	High School→1, Bachelor→2, Master→3, PhD→4
workMode_ord	onsite→1, hybrid→2, remote→3
is_remote	Binary flag for remote work

4.4 Clustering — K-Means

We applied K-Means on the core numeric features to discover natural worker segments, then used the results as additional model features.

Elbow Method — choosing k:

The inertia curve bends noticeably around k=4, selecting 4 clusters as the optimal number.

Cluster Visualization (PCA 2D):

The PCA projection confirms reasonably separated clusters — K-Means found meaningful structure. The two principal components capture a significant portion of variance in the worker feature space.

Cluster Salary Profiles:

Cluster	Likely Profile	Salary Level
0	Junior generalists — low experience, broad tech stack	Lowest
1	Senior specialists — high experience, focused stack	Highest
2	AI-heavy roles — high AI usage, often remote	Above average
3	Mid-level hybrid — moderate experience, standard roles	Middle

Cluster features added:

• cluster_id — which natural segment the worker belongs to
• dist_to_centroid — distance to the cluster center (measures how "typical" the worker is)

4.5 Final Feature Set Summary

Feature	Type	Origin
years_of_experience	Numeric	Derived
is_senior	Binary	Derived
experience_squared	Numeric	Derived
num_technologies	Numeric	List extraction
num_ai_tools	Numeric	List extraction
num_ai_tasks	Numeric	List extraction
num_ai_sources	Numeric	List extraction
uses_premium_ai	Binary	List extraction
companySize_ord	Ordinal	Encoded
education_ord	Ordinal	Encoded
workMode_ord	Ordinal	Encoded
is_remote	Binary	Encoded
cluster_id	Categorical	K-Means
dist_to_centroid	Numeric	K-Means
jobTitle_*	One-hot	Encoded
companyIndustry_*	One-hot	Encoded
gender_*	One-hot	Encoded

---

🤖 Part 5 — Three Improved Regression Models

Using the full engineered feature set, we trained and compared three models.

Setup:

• Train/Test split: 80/20, random_state=42
• Scaling: StandardScaler applied for Linear Regression; tree models use raw features

Model 1 — Linear Regression (Engineered Features)

Same algorithm as the baseline, now with all engineered features. This isolates the impact of feature engineering alone on linear model performance.

Model 2 — Random Forest Regressor

• n_estimators=200, random_state=42
• Ensemble of 200 decision trees — handles non-linearity and feature interactions
• Robust to remaining salary outliers

Model 3 — Gradient Boosting Regressor

• n_estimators=200, learning_rate=0.1, random_state=42
• Trees built sequentially — each one corrects the errors of the previous
• Generally achieves the best bias-variance trade-off on structured tabular data

Results Comparison

Model	MAE (EUR)	RMSE (EUR)	R²
Baseline LR	7,638	11,953	0.3586
LR (engineered)	7,381	11,670	0.3886
Random Forest	7,797	12,412	0.3084
Gradient Boosting	7,306	11,668	0.3888

Feature Importance (Gradient Boosting)

The top predictors according to Gradient Boosting:

• years_of_experience and experience_squared — seniority dominates
• Job title dummies (senior roles) — role type is the second biggest driver
• cluster_id — the K-Means cluster assignment adds real predictive value
• companySize_ord — larger companies pay more
• is_remote / workMode_ord — the remote premium is captured

Notably, cluster_id and dist_to_centroid appear in the top importances, validating that the unsupervised clustering step added genuine signal to the supervised model.

🏆 Winning Regression Model: Gradient Boosting Regressor

Reason	Explanation
Non-linearity	Salary relationships are non-linear — trees handle this naturally
Sequential correction	Each boosting round reduces remaining errors
Feature interactions	Senior + remote + large company → captured automatically
Outlier robustness	Less sensitive than Linear Regression to extreme salaries

---

🎯 Part 7 — Regression to Classification

Strategy: Quantile Binning (3 Classes)

The continuous salary target was converted into 3 tiers using quantile thresholds:

Class	Label	Salary Range
0	Low	Bottom 33%
1	Mid	Middle 33%
2	High	Top 33%

Why quantile binning? Quantile binning guarantees balanced classes — approximately equal representation in each tier. This avoids the class imbalance problem that fixed-threshold splits can create.

Class Distribution

The three classes are nearly equal in size (~33% each), confirming balanced classes. No oversampling or class weighting is required.

---

🔎 Part 8 — Three Classification Models

Precision vs Recall

In this salary classification task, recall matters more, specifically for the High salary class.

A False Negative — predicting Low/Mid when the worker is actually High — is more costly than a False Positive:

• An HR tool that misclassifies a high-earner as mid-level → under-compensation → talent loss
• A recruiting system that misses high-salary candidates → misallocated budget

False Negatives are the more critical error in this domain.

Model 1 — Logistic Regression

• Trained on StandardScaler-normalized features
• Linear decision boundaries between salary classes
• Fast and interpretable, but constrained by linearity

Model 2 — Random Forest Classifier

• n_estimators=200, random_state=42
• Handles non-linear class boundaries
• Ensemble reduces variance compared to a single tree

Model 3 — Gradient Boosting Classifier

• n_estimators=200, learning_rate=0.1, random_state=42
• Focuses on misclassified samples in each sequential round
• Best recall on the High salary class

Confusion Matrices

Observations from the confusion matrices:

• Most errors occur at the Mid class boundaries — this is expected, since Mid salaries overlap with both Low and High at the edges
• Gradient Boosting makes the fewest off-diagonal mistakes overall
• Logistic Regression struggles most with the Mid class due to linear boundaries
• High class recall is best in Gradient Boosting — matching our stated priority

Results Comparison

Model	Accuracy	F1 Macro
Logistic Regression	0.6241	0.6224
Random Forest	0.6007	0.5980
Gradient Boosting	0.6241	0.6221

🏆 Winning Classification Model: Gradient Boosting Classifier

Reason	Explanation
Best F1 Macro	Performs well across all three salary tiers
Best High-class recall	Aligns with our priority metric
Non-linear boundaries	Captures complex salary tier separations
Sequential boosting	Reduces bias vs Random Forest

---

💡 Key Takeaways & Lessons Learned

What worked well

• Feature engineering had the biggest single impact — especially ordinal encodings and list-count features. The improvement from baseline LR to engineered LR shows this clearly.
• K-Means clustering added genuine predictive value — cluster_id appeared in the top feature importances, validating the unsupervised step.
• Gradient Boosting won on both tasks — regression and classification — which is consistent with its known strength on structured tabular data.
• Quantile binning produced balanced classes without any oversampling tricks.

What was challenging

• List-type columns required custom handling throughout — from duplicate detection (hashing issue) to feature extraction (no direct encoding possible).
• High missing rates in AI-related columns meant many rows had partial data — imputation choices affected model quality.
• Salary variance is genuinely high — even with good features, salary prediction has inherent noise due to individual negotiation and company-specific pay bands not captured in the survey.

Domain insights

• Experience is the strongest predictor, but role and company size matter almost as much
• Remote work pays a real premium — consistent across all analyses
• AI adoption correlates with salary — a signal of role sophistication, not a direct cause
• The Italian tech market shows significant salary disparity by region (Milano dominates), which this model does not fully exploit since province was dropped due to high cardinality

---

🚀 How to Use the Models

import pickle
import pandas as pd
import numpy as np

# Load regression model
with open('regression_model.pkl', 'rb') as f:
    reg_model = pickle.load(f)

# Load classification model
with open('classification_model.pkl', 'rb') as f:
    clf_model = pickle.load(f)

# After applying the same feature engineering pipeline from the notebook:
# predicted_salary = reg_model.predict(X_engineered)   # EUR value
# predicted_class  = clf_model.predict(X_engineered)   # 0=Low, 1=Mid, 2=High

Important: Both models expect the fully engineered feature set described in Part 4. Raw dataset columns cannot be fed directly.

---

📚 References

• Dataset: Datapizza Salaries — CC-BY-NC-4.0
• scikit-learn documentation: sklearn.org
• HuggingFace Datasets: huggingface.co/docs/datasets
• Course: Data Science, Reichman University, 2025–2026