You can deploy this model and start earning money today!

adiprog14/
tech-salary-predictor

Name: tech-salary-predictor
Availability: InStock

Views

No views yet

🇮🇹 Italian Tech Salaries — Salary Prediction

Assignment #2 | Classification, Regression, Clustering & Evaluation Course: Data Science | Reichman University Student: Adi Prog | adiprog14 Dataset: datapizza-ai-lab/salaries

🎬 Presentation Video

Can't play the video? Click here to watch

📌 Project Overview

Research Question:

What are the strongest predictors of salary in the Italian tech market — and does AI adoption actually pay off?

Goal: Build a machine learning pipeline to predict the gross annual salary (RAL in EUR) of Italian tech workers, using their professional profile, education, company details, and AI usage habits.

Property	Value
Dataset	Datapizza Italian Tech Salaries
Source	salaries.datapizza.tech
Rows	~26,000 anonymous survey responses
Features	20 (numeric + categorical + list-based)
Target	`salary` — gross annual salary in EUR
License	CC-BY-NC-4.0

📁 Repository Files

File	Description
`README.md`	Full project documentation (this file)
`Assignment2_Salaries.ipynb`	Complete Python notebook — all code, outputs, and analysis
`regression_model.pkl`	Winning regression model (Gradient Boosting)
`classification_model.pkl`	Winning classification model (Gradient Boosting)
`presentation.mp4`	4–6 minute video walkthrough

🔍 Part 1 — Dataset

The Datapizza Salaries dataset is a crowd-sourced anonymous survey of tech workers across Italy, collected and updated weekly by Datapizza.

Feature breakdown:

Type	Features
Numeric	`monthsOfExperience`, `age`, `aiUsageFrequency`, `salary`
Categorical	`jobTitle`, `educationType`, `companyIndustry`, `companySize`, `province`, `workMode`, `gender`, `editor`
List-based	`technologies`, `aiTechnologies`, `aiTaskTypes`, `aiUpdateSources`, `admiredTechnologies`

Note: Submissions before November 2024 contain only partial data — fields like educationType, age, and aiUsageFrequency were added in a later survey version. This is handled during cleaning.

📊 Part 2 — Exploratory Data Analysis

2.1 Data Cleaning

Step	Action	Reason
Invalid rows	Kept only `valid == True`	Remove incomplete/test submissions
Salary outliers	Kept 5,000–300,000 EUR	Remove clearly erroneous entries
Age outliers	Kept 16–70 years	Remove non-realistic working ages
Experience cap	Capped at 480 months (40 years)	Remove data entry errors
Duplicates	Removed exact duplicate rows	List columns required special string-conversion handling
Dropped columns	`valid`, `submittedAt`, `rating`	Not predictive for salary

Technical note: List-type columns (technologies, aiTechnologies, etc.) cannot be directly hashed by pandas. Duplicates were detected by converting list columns to sorted strings before comparison — then the mask was applied to the original dataframe.

2.2 Descriptive Statistics

Metric	salary (EUR)	Experience (months)	Age	AI Usage (1–5)
Mean	~44,000	~72	~33	~3.2
Median	~40,000	~60	~32	~3.0
Min	5,000	1	16	1
Max	300,000	480	70	5

2.3 Research Questions & Visualizations

Q1: What does the salary distribution look like?

The salary distribution is heavily right-skewed — most workers earn between 25k–60k EUR/year, with a long tail of high earners. The log-transformed version reveals a near-normal shape, suggesting a log-transform could help linear models.

Key insight: The majority of Italian tech workers earn under 60k EUR. Very high salaries (>100k) exist but are rare, mostly concentrated in senior leadership or specialized AI/cloud roles.

Q2: Which job titles earn the most?

Roles commanding the highest median salaries include solutions_architect, it_manager, and machine_learning_engineer. At the bottom sit data_analyst, junior_developer, and intern.

Key insight: Architecture and management roles consistently outperform pure coding roles — the scope of responsibility matters more than the technical stack alone.

Q3: Does experience correlate with salary?

The Pearson correlation between monthsOfExperience and salary is approximately 0.35 — a moderate positive relationship. The high variance in the scatter plot shows that experience alone does not determine salary.

Key insight: Experience is necessary but not sufficient. Two workers with the same seniority can earn very different amounts depending on role, company size, and location.

Q4: Does work mode affect salary?

Work Mode	Median Salary
Remote	Highest
Hybrid	Middle
Onsite	Lowest

Key insight: Remote workers earn a clear premium. This likely reflects that remote roles attract senior or specialized talent, and that companies hiring remotely compete in a national or international talent market.

Q5: Does education level impact salary?

A clear step-up pattern exists across education levels. The gap between High School and Master's degree is approximately 10,000 EUR/year in median salary. The Master's-to-PhD jump is smaller, since many senior tech roles don't require a PhD.

Key insight: Education pays off in the Italian tech market, with each level adding measurable salary uplift.

Q6: Does AI usage frequency correlate with salary?

Workers who report higher AI tool usage (scale 1–5) tend to earn more, with a consistent upward trend.

Key insight: AI adoption correlates with higher salary — but this is likely a proxy for role sophistication. Roles like ML Engineer naturally involve more AI usage and also pay more. The causality is indirect.

Q7: Which company sizes pay best?

Mid-to-large companies (201–5000 employees) pay the most. Very small companies (1–10) pay the least, likely reflecting startup equity-over-salary trade-offs not visible in this data.

Q8: Correlation Heatmap

Among numeric features, monthsOfExperience has the strongest correlation with salary. age and aiUsageFrequency show weaker but positive associations.

2.4 EDA Summary

Finding	Implication for Modeling
Salary is right-skewed	Log-transform may help linear models
Experience → salary (~0.35 correlation)	Strong baseline numeric feature
Remote work premium exists	`workMode` is a valuable feature
Education shows stepwise effect	Ordinal encoding is appropriate
AI usage → salary	Worth engineering as count/binary features
Larger companies → higher pay	Ordinal company size encoding useful
List columns contain rich information	Need count-based feature extraction

⚙️ Part 3 — Baseline Regression Model

Setup

Before any feature engineering, we established a benchmark using the simplest available features:

Features: monthsOfExperience + one-hot encoded jobTitle, educationType, companySize, workMode, companyIndustry, gender
Missing categoricals → filled with 'unknown'
Missing numeric → filled with column median
Train/Test split: 80% / 20%, random_state=42
Model: LinearRegression() — default scikit-learn parameters, no tuning

Baseline Results

Metric	Value
MAE	7,638 EUR
RMSE	11,953 EUR
R²	0.3586

The Actual vs Predicted scatter shows significant spread around the perfect-prediction diagonal. The model struggles with extreme salaries — it tends to overestimate low earners and underestimate high earners. The residuals are roughly centered at zero but with heavy tails, confirming that linear assumptions miss the non-linear patterns in salary data.

Feature Importance — Coefficients

The largest positive coefficients come from senior job titles and remote work mode. The largest negative coefficients come from junior roles and high school education. This is directionally correct and validates the EDA findings.

Baseline Limitations

Limitation	Impact
Ignores list features (technologies, AI tools)	Misses important skill-based signals
Assumes linear relationships	Salary is non-linear in reality
Simple missing value imputation	May introduce bias
Cannot capture feature interactions	e.g. senior + remote + big company → very high salary

These limitations directly motivate the feature engineering work in Part 4.

🛠️ Part 4 — Feature Engineering

Feature engineering is the most impactful step in the pipeline. We built 14 new features from the raw data, organized into four categories.

4.1 Numeric Transformations

Feature	Formula	Rationale
`years_of_experience`	`months / 12`	More interpretable unit
`is_senior`	`1 if years >= 5 else 0`	Captures the junior/senior threshold effect
`experience_squared`	`years²`	Models diminishing salary returns at high seniority

4.2 List-Based Features

The dataset contains list columns (e.g. technologies = ["Python", "AWS", "SQL"]). We extracted count-based and binary signals from them:

Feature	Description
`num_technologies`	Count of technologies used
`num_ai_tools`	Count of AI tools used
`num_ai_tasks`	Count of task types where AI is applied
`num_ai_sources`	Count of AI news sources followed
`uses_premium_ai`	Binary: uses Claude / Cursor / GitHub Copilot / Claude Code

Why uses_premium_ai? These tools are associated with more senior/specialized developers and higher-paying roles.

4.3 Ordinal Encodings

Rather than one-hot encoding ordered categories (which loses the ordering), we mapped them to integers:

Feature	Mapping
`companySize_ord`	1-10→1, 11-50→2, 51-200→3, ... 5001+→7
`education_ord`	High School→1, Bachelor→2, Master→3, PhD→4
`workMode_ord`	onsite→1, hybrid→2, remote→3
`is_remote`	Binary flag for remote work

4.4 Clustering — K-Means

We applied K-Means on the core numeric features to discover natural worker segments, then used the results as additional model features.

Elbow Method — choosing k:

The inertia curve bends noticeably around k=4, selecting 4 clusters as the optimal number.

Cluster Visualization (PCA 2D):

The PCA projection confirms reasonably separated clusters — K-Means found meaningful structure. The two principal components capture a significant portion of variance in the worker feature space.

Cluster Salary Profiles:

Cluster	Likely Profile	Salary Level
0	Junior generalists — low experience, broad tech stack	Lowest
1	Senior specialists — high experience, focused stack	Highest
2	AI-heavy roles — high AI usage, often remote	Above average
3	Mid-level hybrid — moderate experience, standard roles	Middle

Cluster features added:

cluster_id — which natural segment the worker belongs to
dist_to_centroid — distance to the cluster center (measures how "typical" the worker is)

4.5 Final Feature Set Summary

Feature	Type	Origin
`years_of_experience`	Numeric	Derived
`is_senior`	Binary	Derived
`experience_squared`	Numeric	Derived
`num_technologies`	Numeric	List extraction
`num_ai_tools`	Numeric	List extraction
`num_ai_tasks`	Numeric	List extraction
`num_ai_sources`	Numeric	List extraction
`uses_premium_ai`	Binary	List extraction
`companySize_ord`	Ordinal	Encoded
`education_ord`	Ordinal	Encoded
`workMode_ord`	Ordinal	Encoded
`is_remote`	Binary	Encoded
`cluster_id`	Categorical	K-Means
`dist_to_centroid`	Numeric	K-Means
`jobTitle_*`	One-hot	Encoded
`companyIndustry_*`	One-hot	Encoded
`gender_*`	One-hot	Encoded

🤖 Part 5 — Three Improved Regression Models

Using the full engineered feature set, we trained and compared three models.

Setup:

Train/Test split: 80/20, random_state=42
Scaling: StandardScaler applied for Linear Regression; tree models use raw features

Model 1 — Linear Regression (Engineered Features)

Same algorithm as the baseline, now with all engineered features. This isolates the impact of feature engineering alone on linear model performance.

Model 2 — Random Forest Regressor

n_estimators=200, random_state=42
Ensemble of 200 decision trees — handles non-linearity and feature interactions
Robust to remaining salary outliers

Model 3 — Gradient Boosting Regressor

n_estimators=200, learning_rate=0.1, random_state=42
Trees built sequentially — each one corrects the errors of the previous
Generally achieves the best bias-variance trade-off on structured tabular data

Results Comparison

Model	MAE (EUR)	RMSE (EUR)	R²
Baseline LR	7,638	11,953	0.3586
LR (engineered)	7,381	11,670	0.3886
Random Forest	7,797	12,412	0.3084
Gradient Boosting	7,306	11,668	0.3888

Feature Importance (Gradient Boosting)

The top predictors according to Gradient Boosting:

years_of_experience and experience_squared — seniority dominates
Job title dummies (senior roles) — role type is the second biggest driver
cluster_id — the K-Means cluster assignment adds real predictive value
companySize_ord — larger companies pay more
is_remote / workMode_ord — the remote premium is captured

Notably, cluster_id and dist_to_centroid appear in the top importances, validating that the unsupervised clustering step added genuine signal to the supervised model.

🏆 Winning Regression Model: Gradient Boosting Regressor

Reason	Explanation
Non-linearity	Salary relationships are non-linear — trees handle this naturally
Sequential correction	Each boosting round reduces remaining errors
Feature interactions	Senior + remote + large company → captured automatically
Outlier robustness	Less sensitive than Linear Regression to extreme salaries

🎯 Part 7 — Regression to Classification

Strategy: Quantile Binning (3 Classes)

The continuous salary target was converted into 3 tiers using quantile thresholds:

Class	Label	Salary Range
0	Low	Bottom 33%
1	Mid	Middle 33%
2	High	Top 33%

Why quantile binning? Quantile binning guarantees balanced classes — approximately equal representation in each tier. This avoids the class imbalance problem that fixed-threshold splits can create.

Class Distribution

The three classes are nearly equal in size (~33% each), confirming balanced classes. No oversampling or class weighting is required.

🔎 Part 8 — Three Classification Models

Precision vs Recall

In this salary classification task, recall matters more, specifically for the High salary class.

A False Negative — predicting Low/Mid when the worker is actually High — is more costly than a False Positive:

An HR tool that misclassifies a high-earner as mid-level → under-compensation → talent loss
A recruiting system that misses high-salary candidates → misallocated budget

False Negatives are the more critical error in this domain.

Model 1 — Logistic Regression

Trained on StandardScaler-normalized features
Linear decision boundaries between salary classes
Fast and interpretable, but constrained by linearity

Model 2 — Random Forest Classifier

n_estimators=200, random_state=42
Handles non-linear class boundaries
Ensemble reduces variance compared to a single tree

Model 3 — Gradient Boosting Classifier

n_estimators=200, learning_rate=0.1, random_state=42
Focuses on misclassified samples in each sequential round
Best recall on the High salary class

Confusion Matrices

Observations from the confusion matrices:

Most errors occur at the Mid class boundaries — this is expected, since Mid salaries overlap with both Low and High at the edges
Gradient Boosting makes the fewest off-diagonal mistakes overall
Logistic Regression struggles most with the Mid class due to linear boundaries
High class recall is best in Gradient Boosting — matching our stated priority

Results Comparison

Model	Accuracy	F1 Macro
Logistic Regression	0.6241	0.6224
Random Forest	0.6007	0.5980
Gradient Boosting	0.6241	0.6221

🏆 Winning Classification Model: Gradient Boosting Classifier

Reason	Explanation
Best F1 Macro	Performs well across all three salary tiers
Best High-class recall	Aligns with our priority metric
Non-linear boundaries	Captures complex salary tier separations
Sequential boosting	Reduces bias vs Random Forest

💡 Key Takeaways & Lessons Learned

What worked well

Feature engineering had the biggest single impact — especially ordinal encodings and list-count features. The improvement from baseline LR to engineered LR shows this clearly.
K-Means clustering added genuine predictive value — cluster_id appeared in the top feature importances, validating the unsupervised step.
Gradient Boosting won on both tasks — regression and classification — which is consistent with its known strength on structured tabular data.
Quantile binning produced balanced classes without any oversampling tricks.

What was challenging

List-type columns required custom handling throughout — from duplicate detection (hashing issue) to feature extraction (no direct encoding possible).
High missing rates in AI-related columns meant many rows had partial data — imputation choices affected model quality.
Salary variance is genuinely high — even with good features, salary prediction has inherent noise due to individual negotiation and company-specific pay bands not captured in the survey.

Domain insights

Experience is the strongest predictor, but role and company size matter almost as much
Remote work pays a real premium — consistent across all analyses
AI adoption correlates with salary — a signal of role sophistication, not a direct cause
The Italian tech market shows significant salary disparity by region (Milano dominates), which this model does not fully exploit since province was dropped due to high cardinality

🚀 How to Use the Models

python

import pickle
import pandas as pd
import numpy as np

# Load regression model
with open('regression_model.pkl', 'rb') as f:
    reg_model = pickle.load(f)

# Load classification model
with open('classification_model.pkl', 'rb') as f:
    clf_model = pickle.load(f)

# After applying the same feature engineering pipeline from the notebook:
# predicted_salary = reg_model.predict(X_engineered)   # EUR value
# predicted_class  = clf_model.predict(X_engineered)   # 0=Low, 1=Mid, 2=High

Important: Both models expect the fully engineered feature set described in Part 4. Raw dataset columns cannot be fed directly.

📚 References

Dataset: Datapizza Salaries — CC-BY-NC-4.0
scikit-learn documentation: sklearn.org
HuggingFace Datasets: huggingface.co/docs/datasets
Course: Data Science, Reichman University, 2025–2026

adiprog14/tech-salary-predictor

🇮🇹 Italian Tech Salaries — Salary Prediction

🎬 Presentation Video

📌 Project Overview

📁 Repository Files

🔍 Part 1 — Dataset

📊 Part 2 — Exploratory Data Analysis

2.1 Data Cleaning

2.2 Descriptive Statistics

2.3 Research Questions & Visualizations

2.4 EDA Summary

⚙️ Part 3 — Baseline Regression Model

Setup

Baseline Results

Feature Importance — Coefficients

Baseline Limitations

🛠️ Part 4 — Feature Engineering

4.1 Numeric Transformations

4.2 List-Based Features

4.3 Ordinal Encodings

4.4 Clustering — K-Means

4.5 Final Feature Set Summary

🤖 Part 5 — Three Improved Regression Models

Model 1 — Linear Regression (Engineered Features)

Model 2 — Random Forest Regressor

Model 3 — Gradient Boosting Regressor

Results Comparison

Feature Importance (Gradient Boosting)

🏆 Winning Regression Model: Gradient Boosting Regressor

🎯 Part 7 — Regression to Classification

Strategy: Quantile Binning (3 Classes)

Class Distribution

🔎 Part 8 — Three Classification Models

Precision vs Recall

Model 1 — Logistic Regression

Model 2 — Random Forest Classifier

Model 3 — Gradient Boosting Classifier

Confusion Matrices

Results Comparison

🏆 Winning Classification Model: Gradient Boosting Classifier

💡 Key Takeaways & Lessons Learned

What worked well

What was challenging

Domain insights

🚀 How to Use the Models

📚 References

adiprog14/
tech-salary-predictor