--- language: - en tags: - salary-prediction - regression - classification - clustering - tabular - scikit-learn - stack-overflow - developer-survey - feature-engineering - gradient-boosting datasets: - stack-overflow-developer-survey-2025 base_model: None --- # Assignment 2 – Developer Salary Prediction ### Stack Overflow Developer Survey 2025 | Classification, Regression & Clustering --- ## Video # --- ## Overview This project uses the **Stack Overflow Annual Developer Survey 2025** (49,123 responses, 170 features) to predict a software developer's annual salary. The pipeline covers end-to-end data science: exploratory analysis, feature engineering, unsupervised clustering, regression, and multi-class classification. **Research Question:** Can we predict a software developer's annual salary from their professional profile, and which factors matter most? --- ## Dataset | Property | Value | |----------|-------| | Source | Stack Overflow Developer Survey 2025 (Kaggle) | | Raw rows | 49,123 | | Raw columns | 170 | | Target column | `ConvertedCompYearly` (annual salary in USD) | | Final feature count | 253 (after engineering + cluster feature) | --- ## Part 1 – Setup - Environment: Google Colab compatible - Reproducibility seed: `SEED = 42` - Key libraries: `pandas`, `numpy`, `scikit-learn`, `matplotlib` --- ## Part 2 – Exploratory Data Analysis ### 2.1 Data Cleaning - Removed rows with no salary value - Clipped extreme outliers at the 1st and 99th percentile (final median salary ≈ $75K) - Removed 44 columns with >60% missing values (170 → 126 columns), protecting the top-15 salary correlates regardless of missingness ### 2.2 Missing Value Analysis  Several columns exceed 60% missingness and are dropped. The protected essential columns are retained despite high missingness and imputed later. ### 2.3 Descriptive Statistics | Statistic | Value | |-----------|-------| | Median salary | ~$75,000 | | Distribution | Right-skewed | | Outlier treatment | 1st–99th percentile clip | ### 2.4 Salary Distribution  The raw distribution is heavily right-skewed with a long tail above $200K. ### 2.5 Research Questions & Findings **Q1: Does coding experience predict salary?**  Salary increases steeply through the first 15–20 years of experience then flattens. There is wide variance at every experience level, suggesting experience alone is not sufficient to predict salary. **Q2: How does education level affect salary?**  Median salary rises with education level, but the gap between a Bachelor's and Master's degree is smaller than expected. Professional degrees and doctoral holders show the highest median salaries. **Q3: Which countries pay developers the most?**  The US dominates with a median salary roughly 2–3× the global median. Israeli, Western European and Australian developers cluster in a second tier, while developers in Asia and South America earn less. **Q4: Do remote workers earn more?**  Fully remote developers show a slight salary premium over hybrid and in-office roles. The difference is modest, suggesting remote work correlates with higher-paying companies rather than being a direct cause. **Q5: How does salary vary across developer roles?**  C-Suite and ML/Data Science roles have the widest salary ranges and highest medians. Full-stack and front-end developers cluster around the global median with less variance. ### 2.6 Final Feature Selection (~20 Columns) | Category | Features | |----------|----------| | Target | `ConvertedCompYearly` | | Numeric | `YearsCode`, `WorkExp`, `JobSat`, `JobSatPoints_11`, `JobSatPoints_4` | | Demographics | `Age`, `Country`, `EdLevel`, `MainBranch` | | Work profile | `Employment`, `RemoteWork`, `DevType`, `OrgSize` | | Tech & AI | `LanguageHaveWorkedWith`, `AISelect` | | Learning | `LearnCodeChoose`, `SOVisitFreq` | ### 2.7 EDA Takeaways 1. **Salary** is right-skewed; median ~$75K after cleaning 2. **Work experience** (`WorkExp`) and **coding experience** (`YearsCode`) are the strongest numeric predictors 3. **Country** is the dominant signal — geography explains more variance than any other feature 4. **Remote work** carries a small positive premium 5. **Developer role and education** have meaningful but secondary effects --- ## Part 3 – Baseline Model A simple Linear Regression trained on raw numeric columns only — no encoding, no feature engineering. ### Train/Test Split - 80/20 random split, `SEED=42` ### Results | Metric | Baseline | |--------|----------| | MAE | $45,810 | | RMSE | $61,947 | | R2 | 0.1598 | ### Predicted vs. Actual  The baseline model struggles with high earners — predictions cluster around the mean and fail to capture the upper salary range. The scatter is wide, consistent with an R2 of only 0.16. --- ## Part 4 – Feature Engineering & Clustering ### Engineering Steps | Step | Description | |------|-------------| | 4.1 Numeric features | Derived ratio/interaction features | | 4.2 Ordinal encoding | `EdLevel`, `OrgSize` mapped to integers | | 4.3 One-hot encoding | `Country`, `RemoteWork`, `Employment`, `MainBranch`, `AISelect`, `SOVisitFreq`, `Age`, `PrimaryDevType` | | 4.4 Language flags | Binary flag for each of the top-10 programming languages | | 4.5 Imputation & scaling | Median imputation + `StandardScaler` → 249 features | ### KMeans Elbow Method  The inertia curve decreases gradually without a sharp elbow, reflecting the high-dimensional and overlapping nature of the data. k=4 was selected as a reasonable balance between cluster granularity and interpretability. ### Silhouette Score Comparison  Silhouette scores are low across all values of k, confirming that natural cluster separation is weak in this dataset. Agglomerative clustering consistently outperforms KMeans, peaking around k=4. ### Three Clustering Algorithms | Algorithm | k / params | Silhouette | |-----------|-----------|------------| | KMeans | k=4 | 0.0109 | | DBSCAN | eps=5, min_samples=25 | 0.0912 (7 clusters) | | Agglomerative Ward | k=4 | **0.0224** | The data's high dimensionality (249 features) makes density-based clustering (DBSCAN) impractical — inter-point distances are too large for meaningful core-point detection. ### Cluster Visualisations (PCA 2D)  KMeans splits the data into four roughly equal blobs with significant overlap in the PCA projection. The clusters correspond loosely to salary level but boundaries are indistinct.  DBSCAN classifies the vast majority of points as noise, forming 7 clusters. High dimensionality makes distance-based density estimation ineffective on this dataset.  Agglomerative clustering produces the clearest separation, isolating a distinct high-salary cluster on the right of the PCA plot. The four tiers align visually with low, mainstream, high-mid, and elite salary groups. ### Cluster Profiles – Agglomerative (Chosen) | Cluster | Mean Salary | Median Salary | Count | |---------|------------|---------------|-------| | 0 | $86,041 | $74,000 | 18,192 | | 1 | $109,980 | $95,000 | 4,069 | | 2 | $27,574 | $13,949 | 877 | | 3 | $101,735 | $93,387 | 317 | **Winner: Agglomerative Ward (k=4)** — highest silhouette score and four interpretable salary tiers (low-income, mainstream, high-mid, elite). ### Cluster Feature Added `cluster_id` one-hot encoded and appended → **253 final features** --- ## Part 5 – Improved Regression Models Three models trained on the full 253-feature matrix (249 engineered features + 4 cluster dummies). ### Results | Model | MAE | RMSE | R2 | |-------|-----|------|----| | Baseline Linear Regression | $45,810 | $61,947 | 0.1598 | | Improved Linear Regression | $30,688 | $44,314 | 0.5701 | | Random Forest (200 trees) | $31,998 | $45,784 | 0.5411 | | **HistGradientBoosting (300 iters)** | **$28,991** | **$43,039** | **0.5944** | ### Model Performance Comparison  HistGradientBoosting wins on all three metrics. The jump from baseline to improved linear regression is dramatic — encoding Country alone accounts for the majority of the R2 improvement from 0.16 to 0.57. ### Feature Importance  Country dummies (especially US) dominate feature importance across all three models. Work experience and years of coding rank consistently high. The cluster feature appears in the top 20 for linear regression, validating the clustering step. ### Winning Model – Predicted vs. Actual  The HistGradientBoosting model tracks the perfect-prediction diagonal much more closely than the baseline. It still under-predicts some very high earners above $300K but captures the mid-range salary distribution well. ### Discussion - **Baseline → Improved Linear Regression (+0.41 R2):** One-hot encoding `Country` was the single biggest improvement. Geography is the dominant salary signal. - **Random Forest vs. Linear Regression:** Non-linear feature interactions (e.g. senior developer × US location) are captured naturally by trees. - **HistGradientBoosting wins:** Sequential boosting focuses on the hardest predictions. Natively handles missing values and is 10–100× faster than standard GradientBoosting. - **Cluster feature:** Pre-computed salary-tier signal from Part 4 particularly boosts Linear Regression. ### Winner: HistGradientBoosting Regressor | Metric | Value | |--------|-------| | MAE | $28,991 | | RMSE | $43,039 | | R2 | 0.5944 | --- ## Part 6 – Winning Regression Model Export The winning regression model is saved to `winning_model_regression.pkl`. --- ## Part 7 – Salary Classification Setup The continuous salary target is binned into four ordered classes: | Class | Label | Range (USD/year) | |-------|-------|------------------| | 0 | Low |...
