--- --- tags: - regression - classification - clustering - tabular - linkedin - job-postings - sklearn - random-forest - decision-tree - kmeans - shap license: mit --- # 📊 LinkedIn Job Posting Engagement Analysis > **Which LinkedIn job posting characteristics predict candidate engagement (views) — and how well can engagement be predicted or classified using only posting-level features?** **Personal motivation:** As someone in entrepreneurship, understanding which job posting features attract candidates is directly relevant to future hiring decisions. --- ## 📹 Presentation Video --- I've added the link in case the video doesnt work in the readme- https://www.loom.com/share/c7d9b89a54234f699204b16a9a313c7d ## 🚀 Interactive Dashboard 👉 **[Open the LinkedIn Job Engagement Dashboard](https://huggingface.co/spaces/MichaelYitzchak/linkedin_Job_Engagement)** | Tab | Description | |---|---| | 🎯 Engagement Predictor | Enter posting details → get predicted views + High/Normal classification in real time | | 📊 EDA Dashboard | All 5 EDA findings as interactive charts | | i️ About | Feature groups, model details, limitations | --- ## 📋 Dataset at a Glance | Property | Value | |---|---| | **Source** | [LinkedIn Job Postings — arshkon/linkedin-job-postings (Kaggle)](https://www.kaggle.com/datasets/arshkon/linkedin-job-postings) | | **Original size** | 123,850 rows × 49 columns | | **Working sample** | 30,000 rows · `random_state=42` | | **After join with companies** | 30,000 rows × 40 columns | | **After cleaning** | 29,572 rows × 51 columns (in `df_model`) | | **Train / Test split** | 23,657 / 5,915 (80/20, `random_state=42`) | | **Regression target** | `log_views = log1p(views)` — log-transformed to handle right skew | | **Classification target** | `high_engagement` — top 25% of training views (threshold derived from training set only) | --- ## ⚠️ Scope & Limitations > LinkedIn's algorithm, sponsored status, and company follower counts drive the **majority of view variance** and are **unobservable** in this dataset. Models use posting-level features only. The practical goal is **ranking postings by predicted engagement**, not exact point prediction. Results show associations, not causal relationships. --- ## 🗂️ Repository Files | File | Description | |---|---| | `notebook.ipynb` | Full pipeline: Cleaning → EDA → Feature Engineering → Clustering → Regression → Classification → Bonus | | `linkedin_regression_model.pkl` | Winning regression model: Random Forest (Tuned via RandomizedSearchCV) | | `linkedin_classification_model.pkl` | Winning classification model: Decision Tree (max_depth=8, class_weight="balanced") | | `regression_model_results.csv` | Full regression model comparison table | | `classification_model_results.csv` | Full classification model comparison table | --- ## 🧹 Data Cleaning Pipeline **7 steps from 123,850 raw rows to a clean, leakage-free modelling matrix:** ``` Step 1 — Reproducible sampling 123,850 rows → sample(n=30,000, random_state=42) Joined with companies.csv on company_id (left join, rows preserved) Result: 30,000 rows × 40 columns Step 2 — Duplicate & missing target removal Removed duplicate rows Dropped rows where views is NaN or negative Result: 29,572 usable rows Step 3 — Date parsing listed_time, original_listed_time, expiry, closed_time → parsed to datetime Extracted: posting_year, posting_month, posting_dayofweek, posting_weekend Step 4 — Missing value analysis & column dropping Threshold: >70% missing → drop Dropped: closed_time (99.2%), skills_desc (98.1%), med_salary (95.1%), remote_allowed (87.9%), applies (81.1%), max_salary/min_salary (76%) Step 5 — Leakage columns excluded expiry, applies → removed (post-publication outcomes) views → kept as target only, never as a feature Step 6 — Salary imputation strategy has_salary_info = 1 if salary present, else 0 salary_midpoint computed from min/max salary where available Missing salary → imputed inside sklearn Pipeline on training data only Step 7 — Log transformation of target Raw views: mean=14.9, std=98.8, max=9,949 — heavily right-skewed log_views = log1p(views) — compresses scale, improves regression fit Predictions converted back via expm1() for interpretation Outliers (IQR method): 4,074 (13.8%) — kept, not removed ``` --- ## 🔍 EDA — 5 Research Questions > **Note on notebook ordering:** Q1=Work type, Q2=Salary, Q3=Description, Q4=Day of week, Q5=Seniority. Presented below in order of business impact. --- ### 💰 Q2 — Salary Transparency vs Views ``` No salary info ████████████░░░░░░░░░░░░░ ~12 avg views (70.1% of postings) Has salary info ████████████████████████░ ~21 avg views (29.9% of postings) +74.3% lift ✓ ``` > Only **8,562 of 29,572 postings (29.9%)** disclose salary. Transparent postings attract **74.3% more views** on average. This is the highest-leverage, lowest-cost recruiter action available. --- ### 📝 Q3 — Description Length vs Views ``` 1000 words ███████░░░░░░░░░░░░░ ~10 avg views — overwhelms candidates ``` > Non-linear relationship confirmed. Sweet spot: **250–500 words**. This motivated `description_density` — the **#1 feature** in the winning regression model. --- ### 📅 Q4 — Day of Week vs Views ``` Monday ████████████████████ 39 avg views ★ best day (n=1,837) Tuesday █████████████████░░░ 25 avg views Wednesday ████████████████░░░░ 22 avg views Thursday ███████████████░░░░░ 18 avg views Friday ███░░░░░░░░░░░░░░░░░ 7 avg views ✗ worst day (n=10,076) Saturday ████████████░░░░░░░░ 28 avg views (weekend — n=2,116 total, noisier) Sunday ████████████░░░░░░░░ 28 avg views (weekend — noisier) ``` > **Counterintuitive finding:** Weekend postings show higher averages (~28), but the weekend sample is tiny (2,116 obs total) making these estimates unreliable. Monday is the clear best weekday at 39 avg views. The day-of-week signal is modest and should not override content features. --- ### 💼 Q1 — Work Type vs Views ``` Contract ████████████████████ 29.97 avg views median: 7.0 Internship █████████████████░░░ 25.71 avg views median: 5.0 Full-time ████████░░░░░░░░░░░░ 13.70 avg views median: 4.0 ← 80% of volume Other ███████░░░░░░░░░░░░░ 11.27 avg views median: 4.0 Part-time ██████░░░░░░░░░░░░░░ 9.59 avg views median: 4.0 ``` > Contract and Internship roles show the highest engagement. However, **Full-time dominates volume** (23,674 of 29,572 postings = 80%). Work type is a useful predictive feature but should not be interpreted as causal. --- ### 🎓 Q5 — Seniority Level vs Views ``` Entry-level ████████████████████ 18 avg views n=792 Senior-level ████████████░░░░░░░░ 16 avg views n=3,577 Other/Mid ██████████░░░░░░░░░░ 15 avg views n=25,203 Entry vs Senior: +12.4% more views Entry vs Other: +18.9% more views ``` > Supply-side effect — more candidates qualify for junior roles, so the pool is larger. `is_entry_role` carries predictive signal because it proxies for **candidate pool size**, not intrinsic desirability. --- ### 🔥 Feature Correlation with log(views+1) ``` Feature Corr Direction Note ───────────────────────────────────────────────────────────────────── desc_salary_interaction +0.18 ↑ views strongest single predictor has_salary_info +0.14 ↑ views salary transparency salary_log +0.12 ↑ views salary level description_density +0.10 ↑ views content quality description_word_count +0.08 ↑ views description length is_software_role +0.08 ↑ views tech role demand is_data_role +0.07 ↑ views data role demand is_entry_role +0.06 ↑ views larger candidate pool posting_weekend -0.04 ↓ views small negative signal is_senior_role -0.03 ↓ views smaller candidate pool ───────────────────────────────────────────────────────────────────── Internal correlations (structural — not data leakage): salary_log ↔ salary_midpoint +0.96 log transform of same variable desc_wc ↔ desc_density +0.55 density uses length in formula is_software ↔ is_data +0.35 often co-occur in job titles is_senior ↔ is_entry -0.28 mutually exclusive by construction ``` > Most features show **weak linear correlation** — no single feature dominates. This motivated tree-based models (Random Forest, Gradient Boosting) which capture non-linear interactions and feature combinations. ### 🌡️ Correlation Heatmap (feature-to-feature + target) ``` log desc has sal desc is_ is_ is_ post is_ views dens sal log wc soft data entr wknd snr ────────────────────────────────────────────────────────────────────────────────────── log_views │ 1.00 0.10 0.14 0.12 0.08 0.08 0.07 0.06 -0.04 -0.03 description_density │ 0.10 1.00 0.02 0.04 0.55 0.01 0.01 -0.01 0.00 0.00 has_salary_info │ 0.14 0.02 1.00 0.72 0.03 0.06 0.07 -0.03 -0.01 -0.02 salary_log │ 0.12 0.04 0.72 1.00 0.04 0.05 0.06 -0.02 -0.01 -0.01 description_word_count │ 0.08 0.55 0.03 0.04 1.00 0.01 0.01 -0.01 0.00 0.00 is_software_role │ 0.08 0.01 0.06 0.05 0.01 1.00 0.35 -0.08 0.00 -0.05 is_data_role │ 0.07 0.01 0.07 0.06 0.01 0.35 1.00 -0.06 0.00 -0.04 is_entry_role │ 0.06 -0.01 -0.03 -0.02 -0.01 -0.08 -0.06 1.00 0.01 -0.28 posting_weekend │ -0.04 0.00 -0.01 -0.01 0.00 0.00 0.00 0.01 1.00 0.00 is_senior_role │ -0.03 0.00 -0.02 -0.01 0.00 -0.05 -0.04 -0.28 0.00 1.00 ────────────────────────────────────────────────────────────────────────────────────── Key structural correlations: salary_log ↔ has_salary_info +0.72 same underlying signal, different form desc_wc ↔ desc_density +0.55 density formula uses word count is_software ↔ is_data +0.35 frequently co-occur in job titles is_entry ↔ is_senior -0.28 mutually exclusive flags ``` > The heatmap confirms no multicollinearity crisis — the highest inter-feature correlation (salary_log ↔ has_salary_info at 0.72) is a structural relationship between two forms of the same signal, not a data problem. All correlations with log_views are weak, validating the move to non-linear tree-based models. --- ## ⚙️ Feature Engineering — 20 Base + 6 Cluster = 30 Total Features | Group | Features | |---|---| | Text length | `title_length`, `title_word_count`, `description_length`, `description_word_count` | | Text structure | `description_density` ★, `title_desc_ratio` | | Salary | `salary_midpoint`, `salary_range`, `has_salary_info`, `salary_log` | | Role keywords | `is_senior_role`, `is_entry_role`, `is_software_role`, `is_data_role`, `is_manager_role`, `is_sales_role`, `is_marketing_role`, `is_remote_text` | | Interactions | `desc_salary_interaction` ★, `senior_salary`, `weekend_remote`, `title_desc_word_interaction`, `salary_density_interaction`, `salary_description_interaction`, `title_density_interaction` | | Clustering | `cluster_0`, `cluster_1`, `cluster_2`, `cluster_3`, `cluster_4`, `cluster_5` | **Missing value strategy:** - Columns with >70% missing → dropped - Salary → `has_salary_info` flag + `salary_midpoint` where available; remaining NaN imputed inside sklearn Pipeline on training data only - Remaining numeric → `SimpleImputer(strategy="median")` inside Pipeline --- ## 🔵 Clustering — KMeans k=6 **Features used for clustering (12 total, leakage-checked):** `title_word_count`, `description_word_count`, `salary_log`, `description_density`, `has_salary_info`, `is_senior_role`, `is_entry_role`, `is_software_role`, `is_data_role`, `is_manager_role`, `is_sales_role`, `is_marketing_role` **Methods used to select k:** 1. Elbow method — inconclusive, no sharp elbow 2. KMeans silhouette scores on full training matrix 3. Cluster-size stability table 4. Interactive K-Means widget (visualization aid — uses sample) 5. Hierarchical clustering dendrogram (Ward linkage, 300 obs) 6. Agglomerative clustering comparison (k=2–10) ``` Silhouette scores by k (full training matrix): k=2 ████████░░░░░░░░░░░░ 0.198 smallest cluster: 6,830 (28.9%) k=3 █████████░░░░░░░░░░░ 0.221 smallest cluster: 2,100 (8.9%) k=4 ████████████████░░░░ 0.312 ← strong BUT largest=72% of data k=5 ██████████░░░░░░░░░░ 0.250 smallest: 526 (unstable) k=6 ████████████░░░░░░░░ 0.290 ← SELECTED ★ smallest: 583 (2.5%) k=7...
