Welcome! This project is an end-to-end Machine Learning exploration of the Israeli Supermarkets 2024 dataset. We’ve built a robust pipeline that doesn't just look at prices, but understands the patterns behind them—predicting exact costs and categorizing items into "Low," "Medium," and "High" price tiers.
📺 Project Walkthrough
Want to see the logic in action? Check out the video presentation below:
The Scale: We started with a massive pool of 10M+ rows. Through careful selection and cleaning, we focused on a high-quality sample of ~39,000 rows to keep our models sharp and efficient.
Core Features: We analyzed everything from itemcode and manufacturer to unitofmeasure and temporal data.
The Mission: Master the art of predicting itemprice (Regression) and price-tier categorization (Classification).
🛠️ Exploratory Data Analysis (EDA)
Before the "learning" starts, we had to clean up the noise. Our EDA focused on:
Filtering Noise: Dropped columns with >90% missing data and removed 10k+ duplicate entries.
Outlier Defense: We used the IQR method to ensure our model wasn't distracted by extreme price anomalies.
Sanity Checks: Removed impossible data points, like items with zero or negative quantities.
Refining the Target: Capped itemprice at the 99th percentile to create a more stable learning target.
Item Price Distribution
Capped Item Price Distribution
Item Price vs Quantity
🧪 Feature Engineering
We turned raw numbers into meaningful insights:
📅 The "When": Extracted specific time features (Day of week, Month, etc.) to see if prices fluctuate on weekends or holidays.
🏷️ The "Who": Applied Frequency Encoding to handle high-variety fields like item names and manufacturers.
🤖 The "Groups": We used K-Means Clustering to let the data group itself. These "hidden" clusters became a powerful new feature for our models.
Elbow Method Plot
PCA Clusters
📈 Regression Models
We progressed from a simple baseline to high-performance ensemble models.
Linear Regression (Improved): Leveraged our new engineered features.
Improved Linear Regression Coefficients
Random Forest Regressor: Captured complex, non-linear relationships.
Random Forest Model
Gradient Boosting Regressor: Our top performer!
Gradient Boosting Model
🏆 Regression Winner: Random Forest Regressor
Reasoning: It demonstrated the highest R2 score and lowest error metrics, proving that sequential learning is incredibly effective for retail price prediction.
🎯 Classification Models
We split the items into three balanced buckets: Low, Medium, and High using quantile binning.
Classification Class Distribution
The Classifiers
Logistic Regression: Our statistical baseline.
Logistic Regression Confusion Matrix
Random Forest Classifier: A robust ensemble approach.
Random Forest Classifier Confusion Matrix
Gradient Boosting Classifier: A strong competitor in precision.
Gradient Boosting Classifier Confusion Matrix
🏆 Classification Winner: Random Forest Classifier
Reasoning: The Random Forest Classifier emerged as the winner with 90.31% accuracy. Its strength lies in its high F1-score, indicating a near-perfect balance between Precision (minimizing mislabeling) and Recall (ensuring all items in a tier are caught). While simpler models struggled with the overlapping boundaries of price tiers, Random Forest’s ensemble logic successfully decoded the complex interactions between manufacturers, categories, and shopping clusters.
💡 Key Considerations: Precision vs. Recall
In retail analytics, the "cost" of a mistake varies:
Precision: If we label an item "High Price," how sure are we? High precision avoids over-pricing errors.
Recall: Are we catching all the "High Price" items? High recall ensures no premium goods are missed.
The Trade-off: (Add your specific critical error type here.)
🏁 Conclusion & Lessons Learned
Feature Power: Data context (like K-Means clusters) often provides a bigger boost than just "tuning the knobs."
Handling Scale: Processing millions of rows requires a balance of memory management and smart sampling.
Iteration is Key: The path from a basic Linear Regression to a Gradient Boosting winner highlights the importance of the iterative Data Science lifecycle.
🧑💻 How to Use
To use the winning models from this project, you can download the .pkl files and load them using the following code:
python
1import joblib
2import pandas as pd
34# 1. Load the Regression Winner (Random Forest)5regressor = joblib.load('random_forest_regressor_model.pkl')67# 2. Load the Classification Winner (Gradient Boosting)8classifier = joblib.load('gradient_boosting_classifier_model.pkl')
.png)
.png)
