For this assignment, I worked with a single dataset named marketing_campaign_dataset.csv, which contains information about digital marketing campaigns and their performance.
The dataset includes:
• Performance Metrics
Impressions
Clicks
Cost-related metrics
Existing engagement-related fields
ROI and other campaign performance indicators
• Campaign Attributes
Marketing Channel (social, search, email, etc.)
Audience / Segment type
Campaign type (awareness, conversion, remarketing, etc.)
Date fields
• Target Variable
A continuous target called Engagement_Score was engineered and scaled to a 0–100 range.
This score represents how strongly users reacted to content, based on core performance metrics.
• Assignment Structure
Using the same dataset, I performed three required machine learning tasks:
Regression – Predict the continuous Engagement_Score
Classification – Convert the score into Low / Medium / High engagement levels
Clustering – Use K Means to segment campaigns into behavioral groups and add a new feature: Cluster_ID
All processing and modeling were performed in Google Colab.
Final models were exported as pickle files for upload to HuggingFace.
Exploratory Data Analysis (EDA)
Before modeling, I conducted a structured EDA process to understand the dataset and identify insights relevant to campaign performance.
Basic checks included:
Inspecting data types (df.info())
Verifying missing values (df.isnull().sum())
Removing duplicates
Visualizing distributions and relationships between key variables
This confirmed that the dataset is complete and ready for downstream modeling.
Question 1: Are there problematic outliers in the numeric features?**
I examined boxplots of the main numeric performance indicators such as Impressions, Clicks, and ROI.
Findings.
Several campaigns show extremely large values (very high impressions or clicks).
These are real campaign behaviors, not data errors.
The distribution is heavy-tailed, which is typical in digital marketing.
Conclusion:
Outliers were kept, because they represent high/low performance extremes that are important for predictive modeling.
Outlier Detection in Key Numeric Features
Question 2: What is the distribution of the Engagement_Score?
I plotted a histogram to examine the variability of Engagement_Score.
Findings.
The distribution is spread across a wide range.
There is no single dominant peak.
Engagement levels vary substantially between campaigns.
Conclusion:
The wide spread is ideal for regression, and it also justifies creating clear categories later for classification.
Distribution of Engagement Score
Question 3: Do more impressions lead to higher engagement?
A scatter plot of Impressions vs Engagement_Score was generated.
Findings.
Engagement does not increase with impressions.
There is no visible linear or nonlinear correlation.
High-impression campaigns may still receive low engagement and vice versa.
Conclusion:
Volume alone does not drive engagement.
This suggests that targeting, channel, and content type matter more than exposure.
Impressions_vs_Engagement_Score
Question 4: Are the numeric features correlated?
A correlation heatmap was generated for all numeric variables.
Findings.
Correlations are near zero across all features.
No strong positive or negative relationships exist.
These results imply that the dataset lacks linear structure.
Conclusion:
Linear models may struggle with this dataset.
Feature engineering and flexible models (trees, ensembles) will likely improve performance.
Impressions_vs_Engagement_Score
Baseline Regression Model
To establish a performance benchmark, a simple Linear Regression model was trained using the original features of the dataset, without any feature engineering or advanced preprocessing. The goal of this baseline model is not to achieve high performance, but to understand how well a linear structure fits the data before applying more sophisticated methods.
Train-test Split:
The dataset was split into 80 percent training and 20 percent testing, using a fixed random state to ensure reproducibility.
Model Description:
The baseline model used only the raw input features. No new features, transformations, or non-linear techniques were applied. This allows us to test whether the relationship between the campaign attributes and the engagement score can be captured through a simple linear mapping.
Results:
The Linear Regression model produced very weak results:
• MAE: approximately 2.50
• RMSE: approximately 2.88
• R²: very low and close to zero (even slightly negative)
These results indicate that the model is unable to explain the variance of the Engagement_Score. The relationship between the features and the target is not linear, and the baseline model struggles to capture any meaningful pattern.
Actual vs Predicted Plot:
Baseline Actual vs Predicted
The predictions deviate significantly from the ideal diagonal line. The points are widely scattered, indicating that the model cannot approximate the true values. This confirms the low R² score and the inability of a linear model to capture the underlying structure.
Error Distribution:
Baseline Error Distribution
The error values are spread across a wide range rather than clustering around zero. While there is no directional bias, the noise level is high, and the predictions lack stability.
Baseline Conclusions:
• The data clearly does not follow a linear pattern.
• The original feature set is insufficient for accurate prediction.
• More flexible models are needed, such as tree-based regressors, which can capture
complex, nonlinear relationships.
• Feature engineering, such as adding a Cluster_ID feature, will be necessary to improve
model performance.
• This baseline provides a minimum threshold that the advanced models must surpass.
Feature Engineering and Performance Clustering
In this stage, a new working dataset (df_fe) was created to support feature engineering and performance analysis. First, all critical numerical columns were converted to proper numeric types, including Clicks, Impressions, Conversion_Rate, ROI, and Engagement_Score. The Acquisition_Cost column was cleaned from currency symbols and converted to float as well.
Next, several behavioral performance features were engineered to better describe campaign behavior: CTR (click-through rate), Cost_per_Click, and Cost_per_Impression. The Duration column was transformed into a numeric Duration_days feature. The Date column was parsed into datetime format, followed by extracting Month and DayOfWeek as temporal features. An auxiliary label High_Engagement was also added to indicate whether the original engagement score was 7 or above.
To identify natural behavioral groups of campaigns, KMeans clustering was performed using numeric features such as Impressions, Clicks, Conversion_Rate, ROI, Acquisition_Cost, and CTR. Missing values were removed, and all clustering features were standardized using StandardScaler before running KMeans with 4 clusters. The resulting cluster labels were added as a new feature called Performance_Cluster.
Campaign Clusters in 2D PCA Space
To visualize the clustering structure, the clustered data was reduced to two dimensions using PCA. The resulting scatter plot shows how campaigns are distributed across the transformed space, with each cluster forming a different pattern in the 2D representation. This provides an intuitive view of behavioral similarities among campaigns.
Next, the relationship between clusters and the original Engagement_Score was evaluated by calculating the average engagement level of each cluster. Results showed almost identical averages across clusters, indicating that the original engagement score did not meaningfully differentiate between campaign types.
Average Engagement Score by Performance Cluster
To capture a stronger and more informative performance signal, a new continuous Engagement_Score was engineered. This score was constructed from a weighted combination of ROI, Impressions, and Clicks, with added random noise for natural variation and final clipping to the 0–100 range. After recalculating cluster averages, clear differences emerged: some clusters represented high-performing campaigns, while others showed significantly lower engagement.
Average Engagement Score by Performance Cluster (Updated)
Finally, a full preprocessing pipeline was defined for downstream machine learning models. A ColumnTransformer was used to jointly process numeric and categorical data:
1. Numeric features were standardized using StandardScaler (including engineered features such as CTR, Cost_per_Click, Cost_per_Impression, Duration_days, temporal features, and Performance_Cluster).
2. Categorical features such as Company, Campaign_Type, Target_Audience, Channel_Used, Location, Language, and Customer_Segment were encoded using OneHotEncoder with support for unseen categories.
This pipeline produces a rich, cleaned, and fully encoded feature matrix that integrates numeric performance indicators, categorical descriptors, engineered features, and unsupervised cluster labels. It forms the foundation for training the improved regression and classification models in the next section.
Building and Evaluating Advanced Regression Models
At this stage, a complete and robust data-preparation workflow was constructed to ensure that all regression models operate on a clean and consistent feature matrix. First, a working copy of the dataset (df_fe) was created. The Acquisition_Cost column was cleaned and converted into a numeric format whenever needed. Several engineered behavioral features were added, including CTR (click-through rate), Total_Cost (Acquisition_Cost multiplied by Clicks), and Cost_Per_Engagement (Total_Cost divided by Engagement_Score). These features provide a more precise description of the financial efficiency and performance of each marketing campaign.
To incorporate unsupervised learning into the predictive pipeline, new performance clusters were generated using KMeans. Numerical performance features such as Impressions, Clicks, Conversion_Rate, ROI, Acquisition_Cost, CTR, Total_Cost, and Cost_Per_Engagement were scaled with StandardScaler before clustering. The resulting cluster assignments were stored in the Performance_Cluster feature. Following this, one-hot encoding was applied to all available categorical features (e.g., Campaign_Type, Target_Audience, Channel_Used, Location, Language), producing a fully encoded dataset (df_encoded) containing numeric features, engineered features, cluster labels, and categorical dummy variables.
A final feature matrix X was then built by combining all numerical engineered features with the one-hot encoded categorical columns. The target variable y was defined as the Engagement_Score. The dataset was split into training and testing sets using an 80–20 split with a fixed random state to ensure reproducibility.
Using this enriched dataset, three regression models were trained and compared. The first model, an Improved Linear Regression, was trained on the new feature matrix. Its evaluation on the test set yielded: MAE ≈ 1.77, RMSE ≈ 2.08, and R² ≈ 0.48. This represents a significant improvement over the baseline model, but still indicates limited ability to capture the nonlinear structure of campaign performance.
The second model was a lightweight Random Forest Regressor, trained on a reduced sample (up to 30,000 rows) to improve computational speed. The model used fewer estimators, limited tree depth, and a minimum leaf size to balance accuracy and runtime. This model achieved outstanding performance: MAE ≈ 0.044, RMSE ≈ 0.15, and R² ≈ 0.997. These results demonstrate that the Random Forest model was able to capture nearly all variance in the engagement score, making it the strongest regressor among the models tested.
The third model, a lightweight Gradient Boosting Regressor, was also trained on a reduced sample for efficiency. With shallow trees and a moderate learning rate, the model achieved strong results across all metrics, confirming that boosting methods are well suited for structured tabular data.
Based on the overall comparison, the Random Forest Regressor was selected as the winning regression model. It achieved the highest predictive accuracy and the lowest error rates, demonstrating superior ability to model nonlinear relationships and complex feature interactions. This model forms the foundation of the predictive workflow for estimating campaign engagement levels in subsequent stages of the project.
Saving the Winning Model for Deployment
After evaluating all regression models trained in the project, the Random Forest Regressor was selected as the winning model due to its superior performance. At this stage, an object named best_model was created to explicitly reference the final chosen model (rf_reg). The trained model was then saved to a file named winning_model.pkl using the pickle library. Saving the model in this format allows it to be reloaded later without retraining, enabling fast and consistent predictions on new data. Finally, the file was downloaded from Google Colab so it can be uploaded to the HuggingFace repository and used as part of the complete machine learning workflow
Creating Engagement Classes
In this step, the goal was to convert the continuous Engagement Score into 3 categorical classes so that we can later train a classification model. I used quantile-based binning, which splits all campaigns into three equally sized groups based on their score distribution.
The classes are defined as:
• Class 0 – Low Engagement (bottom 33%)
• Class 1 – Medium Engagement (middle 33%)
• Class 2 – High Engagement (top 33%)
Both the train and test targets were transformed using the same method.
A class balance check showed good distribution across the classes (40% / 30% / 30%), which means the dataset is suitable for classification without additional balancing techniques.
Class Distribution
Creating these classes allows the model to predict whether a new campaign will be low, medium, or high engagement – a format that is much more practical for real-world decision making.
Classification Models – Predicting Engagement Classes (Low / Medium / High)
In this stage, the problem was converted from regression to classification by using the three engagement classes created earlier (0 = Low, 1 = Medium, 2 = High).
The goal here was to predict the engagement class for each campaign using the same engineered features from the previous sections.
To keep training time reasonable on the large dataset, all models were trained on a sample of 2,000 rows from the training set and evaluated on the full test set.
Three models were tested:
• Logistic Regression (multinomial, max_iter = 500, solver = “lbfgs”)
• Random Forest Classifier (150 trees, max_depth = 10, min_samples_leaf = 5)
• Gradient Boosting Classifier (120 trees, learning_rate = 0.08, max_depth = 3)
We measured performance using two metrics:
• Accuracy – percentage of correctly classified campaigns
• Macro F1 Score – averages F1 across all three classes equally (important because we care about all classes, not just the majority class)
Results on the full test set:
• Logistic Regression: Accuracy = 1.000, Macro F1 = 1.000
• Random Forest: Accuracy ≈ 0.806, Macro F1 ≈ 0.784
• Gradient Boosting: Accuracy ≈ 0.967, Macro F1 ≈ 0.965
Logistic Regression achieved perfect predictions on both metrics.
Confusion Matrices:
To better understand error patterns, confusion matrices were plotted using 1,000 test samples.
• Logistic Regression – perfect diagonals, zero misclassifications
Confusion Matrix – Logistic Regression
• Random Forest – decent accuracy, but confusion between class 1 (Medium) and the other classes
Confusion Matrix – Random Forest
• Gradient Boosting – very strong performance with small confusion between Medium and High
Confusion Matrix – Gradient Boosting
Conclusion
Logistic Regression was selected as the best classification model, thanks to:
• Perfect accuracy
• Perfect Macro F1 score
• Clean confusion matrix
• Stability and simplicity
The final chosen model was saved using pickle as:
best_classifier.pkl
.png)
.png)
.png)
.png)
