Introduction

In the realm of machine learning, class imbalance is a pervasive challenge that can distort model performance and lead to biased predictions. The Synthetic Minority Over-sampling Technique (SMOTE) is a popular strategy employed to mitigate this issue by artificially augmenting the minority class in a dataset. This publication explores a comprehensive study conducted on 31 imbalanced datasets, with the minority class percentages ranging from 1.4% to 19.3%. We rigorously analyzed the impact of SMOTE on 15 different binary classification models trained across these datasets. By comparing the performance of models trained with and without SMOTE, this analysis aims to reveal detailed insights into how SMOTE consistently improves certain metrics, such as recall, and its overall influence on model outcomes. Our findings provide robust evidence of SMOTE's effectiveness, offering valuable insights for practitioners seeking to enhance their models for more equitable results.

Methodology

Comprehensive Model and Dataset Analysis

To understand the impact of the Synthetic Minority Over-sampling Technique (SMOTE) on machine learning models, we conducted an extensive comparison using 15 binary classification models across 31 imbalanced datasets. Each model was trained twice on every dataset: once with SMOTE applied to balance class distribution, and once without it. The primary goal was to observe and quantify the difference SMOTE makes in handling class imbalances and to evaluate its efficacy across varying conditions.

Performance Metrics Evaluated

We focused on five key performance metrics to assess the impact of SMOTE: accuracy, precision, recall, F1-score, and AUC score. For each metric, predictions from models trained with and without SMOTE were compared to determine which scenario yielded better performance. This was quantified by calculating the percentage of instances where the application of SMOTE led to superior outcomes compared to the no-SMOTE scenario.

Overall and Specific Insights

• General Trends
  The initial phase of our analysis involved aggregating the performance improvements across all models and datasets. This broad view helped identify overall trends in how SMOTE influences model performance, providing a general sense of its effectiveness across diverse scenarios.
• Model-Specific Analysis
  Delving deeper, we examined the impact of SMOTE on each of the 15 models individually. This model-specific analysis helped pinpoint which models benefited most from SMOTE, and under what circumstances its effect was most pronounced. By analyzing the variance in performance improvements, we could draw conclusions about model sensitivity to class distribution changes introduced by SMOTE.
• Dataset-Specific Analysis
  Similarly, we assessed the effect of SMOTE on each of the 31 datasets. This analysis was crucial for understanding how the proportion of the minority class in a dataset influences the efficacy of SMOTE. It provided insights into the threshold levels of class imbalance at which SMOTE is most beneficial, and highlighted datasets where SMOTE may not be as effective.

Results

The results of the paired t-tests were as follows:

• Accuracy: The mean accuracy was significantly lower with SMOTE, with a t-statistic of 5.9847 and a p-value of approximately 0.00000146, indicating a statistically significant change.
• Precision: Similarly, precision showed a significant reduction with SMOTE, evidenced by a t-statistic of 5.6160 and a p-value of about 0.00000410.
• Recall: The improvement in recall was the most significant among all metrics, with a t-statistic of -6.7750 and a p-value of nearly 0.00000016, strongly suggesting that SMOTE is particularly effective in improving sensitivity to the minority class.
• F1-Score: The f1-score, which balances precision and recall, also improved with SMOTE, though with a t-statistic of -2.6444 and a p-value of 0.0129, indicating a moderate level of statistical significance.
• AUC Score: For the AUC score, the changes were not statistically significant, with a t-statistic of 1.4278 and a p-value of 0.1637, suggesting that the effect of SMOTE on the AUC score might be less consistent or significant.

The following bar chart shows the direction of the change per metric:

(placeholder for bar chart)

Statistical Analysis

To rigorously determine whether the differences observed between the SMOTE and no-SMOTE scenarios were statistically significant, we applied the paired t-test to the aggregated results. This test was essential to validate the impact of SMOTE beyond observational metrics by confirming whether the changes were due to the technique or could be attributed to random variations in model performance.

For each metric, we calculated the mean scores across all 15 models for every dataset. This approach allowed us to treat the mean of these scores as a single observation per dataset for each metric. We then compared these aggregated mean scores between the SMOTE and no-SMOTE scenarios using a paired t-test, which is particularly suited for this type of analysis where two related samples are compared (in our case, the same models and datasets with and without SMOTE).