Title: Obstacles in Calibrating Tree-Based Models for Imbalanced Classification Tasks

Abstract: In the context of imbalanced binary classification, a frequent strategy involves subsampling the majority class to generate a more balanced training set. However, this practice introduces bias into model predictions, as the algorithm learns from a dataset that fails to accurately reflect the target population. A standard method to correct this bias is to analytically map the resulting predictions to adjusted values using the majority class sampling rate. Our analysis reveals that applying such analytical calibration to random forests yields detrimental outcomes. Specifically, prevalence estimates become dependent on both the sampling rate applied and the number of predictors evaluated at each split within the forest. We attribute the former dependency to a demonstrated bias in decision trees and the latter to the known properties of random forests combined with analytical calibration techniques. Contrary to prevailing views in much of the existing literature, we demonstrate that decision trees can exhibit a bias toward the minority class. Consequently, these findings suggest that tree-based models trained on undersampled data should not undergo analytical calibration. Instead, methods capable of learning the miscalibration pattern inherent in the original model, such as beta calibration, offer a more appropriate solution.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC