Honesty in Causal Forests: When It Helps and When It Hurts

Abstract

Causal forests are designed to quantify how treatment effects differ among individuals, thereby supporting personalized strategies in fields ranging from public policy and operations to marketing. A conventional approach within this framework is "honest estimation," which involves splitting the dataset into two distinct parts: one subset is used to identify subgroups, while the second is employed to estimate treatment effects within those groups. This method is widely adopted as a standard practice and is often the default setting in various software packages, primarily to mitigate the risk of overfitting. However, we question whether this default choice is always optimal. Our analysis demonstrates that honest estimation may actually diminish the accuracy of individual treatment effect estimates, particularly in scenarios characterized by significant effect heterogeneity and sufficiently large datasets capable of revealing such patterns.

The underlying mechanism for this reduction in accuracy is a bias-variance trade-off. While enforcing honesty reduces the likelihood of overfitting, it simultaneously raises the risk of underfitting by constraining the data available to identify and model heterogeneity. Through an extensive evaluation across more than 7,000 benchmark datasets, we determined that relying on honesty by default can carry a substantial cost; in some cases, it requires up to 27% more data to achieve performance levels comparable to models trained without this constraint. Consequently, honesty should be viewed as a specific type of regularization technique. Its adoption should not be automatic but rather determined by the specific objectives of the application and its empirical results on the data at hand.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC