Title: Establishing a Practical Upper Limit for Selection Bias Impacts in Clinical Prediction Systems

Abstract:

In real-world datasets, selection bias is a pervasive and frequently unpreventable phenomenon that undermines the generalizability of machine learning algorithms. When models developed on skewed data are applied to the wider target population, this lack of generalization can result in significant adverse outcomes, especially within high-stakes environments like healthcare. This danger underscores the critical need for practitioners to accurately evaluate model generalizability before implementation. However, current techniques for forecasting model performance often depend on impractical assumptions, such as having full access to the target distribution or possessing complete knowledge of the selection mechanisms driving the bias.

To overcome these constraints, we introduce a new upper bound for the worst-case model performance on the target population. This approach is designed for realistic scenarios where both the target population data and the specific selection mechanism are only partially observable. We validate the effectiveness and practical applicability of our method through extensive testing on fully synthetic datasets, semi-synthetic data from the All of Us Research Program, and real-world instances of selection bias found in MIMIC-IV. Our study provides a rigorous and actionable framework for estimating the influence of selection bias in settings that are otherwise difficult to analyze, thus empowering practitioners to develop safer and more robust models for healthcare and other domains.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC