Title: RobustModelMaker: Integrating Bootstrap Stability Selection with Leakage-Resistant Nested Cross-Validation for Scientific Machine Learning

Abstract:

In the realm of scientific machine learning, small-to-medium sized datasets are subjected to dual, compounding challenges. First, performing feature selection in a single run often yields feature sets that fluctuate significantly when the training data undergoes minor perturbations. Second, employing the same data for selection, hyperparameter tuning, and final evaluation leads to performance estimates that are optimistically biased. While these two issues are typically addressed as independent problems, they interact critically within scientific data contexts: unstable feature selection amplifies the variance of scores that are already inflated by optimism. Furthermore, conventional fixes for one issue rarely resolve the other.

To address this, we introduce RobustModelMaker, a Python framework that integrates bootstrap stability selection with rigorous nested cross-validation. By executing all preprocessing and feature selection within each fold, the framework generates a feature subset validated for stability alongside a performance estimate that is safe from data leakage. The system accommodates nine different algorithms, covering binary and multiclass classification as well as regression tasks.

The framework’s behavior was validated through a comprehensive deterministic test suite encompassing unit, performance, and reproducibility assessments. This evaluation utilized three real-world scientific datasets and compared RobustModelMaker against three alternative selectors: ANOVA F-test, recursive feature elimination with cross-validation, and Boruta. The comparison focused on both predictive accuracy and selection stability, measured via the Jaccard index. Results indicate that RobustModelMaker matches the predictive scores of the top-performing alternative selectors on each dataset. More importantly, it occupies a unique position on the joint score-stability frontier, a balance that no other method achieves across all three task types.

Finally, two practical applications demonstrate the framework’s utility: discovering biomarkers for ovarian cancer using data from the PLCO Trial and performing critical-temperature regression on the UCI Superconductivity Data. These examples highlight how treating stability as a primary output, rather than an incidental outcome, reveals important trade-offs in scientific machine learning workflows.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC