Title: ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation

Abstract:

The evaluation of generative AI models has become increasingly resource-demanding, driven by the high cost of raters, the sluggish nature of inference, and the expanding ecosystem of benchmarks and models. To address these challenges, we introduce ProEval, a proactive evaluation framework that utilizes transfer learning to efficiently estimate performance and uncover failure cases. ProEval utilizes pre-trained Gaussian Processes (GPs) as surrogate models for the performance score function, which maps model inputs to specific metrics, such as safety violations or error severity. By treating performance estimation as a problem of Bayesian quadrature (BQ) and failure discovery as superlevel set sampling, we formulate uncertainty-aware decision strategies designed to actively select or synthesize highly informative inputs for testing. Theoretically, we demonstrate that our BQ estimator, based on pre-trained GPs, is both unbiased and bounded. Empirical results from extensive experiments across reasoning, safety alignment, and classification benchmarks show that ProEval is significantly more efficient than competitive baselines. Specifically, it achieves estimates within 1% of the ground truth using 8 to 65 times fewer samples, while simultaneously uncovering a wider variety of failure cases under a stricter evaluation budget.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC