Title: CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Original: arXiv:2605.30188v2 Announce Type: replace-cross Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.

Rewritten: Title: CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Abstract: Accurate probability estimates are vital for numerous machine learning applications; however, contemporary classifiers frequently suffer from poor calibration. While post-hoc calibration is a straightforward and commonly adopted remedy, the proliferation of proposed techniques alongside fragmented and small-scale evaluations hinders the ability to identify genuinely effective practical solutions. To address this, we present a comprehensive, standardized benchmark for post-hoc calibration. This resource encompasses nearly 2,000 experiments spanning both tabular data and computer vision domains, incorporating binary, multiclass, and large-scale classification scenarios. By consolidating predictions from a wide array of classical algorithms, modern deep learning structures, and foundation models, our benchmark offers unified and reproducible implementations of dozens of calibration techniques within a single evaluation framework. We propose that Post-Hoc Improvement (PHI), measured via proper scoring rules, serves as a robust alternative to conventional calibration error estimators for assessing post-hoc methods. This metric effectively captures both the enhancement in calibration accuracy and any potential decline in the model's overall predictive power. Leveraging this framework, we execute the most extensive empirical investigation of post-hoc calibration to date. Our findings demonstrate consistent trends across various domains: smooth calibration functions surpass binning-based methods, specialized multiclass techniques are crucial for high-dimensional data, and generic machine learning models fail to compete without calibration-specific modifications. To support ongoing research, we have made all associated data, code, and evaluation tools publicly available, offering a ready-to-use benchmark for the development and comparison of calibration methods.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC