Title: Calibration Hierarchies: The Intersection of Classification and Regression

Abstract

Calibration serves to formalize the alignment between probabilistic forecasts and their actual outcomes. Fundamentally, this implies that observed outcomes should be statistically indistinguishable from random samples drawn from the predicted distributions. This study provides a comprehensive review, extension, and synthesis of calibration frameworks previously established for both regression and classification problems. We place significant focus on the hierarchical interconnections among these concepts, examining their applicability across diverse data types, including general real-valued inputs, continuous and count-based outcomes, as well as nominal and binary categories.

Key contributions of this work include the introduction of "modal calibration" for nominal outcomes, where we differentiate between full, partial, and average calibration. Additionally, we demonstrate that double probability integral transform (PIT) calibration operates as a concept logically distinct from earlier definitions of calibration for discrete data. We also expand upon existing theories regarding calibration concepts defined by properties or functionals of predictive distributions, such as event probabilities, quantiles, and means. To elucidate these concepts and their hierarchical structures, we employ detailed worked examples and supply algorithmic resources designed to facilitate the creation of illustrative cases and counterexamples.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC