Title: A Theoretical Framework for Statistical Evaluability of Generative Models

Original: arXiv:2604.05324v2 Announce Type: replace Abstract: Statistical evaluation aims to estimate the generalization performance of a model using held-out i.i.d. test data sampled from the ground-truth distribution. In supervised learning settings such as classification, performance metrics such as error rate are well-defined, and test error reliably approximates population error given sufficiently large datasets. In contrast, evaluation is more challenging for generative models due to their open-ended nature: it is unclear which metrics are appropriate and whether such metrics can be reliably evaluated from finite samples. In this work, we introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics. We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs), and R\'enyi divergences. We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors. Moreover, when the test class has finite fat-shattering dimension, IPMs can be evaluated with arbitrary precision. In contrast, R\'enyi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events. We also analyze the potential and limitations of perplexity as an evaluation method.

Rewrite: Statistical evaluation seeks to gauge a model's generalization capabilities by leveraging independent and identically distributed test data drawn from the true underlying distribution. In supervised domains like classification, metrics such as error rates are clearly defined, and with ample data, test error serves as a robust proxy for population error. However, assessing generative models presents greater difficulties because of their open-ended output space; there is ambiguity regarding which metrics are suitable and whether they can be accurately assessed from limited samples. This paper presents a theoretical framework for the evaluation of generative models, deriving evaluability guarantees for standard metrics. We examine two distinct classes of metrics: test-based measures, such as integral probability metrics (IPMs) and R\'enyi divergences. Our findings demonstrate that IPMs associated with any bounded test class can be estimated from finite samples, subject to multiplicative and additive approximation errors. Furthermore, if the test class possesses a finite fat-shattering dimension, IPMs allow for evaluation at arbitrary levels of precision. Conversely, R\'enyi and Kullback-Leibler (KL) divergences are deemed non-evaluable from finite samples because their values are heavily influenced by rare occurrences. Additionally, the paper investigates the capabilities and constraints of perplexity as a tool for model assessment.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC