Title: Evaluating the Evaluation: Score-Direction Instability in Class-Split Anomaly Detection

Abstract:

Class-split evaluation within a dataset is frequently employed as a stand-in for fully unconditional out-of-distribution anomaly detection. However, we demonstrate that this protocol becomes ill-posed when the reserved anomaly class intersects with the normal data distribution in representation space. Under these conditions, anomaly scores may regress to random chance levels or even reverse their expected direction, with the optimal score orientation relying on the identity of the unseen anomaly class. To address this, we propose a straightforward, training-free diagnostic metric known as neighborhood class leakage. Our findings indicate that this metric effectively forecasts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette, regardless of whether pixel or VAE latent spaces are utilized. These insights imply that class-split anomaly detection benchmarks should be interpreted as geometry-dependent stress tests rather than definitive proof of an anomaly detection system’s capabilities.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC