Title: Evaluating the Essentials: Synthetic Benchmarks for Concept Bottleneck Models

Abstract:

Concept bottleneck models derive their predictions by analyzing high-level concepts identified within input data. While these concepts offer a straightforward path to leveraging interpretability, the scarcity of datasets containing concept labels poses a significant challenge. This shortage restricts researchers from identifying which problems are appropriate for such models, isolating the variables responsible for success or failure, or determining which algorithms yield the best results. To address this, we introduce synthetic benchmarks tailored for concept bottleneck models, specifically targeting their two primary applications: decision support, where models aid human judgment, and automation, where they execute routine tasks autonomously. These benchmarks allow for the generation of labeled datasets with precise control over key performance drivers, including data modality, the selection of concepts, and the quality and completeness of annotations. We illustrate the utility of these benchmarks in evaluating various classes of concept bottleneck models, demonstrating their capacity to diagnose failure modes and inform subsequent testing phases.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC