Title: The Case for Model Science: Verify, Explore, Steer, Refine

Abstract:

We contend that the artificial intelligence sector has reached a tipping point, necessitating a shift away from simple benchmarking toward a cohesive, systematic discipline for analyzing models, which we define as "Model Science." While complex AI systems now impact billions of users, our comprehension of their underlying mechanisms remains significantly behind our capacity to deploy them. For decades, research driven by benchmarks has yielded substantial advancements, characterized by comprehensive leaderboards, diverse performance metrics, and the tracking of capability improvements across various tasks. However, this approach has also exposed the inherent limitations of benchmarking: while it indicates whether a model performs, it fails to explain why it succeeds or fails. Crucially, benchmarks often overlook vital failure modes, such as hallucinations or the use of shortcuts.

Guidance for this new direction can be drawn from established scientific fields. Cognitive science illustrates that understanding complex systems demands analysis at multiple complementary levels. Neuroscience highlights that in-depth studies of individual cases can uncover insights that broad population studies miss. Medicine demonstrates that specialized training must evolve in tandem with research practices, while agriculture offers a model for how shared infrastructure and principles facilitate cumulative progress.

These insights from other disciplines underpin three core pillars of Model Science. First, we propose unifying research efforts around four functional perspectives—Verify, Explore, Steer, and Refine—which address distinct but complementary questions regarding model behavior. Second, we examine the infrastructure necessary for accumulating knowledge, specifically through the creation of catalogues for datasets, models, and research findings. Third, we emphasize the importance of conducting deep analyses on individual model instances rather than solely focusing on model families, as single-case studies can reveal nuances that broader studies overlook.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC