Title: FALSIFYBENCH: Assessing Inductive Reasoning Capabilities in LLMs Through Rule Discovery Games

Abstract:

As large language models (LLMs) are increasingly utilized as autonomous agents in scientific endeavors, a critical question remains: Can these systems effectively perform the types of inductive reasoning essential to scientific discovery? To address this gap, we present FALSIFYBENCH, an evaluation framework designed to test hypothesis-driven reasoning. Drawing inspiration from the classic Wason 2-4-6 task, this framework requires agents to uncover hidden semantic properties through a cycle of proposing examples and receiving feedback. This process mirrors core components of scientific inquiry, including generating hypotheses, collecting evidence, and revising beliefs based on both confirming and disconfirming data.

We evaluated 12 LLMs spanning various model families and scales. Our findings indicate that reasoning models generally demonstrate superior scientific reasoning capabilities compared to instruction-tuned models, though none achieved optimal performance. A key factor in success was identified as the capacity for negative testing; models that proactively attempted to falsify their hypotheses significantly outperformed those focused primarily on seeking confirmation. Furthermore, our turn-level analysis, which had been overlooked in prior studies, highlights that model failures are linked to specific, identifiable patterns in how they traverse the hypothesis space.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC