Title: Benchmarking at the Edge of Comprehension

Abstract

As frontier Large Language Models (LLMs) rapidly saturate new benchmarks shortly after their release, the field of evaluation faces a critical inflection point. Should these models continue to advance, it will grow increasingly difficult for humans to construct discriminative tasks, establish accurate ground-truth answers, or assess complex solutions. If such evaluation becomes unfeasible, our capacity to measure any meaningful progress in artificial intelligence is jeopardized. We define this scenario as the "post-comprehension regime."

To address this challenge, we introduce Critique-Resilient Benchmarking, an adversarial framework designed to compare models even when complete human understanding is unattainable. This approach is grounded in the concept of critique-resilient correctness, wherein an answer is considered correct unless an adversary can successfully prove otherwise. In contrast to traditional benchmarking methods, humans act as bounded verifiers who concentrate on specific, localized claims. This strategy maintains the integrity of the evaluation process, even when the full scope of the task exceeds human comprehension.

By employing an itemized bipartite Bradley-Terry model, we simultaneously rank LLMs based on their proficiency in solving difficult tasks and their ability to generate challenging yet solvable questions. We demonstrate the efficacy of our method within the mathematical domain, testing it against eight frontier LLMs. The resulting scores exhibit stability and show a strong correlation with external capability measures. Ultimately, our framework reimagines benchmarking as an adversarial game of generation and evaluation, with humans serving as the final arbiters.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC