Title: Stop Gambling, Start GAMBLe: A New Analytical Framework for AI-Driven Research Systems

Abstract

While AI-Driven Research Systems (ADRS)—which integrate Large Language Models (LLMs) with automated evaluation to uncover algorithms, proofs, and designs—are gaining traction and being optimized across various fields, the methodologies required to analyze them lag behind. The performance of ADRS is heavily influenced by complex interactions among its components, which are difficult to explore due to high costs and remain poorly understood. Furthermore, we demonstrate that these systems are not adequately described by standard convergence guarantees. These traditional guarantees depend on structural assumptions that fail to hold within the specific ADRS process we formalize.

To address this gap, we present GAMBLe, a novel framework that breaks down ADRS behavior into four distinct parameters: the generator ($G$), the assessor ($\mathcal{A}$), the discovery mechanism ($\mathcal{M}$), and the budget ($B$). Additionally, it introduces a compositional element known as the effective landscape, defined as $L_{\text{eff}} = \mathcal{A} \circ G$. This formulation highlights how different generator-assessor combinations create structurally unique optimization landscapes for individual problems.

We applied this framework to over 760 replicated runs, totaling more than 46,000 iterations. Our experiments covered a wide spectrum of generators, ranging from standalone LLMs to dynamically adaptive ensembles, and discovery mechanisms spanning from greedy selection to co-evolutionary meta-search. The study focused on three NP-hard problems, utilizing assessors that varied from continuous scoring functions to cliff functions.

The findings indicate that there is no universal hierarchy among generators or mechanisms. State-of-the-art frontier models sometimes perform worse than open-source alternatives, and basic mechanisms can surpass complex, state-of-the-art meta-search strategies. Notably, even with constrained budgets of just 60 iterations per run, selecting the appropriate components can boost performance by 13–67% and enhance search efficiency by a factor of 6 to 39.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC