Title: Moving Past Single Instances: Leveraging AI Agents for Continuous Learning in Field Experiments

Abstract:

While organizations frequently utilize A/B testing, the wealth of data produced by individual experiments is rarely leveraged to optimize future intervention strategies. Extracting actionable insights from historical experimental data to guide new initiatives faces substantial hurdles. This study investigates whether tool-enhanced, agentic AI can autonomously derive lessons from experimental datasets to formulate improved interventions for subsequent trials.

We conducted a two-stage field experiment focused on healthcare prescription messaging, encompassing a total of 693,139 patient visits. The research compares two methodologies: a Human + Chatbot approach in Stage 1 and a Tool-Augmented Agentic AI approach in Stage 2. In Stage 1, behavioral experts collaborated with conversational AI to co-design 13 message variants, involving 444,691 patient visits. In Stage 2, the AI system independently analyzed the Stage 1 data to extract underlying principles, generating 17 new message variants for a subsequent group of 248,448 patient visits.

The Agentic AI method, which utilized analytical tools, structured Data-Information-Knowledge-Wisdom (DIKW) reasoning agents, and transparent evidence chains, yielded superior intervention results. The top-performing AI-generated message achieved a click-through rate (CTR) of 69.8%, marking a 6.5 percentage point improvement over the baseline. Crucially, the findings indicate that the efficacy of this approach stems from domain-specific experimental data rather than general reasoning capabilities. Notably, frontier Large Language Models (LLMs) operating without access to experimental data were unable to accurately predict which interventions would succeed.

Furthermore, the field experiments demonstrated that general-purpose behavioral theories do not uniformly apply to specific healthcare contexts, highlighting the need for agentic AI to conduct theory audits at the scale of field experiments. Ultimately, this research demonstrates that tool-augmented AI can effectively learn from experimental data to produce enhanced, domain-relevant interventions, shifting behavioral experimentation from isolated, one-off evaluations to a scalable framework for cumulative design learning.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC