Title: Moving Past Offline A/B Testing: Context-Aware Agent Simulation for Evaluating Recommender Systems

Abstract: Recommender systems serve as the backbone of digital platforms, helping users navigate vast content libraries across diverse categories. Despite their importance, evaluating these systems remains difficult because of the frequent misalignment between offline performance metrics and actual online outcomes. While Large Language Model (LLM) agents present a potential remedy, current research often models users in isolation, overlooking crucial contextual elements like time, location, and intent that drive human decision-making. To address this, we present ContextSim, an LLM-based agent framework that creates realistic user proxies by grounding interactions in everyday activities. A dedicated life simulation component generates scenarios that define the specific when, where, and why of user engagement with recommendations. To ensure the agents’ preferences mirror those of real people, we incorporate internal reasoning processes and maintain consistency across both individual actions and broader behavioral trajectories. Our experiments demonstrate that ContextSim produces user interactions that more closely resemble human behavior compared to previous methods. Furthermore, we validate the framework’s effectiveness through offline A/B testing correlations, revealing that recommender system parameters optimized via ContextSim lead to enhanced engagement in live environments.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC