Title: Moving Beyond Idealized Instructions: A Holistic Framework for Assessing LLMs in Authentic Interactions

Abstract:

Although large language models (LLMs) have made significant strides in their ability to utilize tools, current evaluation benchmarks often fall short of mirroring actual real-world conditions. Most existing metrics are predicated on simulated, idealized user behaviors and lack assessments grounded in practical experience. Consequently, these frameworks overlook critical aspects of genuine user interactions, such as ambiguity, uncooperative conduct, and evolving intentions.

To address these shortcomings, we introduce RUT-Bench, a specialized benchmark engineered to evaluate LLMs across a variety of Real-world User Tool calling scenarios. RUT-Bench facilitates high-fidelity simulations that encompass both standard rational patterns and complex, non-ideal heterogeneous behaviors within both single-turn and multi-turn conversational contexts.

We applied this benchmark to conduct thorough assessments of 19 prominent open-source and proprietary LLMs. Our findings indicate that none of the tested models surpassed a 40% overall success rate. Furthermore, almost all models exhibited significant performance degradation when confronted with more intricate, non-ideal user inputs. The code and data associated with this study are publicly accessible at https://github.com/TorresYangX/RUT-Bench.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC