Title: TravelEval: A Holistic Benchmarking Framework for Assessing LLM-Driven Travel Planning Agents

Abstract

While Large Language Models (LLMs) have markedly enhanced travel planning applications, current evaluation methods remain constrained by significant shortcomings. Existing benchmarks tend to prioritize constraint adherence while overlooking multi-dimensional factors such as spatio-temporal costs. Furthermore, they often rely on datasets that lack real-world authenticity and insufficiently cover essential sectors like accommodation and transportation. Additionally, traditional assessments typically evaluate daily itineraries in isolation, failing to account for critical details—such as the influence of lodging choices and visit pacing—that are necessary for a comprehensive evaluation of an entire travel plan.

To bridge this gap, we present TravelEval, a robust and realistic benchmarking framework. TravelEval introduces three key innovations: first, a novel six-dimensional evaluation framework that holistically assesses travel plans across accuracy, compliance, temporality, spatiality, economy, and utility; second, a high-fidelity data sandbox featuring precise accommodation pricing and authentic intercity transportation information; and third, a simulation-based global evaluation method that replicates complete travel itineraries using API-integrated geographic data and detailed queuing times.

Our evaluation of 12 mainstream LLM approaches using TravelEval yields several critical insights. The results indicate that LLMs face considerable challenges in executing globally optimized, multi-dimensional planning, particularly in areas requiring spatio-temporal reasoning and strict budget adherence. Moreover, the study finds that agentic reasoning strategies do not consistently yield performance improvements. In summary, TravelEval enables rigorous travel plan assessment through grounded spatio-temporal simulation and comprehensive metrics, establishing a solid foundation for the further advancement of LLM-based travel planning research and applications.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC