Title: LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

Abstract:

We present LLM-Wikirace, a new benchmark designed to assess the planning, reasoning, and world knowledge capabilities of large language models (LLMs). This task requires models to traverse Wikipedia hyperlinks sequentially, moving from a specified starting page to a target destination. Success demands not only look-ahead planning but also an understanding of how real-world concepts interconnect. We tested a wide array of both open- and closed-source models, including top-tier systems like Gemini-3, GPT-5, and Claude Opus 4.5. These leading models delivered the best performance on the easier difficulty levels, even achieving superhuman results. However, their performance declined significantly when faced with harder challenges: the top-performing model, Gemini-3, managed to win only 23% of the hard-level games, underscoring persistent difficulties for state-of-the-art systems. Our investigation indicates that while world knowledge is essential for success, it has a limited impact; once a certain threshold is reached, long-horizon reasoning and planning abilities become the primary determinants of performance. Furthermore, an analysis of model trajectories shows that even the most capable models find it difficult to adjust their strategies after making errors, often falling into repetitive loops instead of recovering. LLM-Wikirace serves as a straightforward yet revealing test of current reasoning limitations, providing an open platform where models with strong planning skills still have significant room for improvement. Code and the leaderboard are available at https:/llmwikirace.github.io.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC