Title: Assessing Structural Reasoning in LLMs: A Benchmarking Approach Through Data Structures

Abstract:

As large language models (LLMs) are increasingly applied to complex tasks demanding multi-step decision-making, it becomes essential to comprehend their capacity for algorithmic reasoning. Currently, however, there is a notable absence of diagnostic benchmarks designed to evaluate these specific skills. To address this gap, we propose utilizing data structures as a rigorous framework for assessment. Because data structures serve as the foundational components of algorithms, they effectively probe structural reasoning—the capability to comprehend and manipulate underlying relationships, including order, hierarchy, and connectivity, which are critical to algorithmic logic.

We present DSR-Bench (Data Structure Reasoning Benchmark), a comprehensive evaluation tool encompassing 20 distinct data structures, 35 operational types, and 4,140 problem instances. DSR-Bench is characterized by its hierarchical task organization, fully automated generation and evaluation processes, and capacity for fine-grained diagnostic analysis. Our evaluation of 13 leading state-of-the-art LLMs highlights significant constraints: even the highest-performing model attained a score of just 0.46 out of 1 on difficult instances. Furthermore, three auxiliary probes designed to reflect more practical applications reveal additional deficiencies, showing that models struggle with spatial data, perform poorly in context-rich environments, and face challenges when reasoning about their own code.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC