Title: GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory

Abstract

While large language models (LLMs) are becoming prevalent as self-study aids in technical fields, their trustworthiness as assistants for mathematical reasoning is not yet well characterized. To address this gap, we present GTBench, a novel benchmark grounded in educational curricula designed to assess the capabilities of LLMs as research partners in graph theory. This benchmark features 63 distinct problems structured into three tiers of escalating complexity: Group 1 covers undergraduate-level definitions and fundamental properties; Group 2 focuses on algorithm tracing and structural reasoning; and Group 3 involves graduate-level proof construction. The problem set is drawn from authoritative academic sources, including Diestel’s Graph Theory.

We assessed five leading models—GPT-5, Claude Sonnet 4.6, Gemini 2.5 Flash-Lite, Llama 3.3 70B, and Mistral Large 3—using both zero-shot and chain-of-thought prompting strategies. Evaluation methodologies included exact-match scoring and LLM-as-judge assessments for Groups 1 and 2, while Group 3 utilized a hybrid protocol combining human expert review with LLM judging.

The findings indicate a significant performance disparity among the models. GPT-5 demonstrated near-ceiling performance on Group 1 tasks (achieving 95.8% accuracy in zero-shot settings) and retained substantial proficiency in graduate-level proofs (82% accuracy). In contrast, all other models experienced a marked decline in performance as difficulty increased. Notably, Llama achieved a 0% score on Group 3 under zero-shot conditions when evaluated by human experts.

An analysis of failure modes revealed that "correct algorithm, wrong execution" errors were the most common in Groups 1 and 2. Group 3 introduced additional failure types, such as incomplete reasoning, and highlighted a systematic divergence between human evaluators and the automated judge, particularly regarding verbose or nearly complete proofs (inter-rater reliability, kappa = 0.48–0.83 across human pairs). GTBench establishes the first curriculum-based evaluation framework for graph-theoretic reasoning in LLMs, offering critical insights for the regulation of AI tools in both mathematical education and scientific inquiry.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC