Title: FeynmanBench: Assessing Multimodal LLMs on Diagrammatic Physics Reasoning

Abstract: Existing multimodal benchmarks for scientific reasoning largely focus on the extraction of localized information, where models identify specific symbols and numerical values before engaging in textual inference. However, these assessments typically overlook the model’s ability to reason regarding the global structural attributes of formal diagrams, including topological features, conservation constraints, and the coherent alignment between visual patterns and algebraic expressions. To address this gap, we present FeynmanBench, a comprehensive benchmark comprising more than 2,000 tasks centered on Feynman diagrams that cover the electromagnetic, weak, and strong interactions within the Standard Model. Each task pairs a diagram image with concise textual conventions, challenging models to reconstruct the entire physical context—encompassing vertex inventories, propagator classifications, topological connectivity, momentum routing, and the complete scattering amplitude. We employ an automated pipeline for generating and verifying diagrams, annotations, and reference answers in accordance with standardized protocols. Our evaluation of 19 leading multimodal LLMs reveals a distinct performance disparity: while models demonstrate strong capabilities in local recognition (achieving 70–95% accuracy in identifying vertices and propagators), their performance drops significantly to 13–17% during topological reconstruction (CP3) and falls to near-zero levels for full algebraic derivation (CP5). FeynmanBench serves as a controlled environment for testing multimodal reasoning over formal scientific diagrams, underscoring the fundamental architectural limitations of current models in handling topology-sensitive scientific reasoning.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC