Title: From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

Abstract:

As large language models (LLMs) become more prevalent as chemistry assistants, current evaluation methods remain limited by their focus on final outputs. Most existing benchmarks assess only the correctness of the final answer, which obscures a significant vulnerability: a model might produce the right molecule, product, or option selection while employing reasoning that contradicts fundamental chemical logic. Traditional process-level evaluators struggle with scalability due to the high costs and inconsistencies associated with human step-level annotation and LLM-based judging, both of which are prone to hallucination.

To address these challenges, we present ChemCoTBench-V2, a diagnostic benchmark designed for low-cost, auditable evaluation of structured chemical reasoning traces that are addressable by verifiers. This benchmark covers molecular understanding, molecule editing, molecular optimization, and reaction prediction, comprising 5,620 samples distributed across 18 distinct reporting tasks. The framework requires models to provide key intermediate steps within templates designed by experts. These steps are validated using deterministic chemistry rules and, for closed-answer tasks, reference traces, thereby eliminating reliance on another LLM judge. For open-ended molecular optimization, evaluation relies on oracle-verifiable state constraints rather than strict trace matching.

ChemCoTBench-V2 generates three distinct metrics: final-answer accuracy, adherence to templates, and step-wise verifier correctness based on expert-refined intermediate commitments. Our experiments with frontier models highlight a persistent disconnect between success in final answers and consistency in structured reasoning states. Specifically, models frequently adhere to the required format while failing chemical-step validations, or they arrive at the correct answer despite possessing weak supporting reasoning. By identifying the precise step where a reasoning trace first violates verification rules, ChemCoTBench-V2 facilitates fine-grained model comparisons.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC