Title: ProtStructQA: Establishing a Denotation Threshold in Protein Structural Reasoning

Protein-language models are typically assessed on their ability to generate biologically plausible text; however, structural questions carry more precise semantics, as they refer to specific measurements within a 3D coordinate system. To address this, we present ProtStructQA, an executable benchmark designed for protein structural question answering. In this framework, every natural-language query is derived from a hidden, typed domain-specific language (DSL) program, with the correct answer determined by running that program on structures predicted by AlphaFold.

The ProtStructQA dataset comprises 382,200 questions spanning various categories, including confidence scores, distances, predicted aligned error (PAE), solvent exposure, secondary structure, topology, and contacts. This corpus is divided into a 330,000-item active benchmark featuring 10,000 proteins from four different species, and a 52,200-item hard-negative robustness pool.

We evaluated Qwen3 models ranging from 0.6B to 8B parameters without any fine-tuning. The assessment covered direct prompting, chain-of-thought reasoning, grammar-constrained executable voting, executable voting combined with chain-of-thought, and multi-turn ReAct-style tool use. We also replicated these findings using Gemma-3-1B and Gemma-3-12B.

Our analysis identifies a capability-dependent denotation threshold between Qwen3-1.7B and Qwen3-4B. For models below this threshold, tool-mediated ReAct strategies dominate because the models frequently fail to generate executable denotations. Conversely, for models above this threshold, chain-of-thought shifts from being largely detrimental to significantly advantageous, emerging as the most effective strategy across most evaluation splits.

Further analysis of parse failures and family-level performance indicates that this threshold represents a transition from unparseable language to executable structural denotation. Additionally, we observed that grammar and execution constraints retain selective value specifically for PAE and secondary-structure queries. Ultimately, ProtStructQA repositions scientific question answering as a process of compiling language into measurement, offering a diagnostic testbed to determine when language models can successfully map textual inputs to executable 3D structural metrics.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC