Title: Routing vs. Encoding: Dissecting the Disparity in Scientific Claim Verification Between Tables and Charts

Abstract:

As multimodal large language models (LLMs) become more prevalent in supporting scientific peer review, a critical necessity is the ability to verify whether textual claims are substantiated by a paper’s evidence. Previous studies have indicated a significant performance disparity: models excel when evidence is presented in tabular format but struggle when the identical underlying data is displayed as a chart. This discrepancy prompts a fundamental inquiry: do models fail to extract information from charts entirely, or do they successfully extract the data but subsequently fail to utilize it during the prediction phase?

To investigate this, we employed layer-wise linear probing and attention analysis on three open-weight vision-language models (VLMs), comparing their processing of table versus chart evidence derived from the same data. Our results provide consistent support for the latter hypothesis. We observed that while chart information is successfully encoded within the models’ intermediate representations, it fails to propagate to the prediction layer. This "routing gap" is not present in the processing of tables and remains consistent across all tested conditions. Furthermore, attention analysis indicates that this disconnect manifests in two architecturally distinct ways depending on the model family. Consequently, these findings reinterpret the table-chart performance gap not as a deficiency in visual encoding, but as a failure in routing encoded visual information to the final prediction stage.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC