Title: Enhancing Chain-of-Thought Reasoning in Multimodal Large Language Models via Attention-Guided Fine-Tuning

Abstract:

The utility of Chain-of-Thought (CoT) prompting within Multimodal Large Language Models (MLLMs) is currently under scrutiny. Empirical evidence from various visual reasoning benchmarks indicates that CoT prompting frequently results in inferior performance relative to direct prompting strategies. This study presents a comprehensive examination of CoT dynamics across three contemporary MLLM architectures, evaluated at different model scales and on datasets necessitating step-by-step visual verification.

Our investigation reveals two persistent failure patterns: the premature commitment to an answer and restricted access to direct visual tokens during the generation of reasoning steps. Furthermore, we observe that conventional CoT-style Supervised Fine-Tuning (CoT-SFT) offers only partial resolution to these challenges. In many instances, standard CoT-SFT exacerbates the model’s dependence on textual priors while diminishing its reliance on counterfactual visual information.

Building on these insights, we introduce Attentive-CoT (Att-CoT), a novel fine-tuning objective guided by attention mechanisms. This approach is designed to foster CoT trajectories that postpone final answer determination and preserve continuous access to visual tokens. Notably, Att-CoT can be integrated into existing CoT-SFT training pipelines without necessitating any architectural modifications. Our experimental results, conducted across six MLLMs on three distinct visual reasoning benchmarks, demonstrate that Att-CoT significantly boosts CoT performance compared to standard fine-tuning methods.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC