Title: Enhancing Small Object Grounding in Large Vision Language Models Through Self-Improvement

Abstract:

Is it possible for Large Vision Language Models (LVLMs) to identify accurate bounding boxes for small objects using their internal attention mechanisms, without the need for fine-tuning? This study confirms that they are. We demonstrate that the attention architecture within LVLMs inherently captures grounding quality. Specifically, a lightweight Intersection-over-Union (IoU) regressor, trained exclusively on attention maps, is capable of strong IoU prediction, achieving a Pearson correlation coefficient greater than 0.67.

This regressor serves as the foundation for ACS-Learned, a variant of our Attention-based Candidate Selection (ACS) framework. ACS-Learned enhances object grounding by evaluating multiple sampled candidates and selecting the optimal box. Furthermore, an analysis of the regressor’s learned parameters allows us to identify the specific transformer layers and attention heads that are most influential. Leveraging these insights, we developed ACS-Free, a training-free selector that ranks candidates based on attention entropy within these discriminative heads, requiring no learned components during inference.

Our experiments conducted on the COCO and Objects365 datasets show that this approach can yield up to a 19% improvement in small object localization through self-improvement. Notably, ACS-Free outperforms all other training-free methods, underscoring the value of effective attention structures in boosting both the interpretability and reliability of localization in LVLMs.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC