Title: UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding

Abstract:

Tasks that bridge vision and language, including Visual Question Answering (VQA), SNLI-VE, and Visual Commonsense Reasoning (VCR), present significant challenges due to the necessity for robust reasoning capabilities to interpret the semantics of both visual scenes and natural language. While supervised approaches for these tasks have been extensively investigated, their implementation within a zero-shot context remains underexplored. Given Contrastive Language-Image Pre-training (CLIP)’s demonstrated success in zero-shot image-text matching, earlier studies leveraged this strength by reframing vision-language problems as matching tasks, predominantly focusing on global-level alignment, such as comparing entire images against whole sentences. However, our analysis reveals that fine-grained details—such as specific keywords within text and distinct objects within images—provide crucial semantic insights. Drawing from this observation, we introduce a comprehensive framework designed to harness fine-grained data for zero-shot vision-language learning across various tasks, including VQA, SNLI-VE, and VCR. Our experimental results indicate that this approach surpasses existing zero-shot methods in VQA performance and yields significant gains in both SNLI-VE and VCR. Additionally, ablation studies validate the method’s efficacy and broad applicability.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC