Title: Visual Persuasion: What Drives the Choices of Vision-Language Models?

Abstract:

The internet is saturated with imagery originally intended for human viewing, yet these visuals are now increasingly analyzed by AI agents powered by vision-language models (VLMs). These systems execute visual judgments at massive scales, determining which items to click, suggest, or purchase. Despite this growing reliance, our understanding of the underlying structure governing their visual preferences remains limited. To address this gap, we present a framework that investigates these preferences by subjecting VLMs to controlled, image-based selection tasks and systematically altering their input data.

Our core concept involves treating the agent’s decision-making process as a latent visual utility function, which can be deduced through the lens of revealed preference—specifically, by analyzing choices made between systematically modified images. Beginning with standard visuals like product photographs, we introduce techniques for visual prompt optimization. This approach adapts methods from text optimization to iteratively generate and apply visually plausible changes—such as adjustments to lighting, composition, or background—using image generation models. We subsequently assess how these edits influence selection likelihood.

In extensive experiments involving state-of-the-art VLMs, we show that optimized edits cause significant shifts in choice probabilities during direct comparisons. Furthermore, we have built an automatic interpretability pipeline designed to elucidate these preferences by pinpointing consistent visual themes that drive selection. We contend that this methodology provides a practical and efficient mechanism for uncovering visual vulnerabilities. By revealing safety concerns that might otherwise remain hidden until discovered incidentally in real-world scenarios, this approach facilitates more proactive auditing and governance of image-centric AI agents.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC