Title: Attention, May I Have Your Decision? Localizing Generative Choices in Diffusion Models

Abstract: Although text-to-image diffusion models demonstrate impressive generative power, their internal mechanics remain largely obscure, especially when processing prompts that lack comprehensive detail. In these instances, the models are required to make implicit choices to fill in unspecified details. This study explores the premise that such decision-making processes are not scattered throughout the network but are instead computationally confined to specific architectural components. Although current localization methods primarily address prompt-related interventions, we observe that explicit conditioning does not necessarily mirror implicit decision-making. To address this, we present a probing-based localization strategy designed to pinpoint the layers exhibiting the highest attribute separability for specific concepts. Our results reveal that the disambiguation of vague concepts is primarily managed by self-attention layers, marking them as the optimal targets for intervention. Leveraging this insight, we introduce ICM (Implicit Choice-Modification), a precise steering mechanism that implements targeted interventions within a limited number of layers. Comprehensive experiments demonstrate that intervening in these specific self-attention layers achieves superior debiasing outcomes relative to current state-of-the-art techniques, while effectively reducing the artifacts typically associated with less precise methods. The source code can be accessed at https://github.com/kzaleskaa/icm.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC