Title: Equilibrated Diffusion: Frequency-aware Textual Embedding for Equilibrated Image Customization

Abstract:

Image customization techniques typically learn to recognize target subjects from reference images, subsequently generating new visuals guided by text prompts that primarily alter styles or backgrounds. While existing state-of-the-art methods rely on fine-tuning to consolidate various concept attributes into a single latent embedding, this approach often results in entangled features that make it difficult to isolate and remove irrelevant disturbances related to style or background. To overcome these limitations, we introduce Equilibrated Diffusion, a frequency-based methodology designed to separate mixed concept features, thereby ensuring balanced customization and strong alignment between text and visual outputs.

In contrast to traditional approaches that learn complete concepts using shared embeddings and unified tuning processes, our method leverages the intrinsic relationship between image frequency components and semantic meaning. Specifically, we recognize that low-frequency data corresponds to subject content, whereas high-frequency data aligns with stylistic elements. By decomposing concepts within the frequency domain and optimizing each embedding independently, we allow the denoiser to capture stylistic attributes distinct from subject identity. This separation enhances the model’s ability to generalize effectively toward unseen stylistic prompts. Furthermore, combining embeddings across multiple frequencies maintains the model’s inherent capacity for spatial customization.

To further refine the output, we incorporate mask-guided diffusion, which limits unintended background alterations and improves textual alignment. Additionally, Residual Reference Attention (RRA) is integrated into the spatial attention mechanism to preserve the structural integrity and identity consistency of the subject. Experimental results demonstrate that Equilibrated Diffusion outperforms leading baseline models in both subject fidelity and text adherence, confirming the effectiveness and superiority of our proposed approach.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC