Title: Prompt-Aware Weighting for Training-Free Multi-Concept LoRA Composition

Abstract:

Low-Rank Adaptation (LoRA) has proven effective in enabling personalization within text-to-image generation, allowing pre-trained diffusion models to be tailored to distinct visual concepts and styles. Nevertheless, scaling these models to accommodate multiple concepts simultaneously presents significant difficulties. Simply merging multiple LoRA weights or their respective outputs frequently causes concept interference, which compromises visual fidelity and diminishes the accuracy of individual concept representations relative to their reference images.

To address this, we introduce a straightforward yet potent methodology for multi-concept customization that optimally integrates the outputs of various LoRA modules. Our approach capitalizes on the relative significance of each concept during the generation process, deducing this importance from the associated prompt tokens. We present two novel techniques, W-Switch and W-Composite, which utilize a prompt-aware weighting strategy. In this framework, each LoRA is assigned a weight based on the semantic impact of its trigger words within the specific target prompt.

Furthermore, we enhance current quantitative evaluation standards by introducing a new image-based similarity assessment framework. This framework evaluates both image fidelity and identity preservation by comparing real-world reference images against concept regions automatically segmented from the generated outputs. We tested our method on the ComposLoRA testbed, showing consistent advancements over current state-of-the-art techniques regarding visual quality, identity retention, and compositional accuracy. Additional qualitative assessments, including evaluations by a Large Language Model (LLM) and a user study, corroborate the efficacy of our proposed methods and align with the newly established quantitative metrics. Our code is accessible at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC