Title: KG-FairDiff: Leveraging Knowledge Graphs for Demographically Equitable Text-to-Image Generation via Prompt Refinement

Abstract:

Text-to-Image (TTI) systems have evolved into essential tools for public communication, advertising, education, and journalism. However, the demographic and cultural stereotypes embedded in their training data—such as the under-representation or caricature of women, people of color, older adults, and non-Western cultures—manifest as significant societal harm when these models are deployed at scale. Current mitigation strategies often fall short: they either demand expensive retraining, which is impractical for the closed-source backbones prevalent in consumer applications, or they depend on static demographic templates that overlook cultural nuances.

To address these challenges, we introduce KG-FairDiff, a model-agnostic framework that operates at inference time. This approach treats fairness-aware prompt refinement as a constrained optimization problem, executing it through a closed-loop pipeline. The process begins with a knowledge graph containing approximately 1,200 triples related to culture and bias, which retrieves structured context. An LLM-based rewriter then suggests refinements, while a validator ensures that only prompts reducing a divergence-based fairness loss are accepted, all while maintaining semantic fidelity to the user’s original intent.

We establish a finite-termination bound for this refinement loop and contribute a mathematically rigorous evaluation suite. This suite connects Bias-P and Bias-W metrics to divergence from target distributions, and ENS to KL divergence. Additionally, we conduct audits on eight widely used backbone generators. KG-FairDiff effectively minimizes disparities across gender, race, age, and intersectional categories without compromising prompt semantics, providing a practical and deployment-ready pathway toward more equitable generative AI.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC