Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates
Title: Stabilizing Diffusion Sampling for Inverse Problems Through Resilient Prior Corrections
Abstract:
While diffusion models excel at generating plausible reconstructions for inverse problems, visual realism does not guarantee that the recovered details align with actual measurements. We identify this discrepancy as "measurement-conditioned hallucination," where the model introduces content that appears meaningful but is either physically implausible or contradicts the observed data. Our theoretical analysis decomposes Bayes-rule-based diffusion solvers into distinct prior update and measurement-conditioning phases, revealing that hallucinations often originate from the prior-side proposal before the measurement correction can mitigate them.
To address this, we introduce Robust Prior Update (RPU), a solver-level mechanism designed to assess the local stability of the diffusion prior update. RPU re-anchors the resulting displacement at the current iteration while keeping the measurement update process intact. We implemented RPU within the Diffusion Posterior Sampling (DPS) framework and evaluated its performance on FFHQ and ImageNet datasets for various inverse tasks, utilizing both automated metrics and human evaluation studies.
Quantitative results on FFHQ demonstrate that RPU enhances both PSNR and LPIPS scores compared to standard DPS across tasks including box inpainting, Gaussian deblurring, and motion deblurring. In human faithfulness assessments for FFHQ box inpainting, RPU achieved a 91.9% preference rate in blind non-tie comparisons and a 91.1% preference rate in ground-truth-assisted non-tie comparisons. Although the ImageNet Gaussian reader study yielded many ties, RPU was favored in non-tie instances. These findings substantiate the hypothesis that strengthening the robustness of the prior update significantly enhances instance faithfulness in diffusion-based inverse solvers, particularly when the prior dictates content that is only weakly constrained by measurements.
Source: arXiv Generated at: 2026-06-02 00:00:00 UTC
