Title: PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA

Original: arXiv:2606.00944v1 Announce Type: new Abstract: Applying differential privacy (DP) via DP-SGD to Low-Rank Adaptation (LoRA) is a natural approach for privacy-preserving fine-tuning. However, LoRA's low-rank parameterization poses a fundamental challenge. In LoRA, each trainable update is represented as a low-rank matrix $Z = AB^\top$, but this factorization is inherently non-identifiable: many factor pairs $(A,B)$ represent the same update $Z$. As a result, applying DP-SGD directly to the factors induces gauge-dependent perturbations on $Z$, and we show that this naive DP-LoRA can lead to unbounded noise amplification. We propose PRISM, an intrinsic DP mechanism for LoRA that is gauge invariant by construction, avoids bilinear noise amplification, and admits an efficient low-dimensional noise sampler. Moreover, PRISM yields a closed-form characterization of the effective intrinsic noise induced on $Z$, enabling stable privacy-utility trade-offs through bounded, gauge-invariant perturbations. We establish standard $(\epsilon,\delta)$-DP guarantees for PRISM and introduce a DP-aware, gauge-invariant adaptive update rule that prevents adaptive optimization from amplifying injected privacy noise, improving numerical stability in practice.

Rewrite: arXiv:2606.00944v1 Announce Type: new Abstract: Leveraging DP-SGD within Low-Rank Adaptation (LoRA) offers an intuitive pathway for conducting privacy-preserving fine-tuning through differential privacy (DP). Nevertheless, the low-rank structure inherent to LoRA introduces a core difficulty. While LoRA models each trainable update as a low-rank matrix $Z = AB^\top$, this decomposition suffers from inherent non-identifiability, meaning numerous factor pairs $(A,B)$ can yield the identical update $Z$. Consequently, the direct application of DP-SGD to these factors results in gauge-dependent perturbations on $Z$; we demonstrate that this straightforward DP-LoRA strategy can trigger unbounded noise amplification. To address this, we introduce PRISM, an intrinsic DP mechanism designed for LoRA that is inherently gauge invariant. PRISM circumvents bilinear noise amplification and supports an efficient low-dimensional noise sampling process. Furthermore, PRISM provides a closed-form description of the effective intrinsic noise applied to $Z$, facilitating stable privacy-utility balances via bounded, gauge-invariant perturbations. We prove standard $(\epsilon,\delta)$-DP guarantees for PRISM and develop a DP-aware, gauge-invariant adaptive update rule. This rule mitigates the risk of adaptive optimization amplifying the injected privacy noise, thereby enhancing numerical stability in practical applications.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC