Title: The Safety Mirage: How Spurious Correlations Sabotage VLM Safety Fine-Tuning and How Machine Unlearning Can Fix It

Abstract:

Vision-language models (VLMs) have recently achieved significant progress in generative modeling, especially regarding multimodal inputs such as images and text. Nevertheless, their tendency to produce harmful outputs when presented with unsafe prompts has sparked serious safety concerns. Although existing alignment methods largely depend on supervised safety fine-tuning using curated datasets, we uncover a critical flaw termed the "safety mirage." This phenomenon occurs when supervised fine-tuning inadvertently strengthens spurious correlations between superficial textual cues and safety-related responses, rather than cultivating a deep, intrinsic resistance to harm.

Our analysis demonstrates that these spurious correlations render fine-tuned VLMs susceptible to a straightforward one-word substitution attack. By replacing a single word in a text query with an alternative that triggers a spurious correlation, attackers can effectively circumvent safety safeguards. Furthermore, these correlations lead to excessive caution, causing the models to reject harmless queries unnecessarily. To resolve these challenges, we propose machine unlearning (MU) as a robust alternative to supervised safety fine-tuning. MU eliminates biased feature-label mappings and directly excises harmful knowledge from VLMs without compromising their general capabilities. Comprehensive evaluations across various safety benchmarks reveal that MU-based alignment lowers the attack success rate by as much as 60.27% and reduces unnecessary rejections by more than 84.20%.

WARNING: This content includes AI-generated material that may be offensive.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC