Title: Evading Production Prompt Guards via Controlled-Release Prompting

Abstract:

Recent work by Ball et al. demonstrated that prompt filtering for AI alignment encounters a fundamental theoretical hurdle: assuming standard cryptographic premises, no filter that operates significantly faster than the protected model can universally differentiate between adversarial and benign inputs. This study explores whether this theoretical limitation manifests as tangible security flaws in live large language model (LLM) deployments. We confirm this vulnerability by presenting "controlled-release prompting," a practical application of the theoretical framework that leverages the computational disparity between lightweight input filters and the robust models they are designed to protect. In contrast to purely theoretical models, our method does not necessitate altering the target model. Instead, it crafts malicious prompts that remain opaque to any filter with bounded resources, yet remain fully understandable to the intended LLM. Our experiments show the attack succeeding on four prominent chat platforms—Google Gemini, DeepSeek Chat, xAI Grok, and Mistral Le Chat—where conventional defenses prove ineffective. Furthermore, we demonstrate the extraction of copyrighted material from Gemini using this technique. Finally, we conduct a comprehensive assessment of 14 open-weight prompt guard models, concluding that even filters equipped with reasoning capabilities fail to consistently identify these attacks without imposing unacceptable resource costs.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC