Title: PrivacyPeek: Auditing LLM-Based Agents’ Data Acquisition, Not Just Their Output

Abstract:

Large Language Model (LLM) agents are evolving rapidly, increasingly capable of autonomously calling external tools to execute complex, multi-step tasks on behalf of users. However, a significant privacy risk emerges during this process: agents frequently gather sensitive data that exceeds the specific requirements of the assigned task. Current privacy evaluation frameworks primarily focus on auditing what agents reveal through their responses or outbound actions, largely ignoring the initial acquisition phase where data first enters the agent’s context. This excess information remains vulnerable, potentially leading to direct leaks through a single erroneous action or a targeted attack.

To investigate the extent of this issue, we present PrivacyPeek, a novel benchmark designed to evaluate privacy leakage specifically at the acquisition stage of LLM-based agents. The benchmark comprises 1,182 test cases spanning 16 application domains and seven distinct acquisition behaviors. PrivacyPeek employs two key mechanisms: Acquisition Inspection, which analyzes the agent’s tool-call trajectory—including both invoked tools and received data—to identify the unauthorized collection of sensitive information beyond the task’s scope; and Probe Elicitation, which tests the agent’s security by issuing follow-up queries to measure how easily an attacker could extract the acquired but undisclosed sensitive data.

Our experiments, conducted across 10 LLM-based agents representing four different model families, reveal that the unnecessary acquisition of sensitive information is pervasive. Furthermore, we identified a correlation between an agent’s ability to complete tasks and its propensity for acquisition-stage leakage. While prompt-level defenses were tested, they mitigated only a minor portion of the leakage, leaving most vulnerabilities unaddressed. These findings underscore the urgent need to audit acquisition-stage privacy. The dataset and source code are publicly accessible at https://github.com/Xuan269/PrivacyPeek-Resource.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC