Title: SeClaw: Specification-Driven Security Task Synthesis for Evaluating Autonomous Agents

Abstract:

Autonomous LLM agents are increasingly deployed in stateful environments, granting them access to external services, memory, files, and various tools. While these capabilities facilitate the execution of complex real-world workflows, they simultaneously introduce security vulnerabilities that current evaluation methods struggle to capture. Existing benchmarks for agent security typically depend on manually curated tasks, offer insufficient coverage of emerging threats, and concentrate mainly on final outcomes rather than the execution processes that precipitate unsafe behavior.

To address these limitations, we present SeClaw, a framework that integrates specification-driven security task synthesis with execution-based security evaluation for autonomous agents. This framework allows for the scalable and controlled generation of security tasks from structured risk specifications. Additionally, the SeClaw Docker container serves as a standardized testbed for assessing agent behavior across a wide array of safety-risk scenarios.

The benchmark encompasses risks stemming from intrinsic agent behaviors, environments, user tasks, and resources. It also facilitates trajectory-aware assessment, enabling the detection of unsafe actions throughout the execution process, rather than solely analyzing final responses. By connecting systematic task synthesis with reproducible security evaluation, SeClaw establishes a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The source code is publicly available at https://github.com/seclaw-eval/seclaw-eval.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC