Title: Establishing Benchmarks for Security Risk Detection and Verification in Open Agentic Skill Ecosystems

Abstract:

Open agent platforms enable community members to publish reusable skills that agents can invoke during runtime. While this extensibility offers significant benefits, it introduces supply-chain vulnerabilities, as malicious actors may conceal harmful behaviors within skills that appear harmless upon superficial review. Current defensive measures are difficult to assess due to the lack of a benchmark that evaluates both the detection of malicious skills and their runtime verification. To address this gap, we introduce SkillVetBench, a two-stage security vetting benchmark designed for open agentic skill ecosystems. The initial stage conducts semantic vetting on the natural-language specifications of each skill to uncover hidden malicious intent. The subsequent stage executes flagged skills within an instrumented sandbox to monitor runtime behavior and gather auditable evidence.

The benchmark is constructed using confirmed malicious skills from the live OpenClaw ecosystem, incorporating samples from the recent ClawHavoc supply-chain campaign. In contrast to static-only approaches, SkillVetBench validates detected threats through execution traces. Our experimental results demonstrate three key findings: (1) semantic-only and signature-based baseline methods are inadequate, failing to detect up to 89% of malicious skills where threats stem from natural-language instructions, multi-component logic, or cross-component interactions; (2) runtime attacks are predominantly concentrated in a limited number of high-permission primitives, specifically exec, write_file, install_skill, and spawn; and (3) SkillVetBench offers case studies where sandbox execution provides concrete runtime evidence that directly supports malicious verdicts.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC