Title: RealClawBench: Capturing Live OpenClaw Benchmarks from Actual Developer-Agent Interactions

Abstract

While agent benchmarks aim to mirror the actual queries users pose to deployed systems, current evaluation methods frequently fail to capture the essential realism found in genuine developer-agent interactions. To bridge this gap, we present RealClawBench, a live benchmarking framework derived from authentic OpenClaw sessions. This approach is designed to encapsulate the distribution, variety, and real-world complexity inherent in deployed agent usage.

Benchmarking real-world user requests presents significant hurdles, as these interactions often rely on specific local execution environments, contain implicit or poorly defined intents, and demand complex verification processes. RealClawBench overcomes these obstacles through two primary mechanisms: reconstructed execution environments and deterministic, verifiable scorers. Together, these tools transform actual sessions into reproducible tasks that can be automatically evaluated.

The resulting dataset comprises 281 executable tasks, carefully sampled from a much larger pool of real sessions. This sampling preserves the original source distribution, maintaining a maximum final-versus-source Jensen-Shannon divergence of just 0.0448. Our evaluation of 14 contemporary models reveals that even the top-performing system solves only 65.8% of these tasks, highlighting significant room for improvement when addressing realistic developer-agent workloads. By converting live deployed sessions into controlled evaluation instances, RealClawBench offers a viable route toward benchmarks that more accurately reflect agent capabilities in practical applications. The code is accessible at: https://anonymous.4open.science/r/real-claw-bench-582B.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC