Title: DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration

Abstract:

Real-world professional tasks within specialized engineering and creative software environments typically span extended periods and demand close human-in-the-loop coordination. In these scenarios, agents must actively solicit required information, while users contribute ongoing instructions, clarifications, feedback, or corrections as the project evolves. However, current desktop GUI benchmarks largely oversimplify this dynamic by reducing it to brief, isolated tasks where all user directives are provided in advance.

To bridge this gap, we present DeskCraft, a novel benchmark designed to evaluate desktop GUI agents on long-horizon creative and engineering workflows, with a specific focus on proactive human-agent collaboration. DeskCraft classifies tasks using a multilevel difficulty taxonomy, featuring complex workflows that exceed 50 execution steps. It encompasses a wide array of professional tools used in design, video, audio, and 3D creation.

Additionally, DeskCraft establishes a formal interaction protocol for human-agent collaboration, encompassing both mid-turn and post-turn exchanges. Mid-turn interactions address agent-initiated clarifications when uncertainty arises, as well as user-initiated interruptions during execution. Post-turn interactions allow for user-driven feedback following the agent’s completion signal, thereby capturing the full spectrum of realistic collaborative patterns.

We assessed 18 proprietary and open-source agents across 538 tasks. The results indicate that GPT-5.4 achieved a score of 31.6% on standard tasks and 27.6% on interactive tasks. Further analysis highlights enduring challenges in executing long-horizon workflows and providing proactive clarifications. All evaluation code, tasks, and data will be made publicly available at https://github.com/mrwwk/DeskCraft.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC