Title: AgentDS Technical Report: Evaluating the Horizon of Human-AI Partnership in Specialized Data Science

Abstract:

Data science is essential for converting intricate datasets into actionable intelligence across a wide array of sectors. While recent advancements in large language models (LLMs) and artificial intelligence (AI) agents have driven significant automation within data science workflows, the precise extent to which AI can replicate the output of human experts in specialized domains remains an open question. Furthermore, the specific areas where human knowledge still holds an edge are not yet fully understood. To address these gaps, we present AgentDS, a comprehensive benchmark and competitive platform aimed at assessing the capabilities of standalone AI agents as well as human-AI collaborative efforts in domain-specific data science contexts.

AgentDS features 17 distinct challenges spanning six key industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. Through an open competition that attracted 29 teams and 80 participants, we facilitated a rigorous comparison between collaborative human-AI methodologies and AI-only baseline models. Our analysis reveals that existing AI agents face considerable difficulties with domain-specific reasoning. Specifically, AI-only baselines failed to reach the top quartile of participant performance, whereas the most effective solutions were consistently derived from human-AI collaboration. These outcomes refute the notion of total AI automation, highlighting the persistent value of human expertise in data science while pointing toward critical development paths for future AI systems.

For more information, visit the AgentDS website at https://agentds.org/ and access the open-source datasets at https://huggingface.co/datasets/lainmn/AgentDS.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC