Title: Assessing the Reliability of Benchmarks in Causal Discovery

Abstract: Causal discovery within graphical causal models seeks to build causal graphs by integrating numerical datasets with textual domain knowledge. However, evaluating these methods presents a persistent difficulty, as advancements in field-specific research frequently result in benchmark causal graphs containing outdated or conflicting information. This issue is particularly pronounced for large language model (LLM)-based causal discovery techniques, which are highly responsive to recent academic findings. This study represents the first systematic investigation into the integrity of benchmark causal graphs. To address this, we developed a workflow that automatically fetches pertinent research papers from scientific repositories and utilizes LLMs to verify the alignment between benchmark graphs and current domain literature. We applied this pipeline to 11 widely used real-world benchmarks, processing a total of 38,081 domain papers. Our findings reveal substantial discrepancies in how well these popular benchmarks align with existing research, highlighting significant consequences for the future of causal discovery studies.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC