Title: Assessing Reliability Disparities in Chinese Factual Queries and AI Responses

Abstract: As search engines and artificial intelligence systems become the primary gatekeepers of factual information, assessing their trustworthiness within real-world information-seeking contexts proves challenging. This study addresses this issue within the Chinese web environment by developing a query-driven fact-checking dataset derived from actual Chinese search logs. We benchmark nine distinct systems, encompassing conventional search engines, standalone large language models, and AI Overviews integrated with search capabilities. Our analysis concentrates on factual Yes/No questions in Chinese, evaluating whether these systems deliver correct, incorrect, or uncertain responses relative to evidence-based ground truth.

The findings reveal that while systems exhibit comparable accuracy when issuing definitive answers, they vary significantly in their propensity to do so. Conditional accuracy spans from 73.2% to 78.9%; however, traditional search engines provide definitive answers for more than 83% of queries, whereas Qwen-Max does so for less than half. Additionally, a consistent polarity gap emerges, with all systems demonstrating superior performance on queries labeled "yes" compared to those labeled "no." By leveraging Baidu Index data, we also pinpoint Chinese provinces exhibiting heightened attention to health-related searches, suggesting these regions may face a greater risk of exposure to misinformation. Ultimately, our results indicate that reliability is determined not merely by the correctness of answers provided, but also by the frequency of responses, the handling of negative claims, and the geographical distribution of information demand that may amplify exposure risks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC