Title: WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

Abstract:

As Wikipedia establishes itself as a cornerstone of trustworthy, high-quality information, there is mounting anxiety regarding the influx of substandard machine-generated text (MGT) created by large language models (LLMs). Consequently, the ability to accurately identify MGT has become critical. Yet, current research largely assesses MGT detection tools using generic generation tasks, which diverges from the actual workflows of Wikipedia contributors. This disconnect often results in limited generalizability when these tools are deployed in authentic Wikipedia environments.

To address this gap, we present WETBench, a benchmark designed for MGT detection that is multilingual, supports multiple generators, and focuses on specific tasks. Grounded in the editing scenarios perceived by Wikipedia editors as valuable for LLM assistance, we outline three distinct editing tasks: Paragraph Writing, Summarisation, and Text Style Transfer. We operationalize these tasks through two newly created datasets spanning three languages. Our methodology involves testing three prompts for each writing task, generating MGT using the most effective prompt across various generators, and subsequently evaluating a range of detection models.

Our findings reveal that, across different settings, detectors relying on training data attain an average accuracy of 78%, whereas zero-shot detectors average only 58%. These outcomes highlight the difficulties detectors face when confronted with MGT in realistic generation contexts. Furthermore, they emphasize the necessity of evaluating such models against diverse, task-specific datasets to accurately gauge their reliability in environments driven by human editors.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC