Title: The Critical Role of Low-Probability Tokens in Detecting AI-Generated Content: A Multiscale Uncertainty Framework

Abstract:

As AI-generated content becomes increasingly indistinguishable from human writing, it poses significant practical threats, including the spread of misinformation, academic integrity violations, and the contamination of training corpora. Although statistical detectors are favored for their efficiency and broad applicability, they are hindered by two primary shortcomings. First, boilerplate dominance occurs when common tokens—shared by both human authors and Large Language Models (LLMs)—overpower the signals that distinguish machine-generated text. Second, brittle point estimates arise because relying on a single probability score leads to inconsistent decisions when faced with adversarial attacks.

To overcome these challenges, we introduce Uncertainty, a multiscale estimator designed to highlight informative, low-probability tokens that reveal distributional discrepancies more clearly. At a local level, this method mitigates the issue of boilerplate dominance by averaging the log-probabilities of these rare tokens. Globally, it enhances stability by characterizing the distributional shape of the low-probability region through Rényi entropy, thereby reducing brittleness. We further refine the detector into Uncertainty++ by incorporating conditional independent sampling, which provides even more robust uncertainty estimates. Empirical evaluations across seven datasets and sixteen LLMs confirm the approach’s high effectiveness, generalizability, and resilience. The source code is accessible at https://github.com/guoyikai2000/Uncertainty-AIGT.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC