Title: The Ghost Annotator: A Conformal Prediction Framework for Investigating Human Label Variance in Content Moderation

Abstract:

While the majority of contemporary research prioritizes model performance, uncertainty estimation—especially within contexts where Large Language Models (LLMs) are increasingly employed to produce annotated data—has received comparatively less scrutiny. To address this gap, we present a novel framework that integrates conformal prediction with Collaborative Filtering-style representations of annotators. This approach allows us to model LLM behavior relative to human annotators and to systematically analyze patterns of both agreement and disagreement.

By leveraging Non-Conformity Scores, we propose two key innovations: the "Ghost Prediction" metric and the "Ghost Annotator" representation. These tools are designed to quantify instances where model predictions deviate significantly from all existing human annotations. Furthermore, we utilize cosine similarity measures to investigate variations in model behavior across various sociodemographic dimensions.

Our evaluation involved four LLMs, spanning different sizes and families, across four distinct content moderation datasets. The results indicate that while uncertainty rises for all models in response to annotator disagreement, larger models exhibit heightened confidence when classifying texts that do not align with any human annotation. Ultimately, the Ghost Annotator framework exposes a consistent and robust pattern of demographic misalignment, pointing toward structural biases that likely originate in the pretraining corpora.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC