Title: Examining the Boundaries of LLM Adaptability: How Internalized Priors Influence Annotation Outcomes

Abstract

The deployment of Large Language Models (LLMs) in zero-shot annotation and LLM-as-a-judge scenarios is on the rise; however, their dependability is largely determined by the interplay between user instructions and the priors internalized by the model. This study explores three facets of this dynamic: the influence of an LLM’s prior exposure to data and task definitions on its efficacy, the degree to which supplementary prompt information can rectify zero-shot mistakes (a phenomenon termed "decision stickiness"), and the model's vulnerability to conflicting task definitions.

Our experiments, which utilized both dense and mixture-of-experts architectures to assess toxicity detection across a broad spectrum of datasets—including social media, gaming communities, news outlets, and forums—reveal significant constraints in prompt-based correction. Specifically, we observed that approximately two-thirds of zero-shot errors could not be corrected, resulting in a mere 34.8% overall rescue rate (the proportion of initial errors fixed through prompting). Notably, errors identified with high confidence were particularly difficult to overturn. Furthermore, when presented with definitions that conflicted with their internal priors, LLMs adhered to the provided definitions without any shift in their confidence levels compared to aligned conditions.

To better understand these dynamics, we propose Definition-Specific Familiarity (DSF), a metric quantifying the congruence between a model’s internal conceptual framework and the specific task definition. After accounting for dataset-level confounding variables, DSF demonstrated a positive correlation with model performance (partial r = +0.41). In contrast, three conventional memorization metrics—ROUGE-L, BERTScore, and embedding cosine similarity—failed to exhibit any positive association. These results underscore the inherent limitations of relying on prompts to correct errors in annotation tasks and emphasize the critical role of definition alignment over superficial text-level memorization.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC