Title: Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

Abstract

Human annotation serves as the empirical bedrock for a significant portion of Natural Language Processing (NLP) research, spanning from the creation of datasets to the evaluation of models. However, publications frequently fail to clarify who performed the annotations or how the process was managed. This study presents the first extensive, task-level audit of annotation reporting across prominent NLP venues. We investigate which specific details regarding annotation are documented, what information is commonly absent, and how reporting standards fluctuate based on temporal trends, subject matter, publication venue, and the intended application of the human judgment.

To address these questions, we developed a unified taxonomy for annotation-reporting practices. We then validated an LLM-assisted extraction pipeline against Annotated-gold, a human-adjudicated reference set comprising 41 papers and 72 annotation tasks. The most effective model achieved agreement levels comparable to human adjudicators, recording a Krippendorff’s alpha of 0.606, which closely mirrors the human-human agreement score of 0.585.

Leveraging this pipeline, we compiled Annotated-llm, a comprehensive dataset derived from ACL-venue papers published between 2018 and 2025. This dataset contains 2,667 extracted annotation tasks sourced from 1,603 papers. Our analysis reveals that while authors commonly document operational specifics—such as recruitment methods, annotator qualifications, and total annotation volume—they frequently neglect critical details required to assess validity. Omitted information typically includes training protocols, language proficiency, compensation structures, socio-demographic data, adjudication processes, and inter-annotator agreement metrics, with these omissions being particularly prevalent in model-evaluation studies.

Our findings indicate that although annotation reporting in NLP has shown improvement over time, inconsistencies remain widespread. Consequently, we propose a scalable framework along with minimum reporting guidelines designed to enhance the reliability, reproducibility, and interpretability of human annotation in NLP research.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC