Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Abstract

Evaluating Large Language Model (LLM) judges against human annotations typically involves presenting a suite of agreement statistics, including accuracy, precision, recall, $F_1$ scores, Cohen’s $\kappa$, and various rank correlations. However, an analysis of 24 recent studies on LLM-as-judge frameworks reveals that the selection of metrics is often confounded by factors such as the judgment scale, strategies for handling ties, invalid outputs, and abstention protocols—details that are frequently omitted from reports.

In the context of binary criteria, which are prevalent in rubric-based evaluations where each criterion is rated as either MET or UNMET, many reported metrics are mathematically redundant. For non-degenerate binary data, Pearson’s $r$, Spearman’s $\rho$, Kendall’s $\tau_b$, the phi coefficient $\phi$, and the Matthews Correlation Coefficient all converge to a single value. Consequently, presenting multiple such metrics creates a false impression of corroborating evidence. In contrast, Cohen’s $\kappa$ provides distinct information. While it shares the same numerator as the phi coefficient, its normalization method differs; the disparity between $\kappa$ and $\phi$ specifically quantifies the deviation of the judge’s positive-label frequency from that of the human annotator.

The dynamics shift when a judge is permitted to issue a CANNOT_ASSESS verdict. The three prevalent methods for managing abstentions are not merely interchangeable preprocessing steps; rather, they address different analytical questions and disrupt the binary equivalencies observed in non-abstaining scenarios. Interestingly, these equivalencies re-emerge in multi-judge ensembles evaluated using Fleiss’ $\kappa$ or Krippendorff’s $\alpha$, with only negligible corrections for finite sample sizes.

To improve transparency, we propose a reporting checklist that mandates the disclosure of the judgment scale, the specific modes for handling ties and abstentions, coverage rates, the confusion matrix, and the aggregation level, in addition to any scalar agreement coefficient.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC