Title: Best Practices for Reporting LLM-as-a-Judge Assessments

Abstract:

Large language models (LLMs) have become a popular, scalable alternative to human annotators for evaluating model outputs. However, the inherent imperfections in an LLM judge's sensitivity and specificity can introduce significant bias into straightforward evaluation metrics. To address this, we introduce a straightforward, plug-in framework designed to correct this bias while facilitating statistically robust uncertainty quantification.

Our approach generates confidence intervals that incorporate uncertainty stemming from both the primary test dataset and a secondary calibration dataset labeled by humans. Furthermore, the framework employs an adaptive mechanism to distribute calibration samples strategically, resulting in narrower confidence intervals. Crucially, we identify specific parameter regimes—determined by the actual evaluation score alongside the judge’s sensitivity and specificity—where LLM-based assessments provide more reliable estimates than evaluations conducted by humans alone. Additionally, unlike current methods, our framework demonstrates resilience to distribution shifts between the test and calibration datasets, ensuring unbiased performance.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC