Title: JudgmentBench: A Comparative Analysis of Rubric and Preference-Based Quality Assessment

Abstract:

Current benchmarking practices are primarily defined by two distinct approaches: rubric-based scoring, which assesses items against established criteria, and comparative judgment, which gathers pairwise preferences between different outputs. Despite the widespread adoption of both methods, there is seldom a rigorous justification for choosing one over the other. To address this gap, we introduce JudgmentBench, a comprehensive benchmark comprising 30 real-world legal tasks. This dataset includes 1,539 rubric scores and 1,530 pairwise preference judgments, all annotated by experienced practicing attorneys from major U.S. law firms. Notably, this represents the first publicly available dataset in a high-expertise field where both types of supervisory signals were collected from the same experts evaluating the same items.

In an initial empirical evaluation using LLM-generated outputs across three distinct quality tiers, we demonstrate that comparative judgments significantly outperform rubrics in recovering the intended quality hierarchy. The mean Spearman’s rank correlation for comparative judgments was 0.908, compared to just 0.150 for rubrics, yielding an estimated difference of 0.758 [0.494, 1.021]. Furthermore, the comparative method required less than half the annotation time. These trends were consistent across both human annotators and LLM autograders. Beyond this direct comparison, the unique paired structure of JudgmentBench facilitates a broader research agenda focused on optimizing how expert judgment is elicited, aggregated, and utilized as supervision in domains where verifiable ground truth is unavailable.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC