Title: Mapping Calibration Regimes for LLM Judge Panels

Abstract: This research investigates the optimal calibration strategies for LLM judge panels when human-labeling resources are limited, specifically comparing low-dimensional stackers against joint output tables. While low-dimensional stackers offer cost-effective estimation, they fail to capture complex interactions; conversely, joint-table calibrators can model these interactions but incur higher costs related to cell counts and the challenge of unseen data patterns. We formalize this trade-off through a "finite-calibration regime map," implemented as Finite-Calibration Panel Selection. This tool serves as a practical validation selector that evaluates judge paths, prefix lengths, and aggregator families, utilizing both table-based and parametric estimation diagnostics.

Empirical testing across RewardBench, LLMBar, SummEval, and Arena100K—utilizing a seven-judge pool that includes DeepSeek V4 Flash—reveals that scalar or reliability aggregation methods outperformed alternatives in 16 out of 20 real-world dataset-budget scenarios. This suggests that current judge outputs tend to be largely additive or redundant. However, data from controlled calibration-growth experiments highlight a complementary regime: while additive labels continue to favor scalar approaches, scenarios involving a six-way interaction necessitate larger joint tables. In such cases, once the mass of unseen patterns disappears, the test Mean Squared Error (MSE) significantly decreases from 0.224 to 0.061. Consequently, the critical practical inquiry shifts from simply asking "how many judges are needed?" to determining whether the additional information provided by the next judge can be effectively estimated given the constraints of available human labels.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC