Title: Can LLM Rerankers Predict Their Own Ranking Performance?

Abstract: Since retrieval effectiveness fluctuates significantly depending on the query, it is crucial to assess ranking quality prior to the availability of relevance judgments. Query Performance Prediction (QPP) serves this purpose, yet current solutions typically depend on external predictors applied after the retrieval or reranking stages. This study explores reranker-internal QPP, specifically investigating whether an LLM reranker can accurately estimate the quality of the ranking it has just generated. We analyze both training-based and training-free methodologies. In the training-free context, we evaluate metric-specific self-consistency across sampled rankings alongside verbalized confidence scores generated directly by the reranker. Our experiments, conducted on the TREC Deep Learning datasets from 2019 to 2022 using four different LLMs, reveal that self-consistency performs competitively against the state-of-the-art (SOTA) method and demonstrates superior calibration across nearly all scenarios. Conversely, direct verbalized confidence proves to be excessively overconfident. To address this limitation, we introduce two supervised techniques, Verb-Num and Verb-List, which allow LLM rerankers to yield calibrated estimates of ranking quality with the addition of only a few output tokens.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC