Title: Do Neural Retrievers Favor Specific Documents? Evidence of Learned Relevance Priors

Abstract:

While neural retrievers are designed to gauge query-document relevance based on annotated pairs, the annotation process itself may introduce bias. Since annotators typically label only a subset of documents, this selection criteria can inadvertently privilege certain document categories over others. This study examines whether supervised bi-encoder retrievers inadvertently acquire a document-level relevance prior—a query-independent signal embedded within their representation space as a byproduct of training on labeled data. To quantify this prior, we employ simple classifiers on frozen document embeddings and assess three leading retrievers across various information retrieval benchmarks.

Our results indicate that supervised neural models do encode relevance priors that generalize to unseen documents and remain consistent across different architectures. These priors establish a "findability gap," where documents with lower prior scores are systematically more difficult to retrieve, even if they are genuinely relevant to the query. This phenomenon is prominent in supervised dense retrievers, whereas it is notably weaker and less consistent in BM25, and it holds true even under controlled comparisons of matched documents.

Through LLM-generated explanations, we observe that documents deemed relevant by annotators are typically comprehensive, self-contained summaries of mainstream subjects. Conversely, niche, fragmented, or highly technical content is frequently excluded from judgment. Retrievers internalize this human bias, elevating documents with these favored characteristics above those that lack them, regardless of their true relevance. These findings reveal a structural constraint of supervised retrieval: models trained on annotated data do not merely learn relevance but also absorb the implicit document preferences inherent in their training sets.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC