Title: Enhancing Search Relevance at Scale: Integrating LLM-Derived Judgments into App Store Ranking

Abstract: Commercial search platforms strive to optimize for relevance to facilitate successful user sessions that help individuals locate their desired content. To achieve maximum relevance, we employ two distinct but complementary objectives: behavioral relevance, which focuses on results users are inclined to click or download, and textual relevance, which assesses the semantic alignment between a result and the query. A significant hurdle remains the limited availability of expert-provided textual relevance labels, especially when compared to the wealth of behavioral relevance data. To tackle this issue, we systematically evaluated various LLM configurations and discovered that a specialized, fine-tuned model substantially outperforms much larger pre-trained models in generating highly accurate labels. By deploying this optimal model as a force multiplier, we produced millions of textual relevance labels to mitigate data scarcity. Our findings indicate that integrating these textual relevance labels into our production ranker significantly shifts the Pareto frontier outward: offline NDCG scores improved for both behavioral and textual relevance simultaneously. These offline improvements were confirmed through a global A/B test on the App Store ranker, which revealed a statistically significant 0.24% rise in conversion rates. The most notable performance enhancements were observed in tail queries, where the new textual relevance labels offered a strong signal in situations where reliable behavioral relevance labels were unavailable.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC