Title: Unlocking Test-Time Compute in Frozen Embedding Models via Agentic Program Search

Abstract:

It is commonly assumed that test-time compute offers advantages exclusively to large reasoning models, leaving small models without any benefit. We challenge this perspective in the context of dense retrieval, arguing that modern small embedding models—often distilled or adapted from large language model backbones—possess latent potential for test-time computation. We investigate the extent to which a frozen embedding model can improve retrieval quality solely during inference, utilizing no auxiliary models and undergoing no parameter updates at deployment.

Through an agentic loop, a large language model generates programs interacting with a frozen encoder API. This process explores 144 candidates, resulting in twelve Pareto-optimal programs that balance inference compute against quality across cost ratios ranging from $c=1.2$ to $14.7$. Each of these programs enhances nDCG@10 performance across all 14 discovery tasks. Notably, these programs operate without trainable parameters and effectively replicate classical retrieval primitives, including reciprocal rank fusion, the Fisher linear discriminant, Rocchio pseudo-relevance feedback, and sentence-level MaxSim.

When applied unchanged to nineteen held-out tasks and three previously unseen encoder families, a single fixed program improves performance on the majority of tasks. It achieves a positive median $\Delta$nDCG@10 and secures a 54% to 57% win-rate when $c \ge 4$. The improvements are particularly pronounced on encoder families that were not part of the initial discovery phase. In contrast, a learned projection head, trained on the same tasks with a matched budget, fails to transfer effectively; while it boosts in-domain retrieval by $+0.20$ to $+0.25$ nDCG@10, it performs below the baseline on every held-out encoder.

These findings demonstrate that small embedding models inherit usable test-time compute potential. A frozen encoder can transform inference compute into retrieval gains that generalize to new corpora and encoders without requiring per-domain labels.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC