Title: Transforming "Weak" Cues into Robust Models: Aggregating Preference Deltas via LoRA Merging

Abstract:

Developing high-performing large language models (LLMs) typically demands high-quality supervisory data, a resource that is frequently in short supply. Emerging research indicates that paired preference data generated by weak-weaker model comparisons—such as contrasting Qwen3 4B against 1.7B—can serve as a potent supervisory mechanism. Although individual responses from these pairs may lack superior quality, they offer valuable relative quality deltas, which we classify as "weak" signals. This finding raises a pivotal inquiry: is it feasible to constructively combine multiple "weak" signals to enhance more powerful models, such as Qwen3 8B?

To address this, we introduce Preference Delta Aggregation (PDA), a novel framework that extracts a preference delta from each weak-weaker model pair. Within PDA, this delta is instantiated as a Low-Rank Adaptation (LoRA) adapter, trained via preference optimization, and subsequently aggregated through LoRA merging. To address potential directional interference during the merging process, we propose Geometric Alignment Merging (GAM). This geometry-aware technique aligns adapter subspaces prior to aggregation, facilitating a more stable composition of diverse deltas.

Benchmarks focusing on agentic search and knowledge reasoning demonstrate that synthesizing multiple "weak" signals yields performance improvements that surpass any individual signal, with benefits scaling as more signals are added. Specifically, PDA combined with GAM boosts the strong model’s average performance by 6.8 points in knowledge reasoning and 7.3 points in agentic search. These results outperform all single-delta and multi-delta baseline methods, beating the top single-delta baseline by margins of 2.1 and 4.3 points, respectively. Further investigation suggests that these enhancements stem from the successful integration of complementary capabilities embedded within distinct preference deltas.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC