Title: DLLG: Dynamic Logit-Level Gating of LLM Experts

Abstract:

While employing multiple specialized large language models (LLMs) allows for the combination of complementary capabilities, current methodologies often force a compromise between adaptability and stability. Existing solutions suffer from distinct limitations: routing mechanisms make premature commitments, heuristic ensembling relies on fragile proxies, and parameter merging leads to interference. To address these challenges, we introduce DLLG (Dynamic Logit-Level Gating), a novel framework for dynamic ensembling at the logit level. This approach learns token-level expert fusion using only sparse supervision at the response level. By utilizing a lightweight gating module to predict step-wise fusion weights, DLLG connects trajectory-level correctness to the generation process, eliminating the need for token-level labels or the retraining of individual experts. Our evaluation across various reasoning and code benchmarks demonstrates that DLLG consistently surpasses strong baselines in routing, heuristic ensembling, and parameter merging across different model scales. These results underscore learned logit-level fusion as a robust and scalable paradigm for integrating specialized experts.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC