Title: Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

Abstract: Merging pre-trained Large Language Models (LLMs) offers a promising pathway for handling a wide array of reasoning challenges. Yet, traditional expert selection at the task level is frequently too broad, as individual instances often demand specific, distinct competencies. To overcome this limitation, we introduce Skill-MoE, a gradient-free, symbolic framework that operates on a skill-based Mixture-of-Experts model for selecting experts at the instance level. Skill-MoE extracts relevant skills—such as algebraic proficiency in mathematical problems—from each query, matches them with the most suitable experts, and allows each expert to formulate its own reasoning path. An aggregator, specifically chosen for its capacity to blend varied responses, then synthesizes the k resulting outputs.

Although instance-level selection significantly boosts performance, a direct implementation would suffer from substantial overhead due to the repeated loading and unloading of models. We mitigate this issue through a batch inference strategy that clusters instances according to their assigned experts, ensuring that each model is loaded just once. Consequently, Skill-MoE manages to integrate 16 expert models onto a single GPU, achieving a runtime similar to previous multi-agent baselines that required four GPUs. Evaluated across multiple benchmarks—including MMLU-Pro, GPQA, AIME, and MedMCQA—Skill-MoE demonstrates an average absolute gain of 8.15% compared to the strongest baseline. Furthermore, it shows strong generalization to unfamiliar tasks and surpasses discussion-based approaches, all without the need for costly multi-turn interactions.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC