Title: DOT-MoE: Leveraging Differentiable Optimal Transport for MoEfication

Abstract:

The rapid scaling of Large Language Models (LLMs) has yielded substantial performance improvements but has simultaneously introduced significant hurdles regarding inference efficiency. Although Mixture of Experts (MoE) architectures mitigate this issue by separating model size from computational cost during inference, training MoEs from the ground up is frequently computationally expensive and unstable. Consequently, transforming pre-trained dense models into sparse MoEs has become a viable alternative. However, current techniques generally depend on heuristic neuron clustering or random partitioning to divide the Feed-Forward Network (FFN) into distinct experts.

In this study, we introduce DOT-MoE, a new framework that treats the decomposition of dense layers as a Differentiable Optimal Transport (DOT) problem. Rather than relying on fixed heuristics, we frame neuron assignment as a balanced transport issue, employing differentiable Sinkhorn-Knopp iterations to guarantee strict adherence to expert capacity constraints. Additionally, we integrate Straight-Through Estimators (STE) to facilitate the end-to-end joint learning of discrete neuron-to-expert assignments and token-to-expert routing policies. Comprehensive experiments conducted across various architectures and benchmarks reveal that DOT-MoE surpasses structured pruning, heuristic clustering, and random-split baselines. Notably, the method retains 90% of the original dense model’s performance while cutting the number of active parameters in half.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC