Title: PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

Abstract:

Mixture of Experts (MoE) Large Language Models (LLMs) demonstrate robust performance as they scale up. Nevertheless, applying reinforcement learning (RL) to these MoE architectures frequently encounters training instability. A primary driver of this issue is "router drift," a phenomenon where expert activations shift significantly between model updates and vary across the disaggregated rollout and training stages. This discrepancy leads to a substantial mismatch between rollout and training data, resulting in erratic importance sampling weights within Proximal Policy Optimization (PPO)-style RL algorithms.

While routing replay attempts to solve this by freezing the replay route within individual reasoning trajectories, it fails to account for router evolution during off-policy updates, leading to router staleness. To overcome this constraint, we introduce Predictive Routing Replay (PR2). This method equips each router with a lightweight evolution predictor designed to forecast short-term router changes.

In the rollout phase, PR2 employs the predictive routing distribution to execute top-$k$ routing. This strategy ensures that gradients are transmitted to experts that are probable to activate following updates. Subsequently, during the training phase, the system replays the predicted route to maintain consistency, thereby facilitating stable importance estimation. Both theoretical evaluations and empirical experiments indicate that PR2 effectively minimizes routing-induced mismatches, enhances RL stability, and delivers superior performance across a range of reasoning benchmarks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC