Title: Prioritizing Representation Over Routing: An Analysis of Temporal Routing Pathologies in Multi-Timescale PPO

Abstract:

In reinforcement learning, temporal credit assignment is frequently addressed by incorporating value estimates across various discount factors. A logical progression from this approach involves enabling the actor to dynamically route between these temporal heads, employing methods such as heuristic uncertainty weights or differentiable attention mechanisms. However, this study contends that such routing strategies often establish numerical shortcuts rather than fostering robust temporal abstractions. To investigate this phenomenon, we utilize LunarLander-v2 within a controlled Proximal Policy Optimization (PPO) framework, treating the environment as a visual sandbox for identifying specific failure modes.

Our analysis first formalizes the concept of "Surrogate Objective Hacking." We demonstrate that when a differentiable softmax router is exposed to the PPO surrogate, it receives direct gradients favoring advantage heads that are numerically advantageous for the immediate update. This occurs even if the routing adjustment does not translate to better physical control. Since unnormalized advantages at different discount factors operate on differing effective scales, this dynamic introduces a vulnerability rooted in scale discrepancy.

Secondly, we highlight the "Paradox of Temporal Uncertainty" within gradient-free, error-based routing systems. In these scenarios, short-horizon heads may capture the largest share of routing because their prediction targets are inherently easier to learn, despite being less aligned with long-term task success. As a structural countermeasure, we examine "Target Decoupling." This approach allows the critic to maintain multi-timescale auxiliary heads while restricting the actor’s updates to the long-horizon advantage alone. It is important to note that Target Decoupling is not proposed as a universal performance enhancer; rather, in our experimental set, it serves to eliminate the exploitable actor-side routing pathway, thereby improving the worst-seed return observed. The code for this research is accessible at https://github.com/ben-dlwlrma/Representation-Over-Routing.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC