Title: Deep RL Representation Invariances Are Shaped by Learning Objectives

Abstract:

Reinforcement Learning (RL) has historically functioned as a computational model for goal-oriented animal behavior within neuroscience. The widespread success of modern deep RL across various fields has further solidified this interdisciplinary link. A key factor in this success is the capacity to derive abstract representations from high-dimensional state spaces. Nevertheless, the theoretical comprehension of these learned representations remains insufficient, which impedes direct comparisons between artificial models and biological learning processes. To bridge this gap, we examine deep RL representations using MDP reduction theory. Our investigation of standard RL algorithms in a navigation scenario reveals that, despite similar performance levels, different algorithms induce distinct representational structures: the value-based DQN algorithm produces representations invariant to MDP homomorphism symmetries, whereas the policy-gradient PPO algorithm yields representations invariant to action symmetries. These distinctions are consistent across various domains, influence transfer learning outcomes, and manifest in Large Language Models depending on the prompt. Our results offer a systematic framework for comparing learned representations among RL algorithms, highlighting practical applications and potential implications for understanding neural coding in the brain.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC