Title: Wavelet Fourier Diffuser: A Frequency-Aware Diffusion Model for Reinforcement Learning

Abstract:

While diffusion probabilistic models have demonstrated considerable potential in offline reinforcement learning (RL) by directly modeling trajectory sequences, current methodologies tend to concentrate exclusively on time-domain characteristics. This narrow focus neglects frequency-domain attributes, a limitation that our analysis identifies as a primary cause of frequency shifts and subsequent performance degradation. To overcome these shortcomings, this study re-examines the RL challenge through a frequency-domain lens. We first identify that relying solely on time-domain features inadvertently induces shifts in the low-frequency components of the frequency spectrum, thereby compromising trajectory stability and reducing overall effectiveness.

In response, we introduce the Wavelet Fourier Diffuser (WFDiffuser), a novel diffusion-based RL framework designed to address these issues. WFDiffuser utilizes the Discrete Wavelet Transform to decompose trajectories into distinct low- and high-frequency components. To refine the diffusion modeling process for each segment, the framework incorporates Short-Time Fourier Transform alongside cross-attention mechanisms. These tools are employed to extract frequency-domain features and promote interaction across different frequency bands. Comprehensive experiments conducted on the D4RL benchmark indicate that WFDiffuser successfully alleviates frequency shift phenomena. Consequently, it generates smoother and more stable trajectories, yielding superior decision-making capabilities compared to existing approaches.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC